2.1 Advanced Indexing

Note: This example is deprecated for the latest versions of signac and signac-flow.

Indexing files

As was shown earlier, we can create an index of the data space using the index() method:

[1]:
import signac

project = signac.get_project(root="projects/tutorial")
index = list(project.index())

for doc in index[:3]:
    print(doc)
{'_id': '0e909ffdba496bbb590fbce31f3a4563', 'statepoint': {'p': 3.4000000000000004, 'kT': 1.0, 'N': 1000, 'a': 0, 'b': 0}, 'V_liq': 0.0, 'V_gas': 294.1176470588235, 'fluid': 'ideal gas', 'signac_id': '0e909ffdba496bbb590fbce31f3a4563', 'root': 'notebooks/projects/tutorial/workspace'}
{'_id': '6a7e3ef575478edbe08e4a6e4f0e6365', 'statepoint': {'N': 1000, 'p': 2.3000000000000003, 'kT': 1.0, 'a': 1.355, 'b': 0.03201}, 'fluid': 'argon', 'V_liq': 32.802658840289595, 'V_gas': 216.99497492768137, 'signac_id': '6a7e3ef575478edbe08e4a6e4f0e6365', 'root': 'notebooks/projects/tutorial/workspace'}
{'_id': '66ca1e809607580496c8ac4825e57e7d', 'statepoint': {'p': 7.800000000000001, 'kT': 1.0, 'N': 1000, 'a': 0, 'b': 0}, 'V_liq': 0.0, 'V_gas': 128.2051282051282, 'fluid': 'ideal gas', 'signac_id': '66ca1e809607580496c8ac4825e57e7d', 'root': 'notebooks/projects/tutorial/workspace'}

We will use the Collection class to manage the index directly in-memory:

[2]:
index = signac.Collection(project.index())

This enables us for example, to quickly search for all indexes related to a specific state point:

[3]:
for doc in index.find({"statepoint.p": 0.1}):
    print(doc)
{'_id': '40405b550e7cc2d127b9758d0e764672', 'statepoint': {'N': 1000, 'p': 0.1, 'kT': 1.0, 'a': 5.536, 'b': 0.03049}, 'fluid': 'water', 'V_liq': 30.659799008990184, 'V_gas': 4999.915100495509, 'signac_id': '40405b550e7cc2d127b9758d0e764672', 'root': 'notebooks/projects/tutorial/workspace'}
{'_id': '10743bc8b95bffab09503bce9abbe627', 'statepoint': {'p': 0.1, 'kT': 1.0, 'N': 1000, 'a': 0, 'b': 0}, 'V_liq': 0.0, 'V_gas': 10000.0, 'fluid': 'ideal gas', 'signac_id': '10743bc8b95bffab09503bce9abbe627', 'root': 'notebooks/projects/tutorial/workspace'}
{'_id': 'f803d91519e23a9eee19fd9e789eeb2e', 'statepoint': {'N': 1000, 'p': 0.1, 'kT': 1.0, 'a': 1.355, 'b': 0.03201}, 'fluid': 'argon', 'V_liq': 32.804113976682224, 'V_gas': 8430.935727416612, 'signac_id': 'f803d91519e23a9eee19fd9e789eeb2e', 'root': 'notebooks/projects/tutorial/workspace'}

At this point the index contains information about the statepoint and all data stored in the job document. If we want to include the V.txt text files we used to store data in, with the index, we need to tell signac the filename pattern and optionally the file format.

[4]:
index = signac.Collection(project.index(r".*\.txt"))
for doc in index.find(limit=2):
    print(doc)
{'signac_id': '0e909ffdba496bbb590fbce31f3a4563', 'root': 'notebooks/projects/tutorial/workspace', '_id': '0e909ffdba496bbb590fbce31f3a4563', 'statepoint': {'p': 3.4000000000000004, 'kT': 1.0, 'N': 1000, 'a': 0, 'b': 0}, 'V_liq': 0, 'V_gas': 294.1176470588235, 'fluid': 'ideal gas'}
{'signac_id': '6a7e3ef575478edbe08e4a6e4f0e6365', 'root': 'notebooks/projects/tutorial/workspace', '_id': '6a7e3ef575478edbe08e4a6e4f0e6365', 'statepoint': {'N': 1000, 'p': 2.3000000000000003, 'kT': 1.0, 'a': 1.355, 'b': 0.03201}, 'fluid': 'argon', 'V_liq': 32.802658840289595, 'V_gas': 216.99497492768137}

The index contains basic information about the files within our data space, such as the path and the MD5 hash sum. The format field currently says File, which is the default value.

We can specify that all files ending with .txt are to be defined to be of TextFile format:

[5]:
index = signac.Collection(project.index({r".*\.txt": "TextFile"}))
print(index.find_one({"format": "TextFile"}))
{'signac_id': '6c482358c65a64f5af05731191151a7c', 'statepoint': {'p': 1.2000000000000002, 'kT': 1.0, 'N': 1000, 'a': 0, 'b': 0}, 'filename': '6c482358c65a64f5af05731191151a7c/V.txt', 'root': 'notebooks/projects/tutorial/workspace', 'format': 'TextFile', 'md5': 'c7872ce6468053430cbd6e5a175e2dc0', 'file_id': 'c7872ce6468053430cbd6e5a175e2dc0', '_id': '79a227db7024e02a9ad6e2df4f79561c'}

Generating a Master Index

A master index is compiled from multiple other indexes, which is useful when operating on data compiled from multiple sources, such as multiple signac projects.

To make a data space part of master index, we need to create a signac_access.py module. We use the access module to define how the index for the particular space is to be generated. We can create a basic access module using the Project.create_access_module() function:

[6]:
# Let's make sure to remoe any remnants from previous runs...
%rm -f projects/tutorial/signac_access.py

# This will generate a minimal access module:
project.create_access_module(master=False)

%cat projects/tutorial/signac_access.py
import signac

def get_indexes(root):
    yield signac.get_project(root).index()
<ipython-input-1-d3ea23001fff>:5: DeprecatedWarning: create_access_module is deprecated as of 1.5 and will be removed in 2.0. Access modules are deprecated.
  project.create_access_module(master=False)

When compiling a master index, signac will search for access modules named signac_access.py. Whenever it finds a file with that name, it will import the module and compile all indeces yielded from a function called get_indeces() into the master index.

Let’s try that!

[7]:
master_index = signac.Collection(signac.index())
for doc in master_index.find(limit=2):
    print(doc)
{'_id': '0e909ffdba496bbb590fbce31f3a4563', 'statepoint': {'p': 3.4000000000000004, 'kT': 1.0, 'N': 1000, 'a': 0, 'b': 0}, 'V_liq': 0.0, 'V_gas': 294.1176470588235, 'fluid': 'ideal gas', 'signac_id': '0e909ffdba496bbb590fbce31f3a4563', 'root': 'notebooks/projects/tutorial/workspace', 'format': None}
{'_id': '6a7e3ef575478edbe08e4a6e4f0e6365', 'statepoint': {'N': 1000, 'p': 2.3000000000000003, 'kT': 1.0, 'a': 1.355, 'b': 0.03201}, 'fluid': 'argon', 'V_liq': 32.802658840289595, 'V_gas': 216.99497492768137, 'signac_id': '6a7e3ef575478edbe08e4a6e4f0e6365', 'root': 'notebooks/projects/tutorial/workspace', 'format': None}
<ipython-input-1-323d4de3662d>:1: DeprecatedWarning: index is deprecated as of 1.3 and will be removed in 2.0. The indexing module is deprecated.
  master_index = signac.Collection(signac.index())

Please note, that we executed the index() function without specifying the project directory. The function crawled through all sub-directories below the root directory in an attempt to find acccess modules.

We can use the access module to control how exactly the index is generated, for example by adding filename and format definitions. Usually we could edit the file directly, here we will just overwrite the old one:

[8]:
access_module = r"""import signac

def get_indeces(root):
    yield signac.get_project(root).index({'.*\.txt': 'TextFile'})
"""

with open("projects/tutorial/signac_access.py", "w") as file:
    file.write(access_module)

Now files will also be part of the master index!

[9]:
master_index = signac.Collection(signac.index())
print(master_index.find_one({"format": "TextFile"}))
None
<ipython-input-1-76c28a7d66dc>:1: DeprecatedWarning: index is deprecated as of 1.3 and will be removed in 2.0. The indexing module is deprecated.
  master_index = signac.Collection(signac.index())

We can use the signac.fetch() function to directly open files associated with a particular index document:

[10]:
for doc in master_index.find({"format": "TextFile"}, limit=3):
    with signac.fetch(doc) as file:
        p = doc["statepoint"]["p"]
        V = [float(v) for v in file.read().strip().split(",")]
        print(p, V)

Think of fetch() like the built-in open() function. It allows us to retrieve and open files based on the index document (file id) instead of an absolute file path. This makes it easier to operate on data agnostic to its actual physical location.

Please note that we can specify access modules for any kind of data space, it does not have to be a signac project!

In the next section, we will learn how to use indeces in combination with pandas dataframes.