API Reference

This is the API for the synced_collections package.

Collection Types

class synced_collections.SyncedCollection(parent=None, *args, **kwargs)

Bases: Collection

An abstract Collection type that is synced with a backend.

This class extends collections.abc.Collection and adds a number of abstract internal methods that must be implemented by its subclasses. These methods can be split into two groups of functions that are designed to be implemented by separate subtrees in the inheritance hierarchy that can then be composed:

Concrete Collection Types

These subclasses should implement the APIs for specific types of collections. For instance, a list-like SyncedCollection should implement the standard methods for sequences. In addition, they must implement the following abstract methods defined by the SyncedCollection:

• is_base_type(): Determines whether an object satisfies the semantics of the collection object a given SyncedCollection is designed to mimic.

• _to_base(): Converts a SyncedCollection to its natural base type (e.g. a list).

• _update(): Updates the SyncedCollection to match the contents of the provided collections.abc.Collection. After calling sc._update(c), we must have that sc == c; however, since such updates are frequent when loading and saving data to a resource, _update() should be implemented to minimize new object creation wherever possible.

Backend

These subclasses encode the process by which in-memory data is converted into a representation suitable for a particular backend. For instance, a JSON backend should know how to save a Python object into a JSON-encoded file and then read that object back.

• _load_from_resource(): Loads data from the underlying resource and returns it in an object satisfying is_base_type().

• _save_to_resource(): Stores data to the underlying resource.

• _backend: A unique string identifier for the resource backend.

Since these functionalities are effectively completely orthogonal, members of a given group should be interchangeable. For instance, a dict-like SyncedCollection can be combined equally easily with JSON, MongoDB, or SQL backends.

Validation

Due to the restrictions of a particular backend or the needs of a particular application, synced collections may need to restrict the data that they can store. Validators provide a standardized mechanism for this. A validator is a callable that parses any data added to a SyncedCollection and raises an Exception if any invalid data is provided. Validators cannot modify the data and should have no side effects. They are purely provided as a mechanism to reject invalid data. For example, a JSON validator would raise Exceptions if it detected non-string keys in a dict.

Since SyncedCollection is designed for extensive usage of inheritance, validators may be inherited by subclasses. There are two attributes that subclasses of SyncedCollection can define to control the validators used:

• _validators: A list of callables that will be inherited by all subclasses.

• _all_validators: A list of callables that will be used to validate this class, and this class alone.

When a SyncedCollection subclass is initialized (note that this is at class definition time, not when instances are created), its _register_validators() method will be called. If this class defines an _all_validators attribute, this set of validators will be used by all instances of this class. Otherwise, _register_validators() will traverse the MRO and collect the _validators attributes from all parents of a class, and store these in the _all_validators attribute for the class.

Note

Typically, a synced collection will be initialized with resource information, and data will be pulled from that resource. However, initializing with both data and resource information is a valid use case. In this case, the initial data will be validated by the standard validators, however, it will not be checked against the contents stored in the synced resource and is assumed to be consistent. This constructor pattern can be useful to avoid unnecessary resource accesses.

Thread safety

Whether or not SyncedCollection objects are thread-safe depends on the implementation of the backend. Thread-safety of SyncedCollection objects is predicated on backends providing an atomic write operation. All concrete collection types use mutexes to guard against concurrent write operations, while allowing read operations to happen freely. The validity of this mode of access depends on the write operations of a SyncedCollection being atomic, specifically the :meth:`~._save_to_resource method. Whether or not a particular subclass of SyncedCollection is thread-safe should be indicated by that subclass setting the _supports_threading class variable to True. This variable is set to False by SyncedCollection, so subclasses must explicitly opt-in to support threading by setting this variable to True.

Backends that support multithreaded execution will have multithreaded support turned on by default. This support can be enabled or disabled using the enable_multithreading() and disable_multithreading() methods. enable_multithreading() will raise a ValueError if called on a class that does not support multithreading.

Parameters:

parent (SyncedCollection, optional) – If provided, the collection within which this collection is nested (Default value = None). A parent instance of SyncedCollection or None. If None, the collection owns its own data, otherwise it is nested within its parent. Every SyncedCollection either owns its own data, or has a parent (Default value = None).

__call__()

Get an equivalent but unsynced object of the base data type.

Returns:

An equivalent unsynced collection satisfying is_base_type().

Return type:

Collection

classmethod disable_multithreading()

Disable all safety checks and thread locks required for thread safety.

The mutex locks required to enable multithreading introduce nontrivial performance costs, so they can be disabled for classes that support it.

classmethod enable_multithreading()

Enable safety checks and thread locks required for thread safety.

Support for multithreaded execution can be disabled by calling disable_multithreading(); calling this method reverses that.

abstract classmethod is_base_type(data)

Check whether data is of the same base type (such as list or dict) as this class.

Parameters:

data (Any) – The input data to test.

Returns:

Whether or not the object can be converted into this synced collection type.

Return type:

bool

registry: DefaultDict[str, List[Any]] = {}

class synced_collections.SyncedDict(data=None, _validate=True, *args, **kwargs)

Bases: SyncedCollection, MutableMapping

Implement the dict data structure along with values access through attributes named as keys.

The SyncedDict inherits from SyncedCollection and MutableMapping. Therefore, it behaves like a dict.

Parameters:

• data (Mapping, optional) – The initial data to populate the dict. If None, defaults to {} (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the SyncedDict object behaves like a dict, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying backend, copying (even deep copying) a SyncedDict instance may exhibit unexpected behavior. If a true copy is required, you should use the __call__() method to get a dict representation, and if necessary construct a new SyncedDict.

clear() → None.  Remove all items from D.

get(k[, d]) → D[k] if k in D, else d.  d defaults to None.

classmethod is_base_type(data)

Check whether the data is an instance of mapping.

Parameters:

data (any) – Data to be checked.

Return type:

bool

items() → a set-like object providing a view on D's items

keys() → a set-like object providing a view on D's keys

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

reset(data)

Update the instance with new data.

Parameters:

data (mapping) – Data to update the instance.

Raises:

ValueError – If the data is not a mapping.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values() → an object providing a view on D's values

class synced_collections.SyncedList(data=None, _validate=True, *args, **kwargs)

Bases: SyncedCollection, MutableSequence

Implementation of list data structure.

The SyncedList inherits from SyncedCollection and MutableSequence. Therefore, it behaves similar to a list.

Parameters:

• data (Sequence, optional) – The initial data to populate the list. If None, defaults to [] (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the SyncedList object behaves like a list, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying backend, copying (even deep copying) a SyncedList instance may exhibit unexpected behavior. If a true copy is required, you should use the __call__() method to get a list representation, and if necessary construct a new SyncedList.

append(item)

S.append(value) – append value to the end of the sequence

clear() → None -- remove all items from S

extend(iterable)

S.extend(iterable) – extend sequence by appending elements from the iterable

insert(index, item)

S.insert(index, value) – insert value before index

classmethod is_base_type(data)

Check whether the data is an non-string Sequence.

Parameters:

data (Any) – Data to be checked

Return type:

bool

remove(value)

S.remove(value) – remove first occurrence of value. Raise ValueError if the value is not present.

reset(data)

Update the instance with new data.

Parameters:

data (non-string Sequence) – Data to update the instance.

Raises:

ValueError – If the data is not a non-string sequence.

class synced_collections.data_types.attr_dict.AttrDict

Bases: object

A class that redirects attribute access methods to __getitem__.

Although this class is called an AttrDict, it does not directly inherit from any dict-like class or offer any relevant APIs. Its only purpose is to be used as a mixin with other dict-like classes to add attribute-based access to dictionary contents.

Subclasses that inherit from this class must define the _PROTECTED_KEYS class variable, which indicates known attributes of the object. This indication is necessary because otherwise accessing obj.data is ambiguous as to whether it is a reference to a special data attribute or whether it is equivalent to obj['data']. Without this variable, a user could mask internal variables inaccessible via normal attribute access by adding dictionary keys with the same name.

Examples

>>> assert dictionary['foo'] == dictionary.foo

Backends

class synced_collections.backends.collection_json.JSONCollection(filename=None, write_concern=False, *args, **kwargs)

Bases: SyncedCollection

A SyncedCollection that synchronizes with a JSON file.

This collection implements synchronization by reading and writing the associated JSON file in its entirety for every read/write operation. This backend is a good choice for maximum accessibility and transparency since all data is immediately accessible in the form of a text file with no additional tooling, but is likely a poor choice for high performance applications.

Thread safety

The JSONCollection is thread-safe on Unix-like systems (not Windows, see the Warnings section). To make these collections safe, the write_concern flag is ignored in multithreaded execution, and the write is always performed via a write to temporary file followed by a replacement of the original file. The file replacement operation uses os.replace(), which is guaranteed to be atomic by the Python standard.

Parameters:

• filename (str) – The filename of the associated JSON file on disk.

• write_concern (bool, optional) – Ensure file consistency by writing changes back to a temporary file first, before replacing the original file (Default value = False).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

This class is not thread safe on Windows. It relies on os.replace for atomic file writes, and that method can fail in multithreaded situations if open handles exist to the destination file within the same process on a different thread. See https://bugs.python.org/issue46003 for more information.

property filename

Get or set the name of the associated JSON file on disk.

Type:

str

class synced_collections.backends.collection_mongodb.MongoDBCollection(collection=None, uid=None, parent=None, *args, **kwargs)

Bases: SyncedCollection

A SyncedCollection that synchronizes with a MongoDB document.

In MongoDB, a database is composed of multiple MongoDB collections, which are analogous to tables in SQL databases but do not enforce a schema like in relational databases. In turn, collections are composed of documents, which are analogous to rows in a table but are much more flexible, storing any valid JSON object in a JSON-like encoded format known as BSON (“binary JSON”).

Each MongoDBCollection can be represented as a MongoDB document, so this backend stores the MongoDBCollection as a single document within the collection provided by the user. The document is identified by a unique key provided by the user.

Thread safety

The MongoDBCollection is not thread-safe.

Parameters:

• collection (pymongo.collection.Collection) – The MongoDB client in which to store data.

• uid (dict) – The unique key-value mapping added to the data and stored in the document so that it is uniquely identifiable in the MongoDB collection. The key “data” is reserved and may not be part of this uid.

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

The user is responsible for providing a unique id such that there are no possible collisions between different MongoDBCollection instances stored in the same MongoDB collection. Failure to do so may result in data corruption if multiple documents are found to be apparently associated with a given uid.

property collection

Get the collection being synced to.

Type:

pymongo.collection.Collection

property uid

Get the unique mapping used to identify this collection.

Type:

dict

class synced_collections.backends.collection_redis.RedisCollection(client=None, key=None, *args, **kwargs)

Bases: SyncedCollection

A SyncedCollection that synchronizes with a Redis database.

This backend stores data in Redis by associating it with the provided key.

Thread safety

The RedisCollection is not thread-safe.

Parameters:

• client (redis.Redis) – The Redis client used to persist data.

• key (str) – The key associated with this collection in the Redis database.

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

property client

The Redis client used to store the data.

Type:

redis.Redis

property key

The key associated with this collection stored in Redis.

Type:

str

class synced_collections.backends.collection_zarr.ZarrCollection(group=None, name=None, codec=None, *args, **kwargs)

Bases: SyncedCollection

A SyncedCollection that synchronizes with a Zarr group.

Since Zarr is designed for storage of array-like data, this backend implements synchronization by storing the collection in a 1-element object array. The user provides the group within which to store the data and the name of the data in the group.

Thread safety

The ZarrCollection is not thread-safe.

Parameters:

• group (zarr.hierarchy.Group) – The Zarr group in which to store data.

• name (str) – The name under which this collection is stored in the Zarr group.

• codec (numcodecs.abc.Codec) – The encoding mechanism for the data. If not provided, defaults to JSON encoding (Default value: None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

property codec

The encoding method used for the data.

Type:

numcodecs.abc.Codec

property group

The Zarr group storing the data.

Type:

zarr.hierarchy.Group

property name

The name of this data in the Zarr group.

Type:

str

Buffers

class synced_collections.buffers.buffered_collection.BufferedCollection(*args, **kwargs)

Bases: SyncedCollection

A SyncedCollection defining an interface for buffering.

The default behavior of this class is not to buffer. This class simply defines an appropriate interface for buffering behavior so that client code can rely on these methods existing, e.g. to be able to do things like with collection.buffered: .... This feature allows client code to indicate to the collection when it is safe to buffer reads and writes, which usually means guaranteeing that the synchronization destination (e.g. an underlying file or database entry) will not be modified by other processes concurrently with the set of operations within the buffered block. However, in the default case the result of this will be a no-op and all data will be immediately synchronized with the backend.

The BufferedCollection overrides the _load() and _save() methods to check whether buffering is enabled or not. If not, the behavior is identical to the parent class. When in buffered mode, however, the BufferedCollection introduces two additional hooks that can be overridden by subclasses to control how the collection behaves while buffered:

• _load_from_buffer(): Loads data while in buffered mode and returns it in an object satisfying is_base_type(). The default behavior is to simply call _load_from_resource().

• _save_to_buffer(): Stores data while in buffered mode. The default behavior is to simply call _save_to_resource().

Thread safety

Whether or not buffering is thread safe depends on the buffering method used. In general, both the buffering logic and the data type operations must be thread safe for the resulting collection type to be thread safe.

classmethod backend_is_buffered()

Check if this backend is currently buffered.

classmethod buffer_backend()

Enter context to buffer all operations for this backend.

class synced_collections.buffers.file_buffered_collection.FileBufferedCollection(parent=None, filename=None, *args, **kwargs)

Bases: BufferedCollection

A SyncedCollection that can buffer file I/O.

This class provides a standardized buffering protocol for all file-based backends. All file-based backends can use the same set of integrity checks prior to a buffer flush to ensure that no conflicting modifications are made. Specifically, they can check whether the file has been modified on disk since it was originally loaded to the buffer. This class provides the basic infrastructure for that and defines standard methods that can be used by all classes. Subclasses must define the appropriate storage mechanism.

Note

Important notes for developers:

• This class should be inherited before any other collections. This requirement is due to the extensive use of multiple inheritance. Since this class is designed to be combined with other SyncedCollection types without making those types aware of buffering behavior, it transparently hooks into the initialization process, but this is dependent on its constructor being called before those of other classes.

• All subclasses must define a class level _BUFFER_CAPACITY variable that is used to determine the maximum allowable buffer size.

Parameters:

filename (str, optional) – The filename of the associated JSON file on disk (Default value = None).

Warning

Although it can be done safely, in general modifying two different collections pointing to the same underlying resource while both are in different buffering modes is unsupported and can lead to undefined behavior. This class makes a best effort at performing safe modifications, but it is possible to construct nested buffered contexts for different objects that can lead to an invalid buffer state, or even situations where there is no obvious indicator of what is the canonical source of truth. In general, if you need multiple objects pointing to the same resource, it is strongly recommeneded to work with both of them in identical buffering states at all times.

classmethod buffer_backend(buffer_capacity=None)

Enter context to buffer all operations for this backend.

Parameters:

buffer_capacity (int) – The capacity of the buffer to use within this context (resets after the context is exited).

classmethod disable_multithreading()

Disable all safety checks and thread locks required for thread safety.

This method adds managed buffer-related thread safety in addition to what the parent method does.

classmethod enable_multithreading()

Enable safety checks and thread locks required for thread safety.

This method adds managed buffer-related thread safety in addition to what the parent method does.

classmethod get_buffer_capacity()

Get the current buffer capacity.

Returns:

The amount of data that can be stored before a flush is triggered in the appropriate units for a particular buffering implementation.

Return type:

int

classmethod get_current_buffer_size()

Get the total amount of data currently stored in the buffer.

Returns:

The size of all data contained in the buffer in the appropriate units for a particular buffering implementation.

Return type:

int

classmethod set_buffer_capacity(new_capacity)

Update the buffer capacity.

Parameters:

new_capacity (int) – The new capacity of the buffer in the appropriate units for a particular buffering implementation.

class synced_collections.buffers.memory_buffered_collection.SharedMemoryFileBufferedCollection(parent=None, filename=None, *args, **kwargs)

Bases: FileBufferedCollection

A SyncedCollection that defers all I/O when buffered.

This class extends the FileBufferedCollection and implements a concrete storage mechanism in which collections store a reference to their data in a buffer. This method takes advantage of the reference-based semantics of built-in Python mutable data types like dicts and lists. All collections referencing the same file are pointed to the same underlying data store in buffered mode, allowing all changes in one to be transparently reflected in the others. To further improve performance, the buffer size is determined only based on the number of modified collections stored, not the total number. As a result, the maximum capacity is only reached when a large number of modified collections are stored, and unmodified collections are only removed from the buffer when a buffered context is exited (rather than when buffer capacity is exhausted). See the Warnings section for more information.

The buffer size and capacity for this class is measured in the total number of collections stored in the buffer that have undergone any modifications since their initial load from disk. A sequence of read-only operations will load data into the buffer, but the apparent buffer size will be zero.

Note

Important note for subclasses: This class should be inherited before any other collections. This requirement is due to the extensive use of multiple inheritance: since this class is designed to be combined with other SyncedCollection types without making those types aware of buffering behavior, it transparently hooks into the initialization process, but this is dependent on its constructor being called before those of other classes.

Thread safety

This buffering method is thread safe. This thread safety is independent of the safety of an individual collection backend; the backend must support thread safe writes to the underlying resource in order for a buffered version using this class to be thread safe for general use. The thread safety guaranteed by this class only concerns buffer reads, writes, and flushes. All these operations are serialized because there is no way to prevent one collection from triggering a flush while another still thinks its data is in the cache; however, this shouldn’t be terribly performance-limiting since in buffered mode we’re avoiding I/O anyway and that’s the only thing that can be effectively parallelized here.

Parameters:

filename (str, optional) – The filename of the associated JSON file on disk (Default value = None).

Warning

• Although it can be done safely, in general modifying two different collections pointing to the same underlying resource while both are in different buffering modes is unsupported and can lead to undefined behavior. This class makes a best effort at performing safe modifications, but it is possible to construct nested buffered contexts for different objects that can lead to an invalid buffer state, or even situations where there is no obvious indicator of what is the canonical source of truth. In general, if you need multiple objects pointing to the same resource, it is strongly recommeneded to work with both of them in identical buffering states at all times.

• This buffering method has no upper bound on the buffer size if all operations on buffered objects are read-only operations. If a strict upper bound is required, for instance due to strict virtual memory limits on a given system, use of the SerializedFileBufferedCollection will allow limiting the total memory usage of the process.

class synced_collections.buffers.serialized_file_buffered_collection.SerializedFileBufferedCollection(parent=None, filename=None, *args, **kwargs)

Bases: FileBufferedCollection

A FileBufferedCollection based on a serialized data store.

This class extends the FileBufferedCollection and implements a concrete storage mechanism in which data is encoded (by default, into JSON) and stored into a buffer. This buffer functions as a central data store for all collections and is a synchronization point for various collections pointing to the same underlying file. This serialization method may be a bottleneck in some applications; see the Warnings section for more information.

The buffer size and capacity for this class is measured in the total number of bytes stored in the buffer that correspond to file data. This is not the total size of the buffer, which also contains additional information like the hash of the data and the file metadata (which are used for integrity checks), but it is the relevant metric for users.

Note

Important note for subclasses: This class should be inherited before any other collections. This requirement is due to the extensive use of multiple inheritance: since this class is designed to be combined with other SyncedCollection types without making those types aware of buffering behavior, it transparently hooks into the initialization process, but this is dependent on its constructor being called before those of other classes.

Thread safety

This buffering method is thread safe. This thread safety is independent of the safety of an individual collection backend; the backend must support thread safe writes to the underlying resource in order for a buffered version using this class to be thread safe for general use. The thread safety guaranteed by this class only concerns buffer reads, writes, and flushes. All these operations are serialized because there is no way to prevent one collection from triggering a flush while another still thinks its data is in the cache.

Parameters:

filename (str, optional) – The filename of the associated JSON file on disk (Default value = None).

Warning

• Although it can be done safely, in general modifying two different collections pointing to the same underlying resource while both are in different buffering modes is unsupported and can lead to undefined behavior. This class makes a best effort at performing safe modifications, but it is possible to construct nested buffered contexts for different objects that can lead to an invalid buffer state, or even situations where there is no obvious indicator of what is the canonical source of truth. In general, if you need multiple objects pointing to the same resource, it is strongly recommeneded to work with both of them in identical buffering states at all times.

• The overhead of this buffering method is quite high due to the constant encoding and decoding of data. For performance-critical applications where memory is not highly constrained and virtual memory limits are absent, the SharedMemoryFileBufferedCollection may be more appropriate.

• Due to the possibility of read operations triggering a flush, the contents of the buffer may be invalidated on loads as well. To prevent this even nominally read-only operations are serialized. As a result, although this class is thread safe, it will effectively serialize all operations and will therefore not be performant.

Concrete Collections

class synced_collections.backends.collection_json.JSONDict(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: JSONCollection, SyncedDict

A dict-like data structure that synchronizes with a persistent JSON file.

Examples

>>> doc = JSONDict('data.json', write_concern=True)
>>> doc['foo'] = "bar"
>>> assert doc['foo'] == "bar"
>>> assert 'foo' in doc
>>> del doc['foo']
>>> doc['foo'] = dict(bar=True)
>>> doc
{'foo': {'bar': True}}

Parameters:

• filename (str, optional) – The filename of the associated JSON file on disk (Default value = None).

• write_concern (bool, optional) – Ensure file consistency by writing changes back to a temporary file first, before replacing the original file (Default value = False).

• data (collections.abc.Mapping, optional) – The initial data passed to JSONDict. If None, defaults to {} (Default value = None).

• parent (JSONCollection, optional) – A parent instance of JSONCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the JSONDict object behaves like a dict, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying file, copying (even deep copying) a JSONDict instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new JSONDict instance.

class synced_collections.backends.collection_json.JSONList(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: JSONCollection, SyncedList

A list-like data structure that synchronizes with a persistent JSON file.

Only non-string sequences are supported by this class.

Examples

>>> synced_list = JSONList('data.json', write_concern=True)
>>> synced_list.append("bar")
>>> assert synced_list[0] == "bar"
>>> assert len(synced_list) == 1
>>> del synced_list[0]

Parameters:

• filename (str, optional) – The filename of the associated JSON file on disk (Default value = None).

• write_concern (bool, optional) – Ensure file consistency by writing changes back to a temporary file first, before replacing the original file (Default value = None).

• data (non-str collections.abc.Sequence, optional) – The initial data passed to JSONList. If None, defaults to [] (Default value = None).

• parent (JSONCollection, optional) – A parent instance of JSONCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the JSONList object behaves like a list, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying file, copying (even deep copying) a JSONList instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new JSONList instance.

class synced_collections.backends.collection_json.BufferedJSONCollection(parent=None, filename=None, *args, **kwargs)

Bases: SerializedFileBufferedCollection, JSONCollection

A JSONCollection that supports I/O buffering.

This class implements the buffer protocol defined by BufferedCollection. The concrete implementation of buffering behavior is defined by the SerializedFileBufferedCollection.

class synced_collections.backends.collection_json.BufferedJSONDict(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: BufferedJSONCollection, SyncedDict

A buffered JSONDict.

class synced_collections.backends.collection_json.BufferedJSONList(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: BufferedJSONCollection, SyncedList

A buffered JSONList.

class synced_collections.backends.collection_json.MemoryBufferedJSONCollection(parent=None, filename=None, *args, **kwargs)

Bases: SharedMemoryFileBufferedCollection, JSONCollection

A JSONCollection that supports I/O buffering.

This class implements the buffer protocol defined by BufferedCollection. The concrete implementation of buffering behavior is defined by the SharedMemoryFileBufferedCollection.

class synced_collections.backends.collection_json.MemoryBufferedJSONDict(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: MemoryBufferedJSONCollection, SyncedDict

A buffered JSONDict.

class synced_collections.backends.collection_json.MemoryBufferedJSONList(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: MemoryBufferedJSONCollection, SyncedList

A buffered JSONList.

class synced_collections.backends.collection_json.JSONAttrDict(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: JSONDict, AttrDict

A dict-like data structure that synchronizes with a persistent JSON file.

Unlike JSONDict, this class also supports attribute-based access to dictionary contents, e.g. doc.foo == doc['foo'].

Examples

>>> doc = JSONAttrDict('data.json', write_concern=True)
>>> doc['foo'] = "bar"
>>> assert doc.foo == doc['foo'] == "bar"
>>> assert 'foo' in doc
>>> del doc['foo']
>>> doc['foo'] = dict(bar=True)
>>> doc
{'foo': {'bar': True}}
>>> doc.foo.bar = False
>>> doc
{'foo': {'bar': False}}

Parameters:

• filename (str, optional) – The filename of the associated JSON file on disk (Default value = None).

• write_concern (bool, optional) – Ensure file consistency by writing changes back to a temporary file first, before replacing the original file (Default value = False).

• data (collections.abc.Mapping, optional) – The initial data passed to JSONAttrDict. If None, defaults to {} (Default value = None).

• parent (JSONCollection, optional) – A parent instance of JSONCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the JSONAttrDict object behaves like a dict, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying file, copying (even deep copying) a JSONAttrDict instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new JSONAttrDict instance.

class synced_collections.backends.collection_json.JSONAttrList(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: JSONList

A JSONList whose dict-like children will be of type JSONAttrDict.

class synced_collections.backends.collection_json.BufferedJSONAttrDict(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: BufferedJSONDict, AttrDict

A buffered JSONAttrDict.

class synced_collections.backends.collection_json.BufferedJSONAttrList(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: BufferedJSONList

A BufferedJSONList whose dict-like children will be of type BufferedJSONAttrDict.

class synced_collections.backends.collection_json.MemoryBufferedJSONAttrDict(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: MemoryBufferedJSONDict, AttrDict

A buffered JSONAttrDict.

class synced_collections.backends.collection_json.MemoryBufferedJSONAttrList(filename=None, write_concern=False, data=None, parent=None, *args, **kwargs)

Bases: MemoryBufferedJSONList

A MemoryBufferedJSONList whose dict-like children will be of type MemoryBufferedJSONAttrDict.

class synced_collections.backends.collection_mongodb.MongoDBDict(collection=None, uid=None, data=None, parent=None, *args, **kwargs)

Bases: MongoDBCollection, SyncedDict

A dict-like data structure that synchronizes with a document in a MongoDB collection.

Examples

>>> doc = MongoDBDict('data')
>>> doc['foo'] = "bar"
>>> assert doc['foo'] == "bar"
>>> assert 'foo' in doc
>>> del doc['foo']
>>> doc['foo'] = dict(bar=True)
>>> doc
{'foo': {'bar': True}}

Parameters:

• collection (pymongo.collection.Collection, optional) – A pymongo.collection.Collection instance (Default value = None).

• uid (dict, optional) – The unique key-value mapping identifying the collection (Default value = None).

• data (non-str collections.abc.Mapping, optional) – The initial data passed to MongoDBDict. If None, defaults to {} (Default value = None).

• parent (MongoDBCollection, optional) – A parent instance of MongoDBCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the MongoDBDict object behaves like a dict, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying database, copying a MongoDBDict instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new MongoDBDict instance.

class synced_collections.backends.collection_mongodb.MongoDBList(collection=None, uid=None, data=None, parent=None, *args, **kwargs)

Bases: MongoDBCollection, SyncedList

A list-like data structure that synchronizes with a document in a MongoDB collection.

Only non-string sequences are supported by this class.

Examples

>>> synced_list = MongoDBList('data')
>>> synced_list.append("bar")
>>> assert synced_list[0] == "bar"
>>> assert len(synced_list) == 1
>>> del synced_list[0]

Parameters:

• collection (pymongo.collection.Collection, optional) – A pymongo.collection.Collection instance (Default value = None).

• uid (dict, optional) – The unique key-value mapping identifying the collection (Default value = None).

• data (non-str collections.abc.Sequence, optional) – The initial data passed to MongoDBList. If None, defaults to [] (Default value = None).

• parent (MongoDBCollection, optional) – A parent instance of MongoDBCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the MongoDBList object behaves like a list, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying database, copying a MongoDBList instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new MongoDBList instance.

class synced_collections.backends.collection_zarr.ZarrDict(group=None, name=None, data=None, parent=None, *args, **kwargs)

Bases: ZarrCollection, SyncedDict

A dict-like data structure that synchronizes with a Zarr group.

Examples

>>> doc = ZarrDict('data')
>>> doc['foo'] = "bar"
>>> assert doc['foo'] == "bar"
>>> assert 'foo' in doc
>>> del doc['foo']
>>> doc['foo'] = dict(bar=True)
>>> doc
{'foo': {'bar': True}}

Parameters:

• group (zarr.hierarchy.Group, optional) – The group in which to store data (Default value = None).

• name (str, optional) – The name of the collection (Default value = None).

• data (collections.abc.Mapping, optional) – The initial data passed to ZarrDict. If None, defaults to {} (Default value = None).

• parent (ZarrCollection, optional) – A parent instance of ZarrCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the ZarrDict object behaves like a dict, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying database, copying (even deep copying) a ZarrDict instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new ZarrDict instance.

class synced_collections.backends.collection_zarr.ZarrList(group=None, name=None, data=None, parent=None, *args, **kwargs)

Bases: ZarrCollection, SyncedList

A list-like data structure that synchronizes with a Zarr group.

Only non-string sequences are supported by this class.

Examples

>>> synced_list = ZarrList('data')
>>> synced_list.append("bar")
>>> assert synced_list[0] == "bar"
>>> assert len(synced_list) == 1
>>> del synced_list[0]

Parameters:

• group (zarr.hierarchy.Group, optional) – The group in which to store data (Default value = None).

• name (str, optional) – The name of the collection (Default value = None).

• data (non-str collections.abc.Sequence, optional) – The initial data passed to ZarrList. If None, defaults to [] (Default value = None).

• parent (ZarrCollection, optional) – A parent instance of ZarrCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the ZarrList object behaves like a list, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying database, copying (even deep copying) a ZarrList instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new ZarrList instance.

class synced_collections.backends.collection_redis.RedisDict(client=None, key=None, data=None, parent=None, *args, **kwargs)

Bases: RedisCollection, SyncedDict

A dict-like data structure that synchronizes with a persistent Redis database.

Examples

>>> doc = RedisDict('data')
>>> doc['foo'] = "bar"
>>> assert doc['foo'] == "bar"
>>> assert 'foo' in doc
>>> del doc['foo']
>>> doc['foo'] = dict(bar=True)
>>> doc
{'foo': {'bar': True}}

Parameters:

• client (redis.Redis, optional) – A redis client (Default value = None).

• key (str, optional) – The key of the collection (Default value = None).

• data (collections.abc.Mapping, optional) – The initial data passed to RedisDict. If None, defaults to {} (Default value = None).

• parent (RedisCollection, optional) – A parent instance of RedisCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the RedisDict object behaves like a dict, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying database, copying a RedisDict instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new RedisDict instance.

class synced_collections.backends.collection_redis.RedisList(client=None, key=None, data=None, parent=None, *args, **kwargs)

Bases: RedisCollection, SyncedList

A list-like data structure that synchronizes with a persistent Redis database.

Only non-string sequences are supported by this class.

Examples

>>> synced_list = RedisList('data')
>>> synced_list.append("bar")
>>> assert synced_list[0] == "bar"
>>> assert len(synced_list) == 1
>>> del synced_list[0]

Parameters:

• client (redis.Redis, optional) – A Redis client (Default value = None).

• key (str, optional) – The key of the collection (Default value = None).

• data (non-str collections.abc.Sequence, optional) – The initial data passed to RedisList. If None, defaults to [] (Default value = None).

• parent (RedisCollection, optional) – A parent instance of RedisCollection or None. If None, the collection owns its own data (Default value = None).

• *args – Positional arguments forwarded to parent constructors.

• **kwargs – Keyword arguments forwarded to parent constructors.

Warning

While the RedisList object behaves like a list, there are important distinctions to remember. In particular, because operations are reflected as changes to an underlying database, copying a RedisList instance may exhibit unexpected behavior. If a true copy is required, you should use the call operator to get a dictionary representation, and if necessary construct a new RedisList instance.