Concurrent File Store

The ConcurrentFileStore class enables the use of a concurrent dictionary as a means of storing data that is compressed, locally stored, debounced updated, and subject to revision checks. It handles all of the low-level code necessary to implement a concurrent store, offering benefits such as file verification, backup and restore functionalities, and a rich SDK for programmatic management and interaction.

The "revision checked" feature serves as a safety mechanism in the ConcurrentFileStore, ensuring the integrity of the files written to disk. Before overwriting, each file is assigned a revision, and the bytes are transactionally verified to prevent partial or corrupt file writes. This meticulous verification process safeguards the data by confirming that only complete and accurate revisions are written, thus minimizing the risk of data corruption or loss due to incomplete or erroneous file writes.

This implements File Revision Store under the hood. Check it out!

Example Use

Here's an exampel that shows allocating and using a concurrent file store.

var ctslocal = new CancellationTokenSource();

var cfs = new ConcurrentFileStore<string, string>(
    $"strings.bin", //Path
    ctslocal.Token, //Cancel token
    5000, 10000, //Debounce timers
    () => new Dictionary<string, string>() { { "a", "b" } }, //Initialization callback
    (writeCallbackBytes) => { }, //Called when a file write occurs
    (processFunction) => { }); //A function that is called ONCE after initialization or on startup.

cfs.AddOrUpdate("a", "b");
cfs.AddOrUpdate("a", "c");
cfs.AddOrUpdate("b", "d");

//Await any updates to write
cfs.Await();

//Disposing prevents future writes, forces pending updates to write.
cfs.Dispose()

SDK

ConcurrentFileStore Initialization

This method initializes a new instance of the ConcurrentFileStore class. It takes the following parameters:

• fileStore : Path where the file will be stored.

• cancelToken : Token used for cancelling the operation.

• debounceTimeMS : Specifies the time between updates/deletes until the write occurs. Default is 1000 milliseconds.

• maximumDebounceTimeMS : If repeated updates stretch debounces beyond this limit, it will trigger an immediate write. Default is 10000 milliseconds.

• initialization : Optional. If no value is read or initialized from the disk, and this callback is set, it will be called to initialize the concurrent store.

• writeCallback : Optional. If set, this function is called every time a debounced write is verified and the written bytes are also checked.

• processFunction : Optional. If assigned, the Process function is called once on startup when values are loaded back from disk or from the initialization callback. This is final step to modify the data before regular use starts.

• AES32Key : Optional. If assigned, contents will be encrypted to disk using this AES 32 bit key, in non hex format.

• AES16IV : Optional. If assigned, contents will be encrypted to disk using this AES 16 bit IV, in non hex format.

Example:

string fileStore = "/path/to/store";
CancellationToken cancellationToken = new CancellationToken();
ConcurrentFileStore concurrentFileStore = new ConcurrentFileStore(
    fileStore, 
    cancellationToken, 
    500, 
    2000, 
    null, 
    bytes => Console.WriteLine(Encoding.UTF8.GetString(bytes)), 
    dictionary => dictionary.Add("extra key", "extra value"), 
    AesCrypto.GetNewRandom(32, false), 
    AesCrypto.GetNewRandom(16, false)
);

ReadAndVerifyPath

A helper method that will read and verify a file path using the built-in compression and validation logic. Useful for initializations outside the scope of the process. This method needs a path parameter for the file path, a FailedVerification parameter to check if the verification failed, and optional AES32Key and AES16IV parameters for encryption.

Example:

ConcurrentFileStore.ReadAndVerifyPath("/some/path/to/file", out bool verificationFailed);

DecompressBytes

This function decompresses bytes, optionally with aes keys. The parameters are bytes of the data to be decompressed and optional AES32Key and AES16IV for encryption.

Example:

ConcurrentFileStore.DecompressBytes(byteData);

CompressBytes

This function compresses an object to bytes, optionally with aes keys. It requires a compObject which is the compression object and optional AES32Key and AES16IV for encryption.

Example:

ConcurrentFileStore.CompressBytes(myObject);

Write

This function writes the file using the same logic the ConcurrentFileStore uses internally. Takes parameters Dict which is the dictionary to write, FilePath which is the file to write to, Bytes which are the bytes converted, and optional AES32Key and AES16IV for encryption.

Example:

ConcurrentFileStore.Write(myDict, "/path/to/file", out byte[] writeBytes);

GetCache

This function returns the underlying cache. Any changes made here will be reflected and may cause issues if modified.

Example:

var cache = ConcurrentFileStore.GetCache();

InitializeFromBytes

The function initializes a concurrent dictionary using built-in compression/serialization methods. Requires a bytes parameter to initialize from and optional AES32Key and AES16IV for encryption.

Example:

var initializedDictionary = ConcurrentFileStore.InitializeFromBytes(myBytes);

Await

This function can be used to await any outstanding items. It does not await items being present.

Example:

ConcurrentFileStore.Await();

AwaitInitialization

This function allows to await for any source to cause an initialization of values. This may wait indefinitely if no value is sent to the concurrent writer.

Example:

ConcurrentFileStore.AwaitInitialization();

Get

The function gets an item by key. Requires a key as parameter.

Example:

var value = ConcurrentFileStore.Get(myKey);

Remove

The function removes an item by key. Requires a key as parameter.

Example:

ConcurrentFileStore.Remove(myKey);

Contains

The function checks that an item exists by key. Requires a key as parameter.

Example:

bool doesContain = ConcurrentFileStore.Contains(myKey);

AddOrUpdate

This function adds or updates a value. Takes parameters key and value.

Example:

ConcurrentFileStore.AddOrUpdate(myKey, myValue);

Keys

This function returns all keys.

Example:

var keys = ConcurrentFileStore.Keys();

SignalModified

Signal that the underlying data was modified outside of an AddOrUpdate or Remove. This is not the recommended way of changing or modifying data, but it does exist if you need to trigger a re-save check

Example:

ConcurrentFileStore.SignalModified();

UpdateWithCheckout

This allows you to modify the cache while maintaining proper locking. It's intended only to be used when you need to freeze the entire cache and operate on it, instead of individual Key objects.

Example:

ConcurrentFileStore.UpdateWithCheckout((c) => {
  //Modify c however you like, the whole cache is locked.
  
  //On return, a save request is executed
});

ValuesFromPredicate

This function searches the values using a predicate. The predicate is a function specifying the condition to be met for the values.

Example:

var values = ConcurrentFileStore.ValuesFromPredicate(x => x > 10);