🈁Bit Serializer
Perigee's proprietary serialization tool, Bit, stands out as a high-performance, high-compression module for serializing and deserializing data. Comparable to Avro or Protobuf in function but distinct in its foundation, Bit is developed entirely with a unique algorithm in 100% C# code. It is integral to several internal modules at Perigee, facilitating scenarios that demand exceptional performance and minimal disk usage.
Key Features of Bit include:
Serialization and deserialization of classes, structs, lists, datatables, interfaced types, and properties. All without special "tagging" and pre-work
Support for reserialization and "patching" to update data without full rewrites.
If the type can be serialized, you can easily "deep clone" your object tree
Capability for partial deserialization, useful for adapting to revised data maps or focusing on specific properties.
Built-in compression to minimize data size.
Facilities for map reading and generation, enabling dynamic data structuring.
Tools to generate C# classes and schemas directly from data maps.
Very high performance serialization
Attributes for serialization events
Utilities for reference type patching
How Bit Works
Bit optimizes data storage by creating a map for each property, determining the necessity of writing each field based on defaults and nullability. This selective approach, combined with the efficient compression of values, significantly reduces the size of the output byte array. In practice, this can compress lists of objects by over 95% compared to their JSON document equivalents.
Why Choose Bit?
Unlike many high-compression algorithms that impose strict data input conditions and support limited data types, Bit offers more flexibility. It's fully recursive, supports a wide array of common data types (including custom classes, structs, lists, dictionaries, etc.), and handles data maps without cumbersome limitations.
It even handles Interfaced types without adding additional code. Just make sure the interfaced type has a no-parameters constructor and all will be fine!
Maps and Resilience to Change
Bit's mapping system ensures data resilience over time. For instance, objects serialized with an initial map version (R0) can be seamlessly reserialized to a newer version (R1) as your data schema evolves. This built-in functionality simplifies the process of keeping serialized data up-to-date without the complexity often associated with such transformations.
Compression Ratios
Bit demonstrates exceptional efficiency in data storage, achieving compression ratios of over 90% in many cases, and even up to 98% when optimizing data types and attribute usage, far surpassing the compression achieved with minimized JSON.
An Example
Here's a simple example where we are serializing a single class that has a string
, ushort
, enum
, and recursive list of related "bit people
".
It's extremely easy to use and Bit handles all of the data types and mapping.
Compression Comparison
If you take this example and compare the byte lengths of JSON to Bit, it's pretty crazy. The compression ratios get even bigger when a larger document is at play, as bit automatically tests for additional compression on larger objects.
Documentation
Attributes
Ignore
The first attribute is Bit2Ignore
. This tells the serializer to ignore this property and not serialize it. It is important to put this attribute on properties that are not necessary to store, or should be re-calculated after deserialization.
Serialization Callbacks
There are two serialization attributes as well, they are: Bit2PreSerialize
and Bit2PostDeserialize.
These attribute allow you to specify a singular method to be called before serialization and after deserialization.
These methods must be instanced (not static) methods and the name does not matter.
PostDeserialize Example
In almsot all cases, reference types likely don't matter. In those rare cases they do, like traversing and comparing a directed graph, those reference types are very important. In this example, we fix that by:
Implementing the post deserialize callback.
Checking if our deserialized class held it's comparer.
If #2 is true, calling ReplaceReferenceTypes - Which is a Bit2 utility method.
All of the adjacency lists instances are now re-referenced. If you were to call object.ReferenceEquals(a,b)
for two nodes in this graph, it would return true!
Serialization Cache
For scenarios where you need extreme performance, you can achieve 70-250X faster compression times using the DeSerCache
(Many of our sample test cases were only 2-10 microseconds).
Creating a cache is easy. Simply supply a instantiated object and let the system pre-cache the object and create a bitMap to match. Then supply these objects to any serialization / clone / deserialization calls you want.
This is incredibly helpful if you're serializing many of the same types over and over again (like a realtime communication application or lots of data store requests).
Serialization / Deserialization
Deserialize
Deserializes a byte array into an instance of a class.
Example:
DeserializeHeadless
Deserializes a byte array without a header into an instance of a class. Useful for data serialized without initial header bytes.
Example:
SerializeHeadless
Serializes an object into a byte array without adding a header.
Example:
Serialize
Serializes an object into a byte array, including a header with a version.
Example:
Serialize with Map Bytes Output
Serializes an object into a byte array and outputs the bytes of the map used for serialization.
Example:
Reserialize
Reserializes data from one map version to another. It enables transferring data between different revisions of a map, handling changes like new fields, removed fields, or updated data types.
Example:
Reserialize with Modification Callback
Similar to Reserialize
, but includes a BeforeReserialization
callback allowing modification of the object just before reserialization.
Example:
Patch
Deserializes an object, patches it using a provided callback function, and then reserializes it.
Example:
Utilities
DeepClone
Deep clone an object by serializing and deserializing with the same mapping information.
Please note, this only works with properties that Bit2 is able to serialize. Please verify your data can be serialized properly before relying on DeepClone.
ReplaceReferenceTypes
This allows you to supply an IEqualityComparer<T>, and either one list, an array of lists, or an initial list + additional lists to replace all reference types. It uses the comparer to identify identical classes and then replaces those unique instances with references to itself.
Example:
MapToCSharpByteString
Converts a map to a C# byte array string. Useful for storing maps internally for later deserialization.
Example:
Map
Maps a type to a BitMap
and optionally serializes and returns the bytes of that map.
Example:
ReadHeader
Reads the BitHeader
from a byte array, using a maximum of 6 bytes. It also outputs the version used during serialization if available.
Example:
GetMap
Converts a byte array to a BitMap
.
Example:
GenerateSchema
Generates a schema from a map, producing full C# classes that can be used to deserialize the data.
Example:
Last updated