Ask or search…
K
Links
🈁

Bit Serializer

Perigee's proprietary serialization tool, Bit, stands out as a high-performance, high-compression module for serializing and deserializing data. Comparable to Avro or Protobuf in function but distinct in its foundation, Bit is developed entirely with a unique algorithm in 100% C# code. It is integral to several internal modules at Perigee, facilitating scenarios that demand exceptional performance and minimal disk usage.
Key Features of Bit include:
  • Serialization and deserialization of classes, structs, lists, and properties.
  • Support for reserialization and "patching" to update data without full rewrites.
  • Capability for partial deserialization, useful for adapting to revised data maps or focusing on specific properties.
  • Built-in compression to minimize data size.
  • Facilities for map reading and generation, enabling dynamic data structuring.
  • Tools to generate C# classes and schemas directly from data maps.

How Bit Works

Bit optimizes data storage by creating a map for each property, determining the necessity of writing each field based on defaults and nullability. This selective approach, combined with the efficient compression of values, significantly reduces the size of the output byte array. In practice, this can compress lists of objects by over 95% compared to their JSON document equivalents.

Why Choose Bit?

Unlike many high-compression algorithms that impose strict data input conditions and support limited data types, Bit offers more flexibility. It's fully recursive, supports a wide array of common data types (including custom classes, structs, lists, dictionaries, etc.), and handles data maps without cumbersome limitations.

Maps and Resilience to Change

Bit's mapping system ensures data resilience over time. For instance, objects serialized with an initial map version (R0) can be seamlessly reserialized to a newer version (R1) as your data schema evolves. This built-in functionality simplifies the process of keeping serialized data up-to-date without the complexity often associated with such transformations.

Compression Ratios

Bit demonstrates exceptional efficiency in data storage, achieving compression ratios of over 90% in many cases, and even up to 98% when optimizing data types and attribute usage, far surpassing the compression achieved with minimized JSON.

An Example

Here's a simple example where we are serializing a single class that has a string, ushort, enum, and recursive list of related "bit people".
It's extremely easy to use and Bit handles all of the data types and mapping.
var bitPerson = new BitPerson() {
Name = "Bandit",
Age = 42,
PersonType = BitPersonType.adult,
RelatedPeople = new List<BitPerson>()
{
new BitPerson() { Name = "Bingo", Age = 7, PersonType = BitPersonType.child }
} };
//Simple example without storing a map
byte[] ser = Bit2.Serialize(bitPerson);
BitPerson? des = Bit2.Deserialize<BitPerson>(ser);

Compression Comparison

If you take this example and compare the byte lengths of JSON to Bit, it's pretty crazy!
decimal compSize = Bit2.SerializeHeadless(BitPerson).Length / (decimal)JsonConvert.SerializeObject(BitPerson).Length;
// 0.108 (or a ~ tenth of the size!)

Documentation

Attributes

The primary attribute to use is Bit2Ignore. This tells the serializer to ignore this property and not serialize it. It is important to put this attribute on properties that are not necessary to store, or should be re-calculated after deserialization.

Serialization / Deserialization

Deserialize

Deserializes a byte array into an instance of a class.

Example:

YourClass obj = BitManipulation.Deserialize<YourClass>(data, map);

DeserializeHeadless

Deserializes a byte array without a header into an instance of a class. Useful for data serialized without initial header bytes.

Example:

YourClass obj = BitManipulation.DeserializeHeadless<YourClass>(data);

SerializeHeadless

Serializes an object into a byte array without adding a header.

Example:

byte[] data = BitManipulation.SerializeHeadless(obj, version);

Serialize

Serializes an object into a byte array, including a header with a version.

Example:

byte[] data = BitManipulation.Serialize(obj, version);

Serialize with Map Bytes Output

Serializes an object into a byte array and outputs the bytes of the map used for serialization.

Example:

byte[] mapBytes;
byte[] data = BitManipulation.Serialize(obj, out mapBytes, version);

Reserialize

Reserializes data from one map version to another. It enables transferring data between different revisions of a map, handling changes like new fields, removed fields, or updated data types.

Example:

byte[] newData = BitManipulation.Reserialize<YourClass>(data, oldMap, newMap);

Reserialize with Modification Callback

Similar to Reserialize, but includes a BeforeReserialization callback allowing modification of the object just before reserialization.

Example:

byte[] newData = BitManipulation.Reserialize<YourClass>(data, oldMap, newMap, obj => {
// Modify obj here
return obj;
});

Patch

Deserializes an object, patches it using a provided callback function, and then reserializes it.

Example:

byte[] patchedData = BitManipulation.Patch<YourClass>(data, obj => {
// Patch obj here
}, map);

Utilities

MapToCSharpByteString

Converts a map to a C# byte array string. Useful for storing maps internally for later deserialization.

Example:

string mapString = BitManipulation.MapToCSharpByteString(typeof(YourClass), version);

Map

Maps a type to a BitMap and optionally serializes and returns the bytes of that map.

Example:

BitMap mapped;
byte[] mapBytes = BitManipulation.Map(typeof(YourClass), version, out mapped, true);

ReadHeader

Reads the BitHeader from a byte array, using a maximum of 6 bytes. It also outputs the version used during serialization if available.

Example:

ushort version;
BitHeader header = BitManipulation.ReadHeader(bytes, out version);

GetMap

Converts a byte array to a BitMap.

Example:

BitMap map = BitManipulation.GetMap(mapBytes);

GenerateSchema

Generates a schema from a map, producing full C# classes that can be used to deserialize the data.

Example:

string schema = BitManipulation.GenerateSchema(mapBytes);