# Bit Serializer Perigee's proprietary serialization tool, Bit, stands out as a high-performance, high-compression module for serializing and deserializing data. Comparable to Avro or Protobuf in function but distinct in its foundation, Bit is developed entirely with a unique algorithm in 100% C# code. It is integral to several internal modules at Perigee, facilitating scenarios that demand exceptional performance and minimal disk usage. Key Features of Bit include: * Serialization and deserialization of classes, structs, lists, datatables, interfaced types, and properties. All without special "tagging" and pre-work * Support for reserialization and "patching" to update data without full rewrites. * If the type can be serialized, you can easily "deep clone" your object tree * Capability for partial deserialization, useful for adapting to revised data maps or focusing on specific properties. * Built-in compression to minimize data size. * Facilities for map reading and generation, enabling dynamic data structuring. * Tools to generate C# classes and schemas directly from data maps. * Very high performance serialization * Attributes for serialization events * Utilities for reference type patching ### How Bit Works Bit optimizes data storage by creating a map for each property, determining the necessity of writing each field based on defaults and nullability. This selective approach, combined with the efficient compression of values, significantly reduces the size of the output byte array. In practice, this can compress lists of objects by over 95% compared to their JSON document equivalents. ### Why Choose Bit? Unlike many high-compression algorithms that impose strict data input conditions and support limited data types, Bit offers more flexibility. It's fully recursive, supports a wide array of common data types (including custom classes, structs, lists, dictionaries, etc.), and handles data maps without cumbersome limitations. It even handles Interfaced types without adding additional code. Just make sure the interfaced type has a no-parameters constructor and all will be fine! ### Maps and Resilience to Change Bit's mapping system ensures data resilience over time. For instance, objects serialized with an initial map version (R0) can be seamlessly reserialized to a newer version (R1) as your data schema evolves. This built-in functionality simplifies the process of keeping serialized data up-to-date without the complexity often associated with such transformations. ### Compression Ratios Bit demonstrates exceptional efficiency in data storage, achieving compression ratios of over 90% in many cases, and even up to 98% when optimizing data types and attribute usage, far surpassing the compression achieved with minimized JSON. ## An Example Here's a simple example where we are serializing a single class that has a `string`, `ushort`, `enum`, and recursive list of related "`bit people`". It's extremely easy to use and Bit handles all of the data types and mapping. ```csharp var bitPerson = new BitPerson() { Name = "Bandit", Age = 42, PersonType = BitPersonType.adult, RelatedPeople = new List() { new BitPerson() { Name = "Bingo", Age = 7, PersonType = BitPersonType.child } } }; //Simple example without storing a map byte[] ser = Bit2.Serialize(bitPerson); BitPerson? des = Bit2.Deserialize(ser); ``` #### Compression Comparison If you take this example and compare the byte lengths of JSON to Bit, it's pretty crazy. The compression ratios get even bigger when a larger document is at play, as bit automatically tests for additional compression on larger objects. {% code overflow="wrap" %} ```csharp decimal compSize = Bit2.SerializeHeadless(BitPerson).Length / (decimal)JsonConvert.SerializeObject(BitPerson).Length; // 0.108 (or a ~ tenth of the size!) ``` {% endcode %} ## Documentation ## Attributes ### Ignore The first attribute is **`Bit2Ignore`**. This tells the serializer to ignore this property and not serialize it. It is important to put this attribute on properties that are not necessary to store, or should be re-calculated after deserialization. ```csharp public class BitPerson { public string Name { get; set; } [Bit2Ignore] public string ignoreExample { get; set; } = ""; } ``` ### Serialization Callbacks There are two serialization attributes as well, they are: **`Bit2PreSerialize`** and **`Bit2PostDeserialize.`** These attribute allow you to specify a singular method to be called before serialization and after deserialization. {% hint style="success" %} These methods must be instanced (not static) methods and the name does not matter. {% endhint %} ```csharp public class BitPerson { public string Name { get; set; } [Bit2PreSerialize] public void PreInvoke() { //Called before serialization. //Null out values, or store additional information needed after deserialization } [Bit2PostDeserialize] public void PostDes() { //Called after deserialization. Do whatever you need here! } } ``` #### PostDeserialize Example In almsot all cases, reference types likely don't matter. In those rare cases they do, like traversing and comparing a directed graph, those reference types are very important. In this example, we fix that by: 1. Implementing the post deserialize callback. 2. Checking if our deserialized class held it's comparer. 3. If #2 is true, calling [ReplaceReferenceTypes](#replacereferencetypes) - Which is a Bit2 utility method. All of the adjacency lists instances are now re-referenced. If you were to call `object.ReferenceEquals(a,b)` for two nodes in this graph, it would return true! ```csharp /// /// This allows a deserialized graph to "patch" reference types. /// [Bit2PostDeserialize] public void PatchReferenceTypes() { if (Comparer == null) return; Bit2.ReplaceReferenceTypes(Comparer, adjacencyList.Keys.ToList(), adjacencyList.Select(f => f.Value).ToArray()); } ``` ## Serialization Cache For scenarios where you need extreme performance, you can achieve **70-250X** faster compression times using the **`DeSerCache`** (Many of our sample test cases were only 2-10 microseconds). Creating a cache is easy. Simply supply a instantiated object and let the system pre-cache the object and create a bitMap to match. Then supply these objects to any serialization / clone / deserialization calls you want. ```csharp var bitPerson = new BitPerson(); //Generate a cache and a mpa var cd2 = Bit2.DeSerCacheFromObject(bitPerson); //Serialize with the map and cache var sk2 = Bit2.Serialize(bitPerson, cd2.map, 0, cd2.cache); ``` This is incredibly helpful if you're serializing many of the same types over and over again (like a realtime communication application or lots of data store requests). ## Serialization / Deserialization ### Deserialize Deserializes a byte array into an instance of a class. #### Example: ```csharp YourClass obj = Bit2.Deserialize(data, map); ``` ### DeserializeHeadless Deserializes a byte array without a header into an instance of a class. Useful for data serialized without initial header bytes. #### Example: ```csharp YourClass obj = Bit2.DeserializeHeadless(data); ``` ### SerializeHeadless Serializes an object into a byte array without adding a header. #### Example: ```csharp byte[] data = Bit2.SerializeHeadless(obj, version); ``` ### Serialize Serializes an object into a byte array, including a header with a version. #### Example: ```csharp byte[] data = Bit2.Serialize(obj, version); ``` ### Serialize with Map Bytes Output Serializes an object into a byte array and outputs the bytes of the map used for serialization. #### Example: ```csharp byte[] mapBytes; byte[] data = Bit2.Serialize(obj, out mapBytes, version); ``` ### Reserialize Reserializes data from one map version to another. It enables transferring data between different revisions of a map, handling changes like new fields, removed fields, or updated data types. #### Example: ```csharp byte[] newData = Bit2.Reserialize(data, oldMap, newMap); ``` ### Reserialize with Modification Callback Similar to `Reserialize`, but includes a `BeforeReserialization` callback allowing modification of the object just before reserialization. #### Example: ```csharp byte[] newData = Bit2.Reserialize(data, oldMap, newMap, obj => { // Modify obj here return obj; }); ``` ### Patch Deserializes an object, patches it using a provided callback function, and then reserializes it. #### Example: ```csharp byte[] patchedData = Bit2.Patch(data, obj => { // Patch obj here }, map); ``` ## Utilities ### DeepClone Deep clone an object by serializing and deserializing with the same mapping information. {% hint style="success" %} Please note, this only works with properties that Bit2 is able to serialize.\ Please verify your data can be serialized properly before relying on DeepClone. {% endhint %} ```csharp var bitPerson = new BitPerson() { Name = "Bandit", Age = 42, PersonType = BitPersonType.adult, RelatedPeople = new List() { new BitPerson() { Name = "Bingo", Age = 7, PersonType = BitPersonType.child } } }; var anotherBandit = Bit2.DeepClone(bitPerson); ``` ### ReplaceReferenceTypes This allows you to supply an IEqualityComparer\, and either one list, an array of lists, or an initial list + additional lists to replace all reference types. It uses the comparer to identify identical classes and then replaces those unique instances with references to itself. This is likely not necessary on most deserialized data. However, if it is necessary, you have a helper here :thumbsup: #### Example: ```csharp Bit2.ReplaceReferenceTypes(Comparer, adjacencyList.Keys.ToList(), adjacencyList.Select(f => f.Value).ToArray()); ``` ### MapToCSharpByteString Converts a map to a C# byte array string. Useful for storing maps internally for later deserialization. #### Example: ```csharp string mapString = Bit2.MapToCSharpByteString(typeof(YourClass), version); ``` ### Map Maps a type to a `BitMap` and optionally serializes and returns the bytes of that map. #### Example: ```csharp BitMap mapped; byte[] mapBytes = Bit2.Map(typeof(YourClass), version, out mapped, true); ``` ### ReadHeader Reads the `BitHeader` from a byte array, using a maximum of 6 bytes. It also outputs the version used during serialization if available. #### Example: ```csharp ushort version; BitHeader header = Bit2.ReadHeader(bytes, out version); ``` ### GetMap Converts a byte array to a `BitMap`. #### Example: ```csharp BitMap map = Bit2.GetMap(mapBytes); ``` ### GenerateSchema Generates a schema from a map, producing full C# classes that can be used to deserialize the data. #### Example: ```csharp string schema = Bit2.GenerateSchema(mapBytes); ```