If you're having issues getting a Perigee application to start or load properly, check the below items:

License check

The license check run's immediately on application startup, and if a valid license is not detected, it will immediately shut down the application.

1. Make sure you have a valid license, and your trial period is not expired

2. Make sure you're copying that license file to the output directory, or it is installed on the server.

Serilog configuration

If a Serilog sink is configured to startup on a remote source (like a database), Please verify that it has an open connection and there are no firewalls preventing Serilog from initializing.

Installation

Find the package manager and type install-package Perigee. It's available on .

Anatomy of a Perigee Application

Perigee Applications are created around two main concepts: Threads and Events.

Events are the source for starting a process:

• An API call

• An email received

• A scheduled task at a specific date/time

• A file dropped on disk

• A new message from a queue

• A task recurring every minute

Events live and execute their code within Managed Threads.

We use the to tell our application what events we want to subscribe to, and Perigee automatically manages the relevant threads for you.

Managed Threads

Threads are managed independent code blocks, completely separate from other threads. This is a key part of the application design process, as one thread can be shut down without affecting another thread. One thread can crash without taking down the application or other running threads.

Threads are managed by Perigee internally. This allows for the application to track its running state, restart them if they're down, and allows programmatic access to manage them.

Threads provide a clear separation of running code that does not allow one running block to cancel or crash another. By designing applications that use this separation of logic, we can create highly customizable, scalable, and fault-tolerant applications where a fault in one process doesn't take down other processes.

Threads are configured during the phase. We have some options during this startup, and we aren't forced to start threads immediately. They may be configured to start later or from a different event.

"Real World" example - PDF Document Processor

Imagine for a moment you have a directory watcher to read all the account PDFs from a folder and securely upload them to a third party for processing. Let's take a look at the power of managed threads and how they can help you and your business.

Inherently, you've got several points of failure and things to consider in this scenario:

• The drive the documents live on may fail

• The third party may be offline, or down

• We may need to perform maintenance on the server

• There may be an emergency call to stop processing documents

All of the above scenarios are easily handled within Perigee. Let's create a watcher to notify us of a new PDF on our drive.

Much of the difficult code is handled within the Directory Watcher. As we progress through the "Hello" series, we will be covering all of the fault-tolerant design patterns to follow. Let's take a look at a few of the mentioned issues above:

Have an emergency call to stop processing?

Maybe you need to take that processor down and perform maintenance. You can just shut down that thread, and leave the remainder of the application running and untouched!

The way you implement this logic is up to you. Maybe you decide to expose an internal, authorized endpoint to shut down or restart that process:

Have a critical, unhandled exception that crashes the entire thread?

Maybe the drive had a fault and is now offline, which caused the thread itself to go offline. Perigee implements several powerful ways to assist in the restart of this process. Another powerful feature of threads is that we're able to set up and configure how threads should react when they crash. Below is an example of adding the optional callback for restarting a down thread.

• Line 5 The first callback () => {} allows you to perform logic before telling Perigee to restart that thread. If you return true, the thread will be started again.

• Line 13 restartTimerDelay is to tell Perigee how long to wait until checking the callback again, if it returns false.

Need to shut down the whole application?

Simply press ctrl-c if you're within a console application and wait while Perigee gracefully cancels all running tasks. No data will be lost, and everything will be persisted to disk. You'll see the logs indicate what is currently running, and what is awaiting shutdown.

You may want to write custom logic to shut the app down gracefully as well. Check out the section to see how to cancel the parent token.

Events

Whether you are writing an integration, data pipeline, worker bot, synchronization agent, data process, or even an order transaction coordinator, they all have an event source. There are only two types of event sources. Let's look at each of them:

Triggered

Triggered event sources are used in most types of applications. They can be new accounts created in Salesforce, an email arriving in an inbox, or a new message in a queue. They act as a starting point for additional logic and are events created outside of our control. Here are a few more examples:

• A directory watcher receiving a new file.

• An email client getting an email.

• A Kafka topic receiving a new message.

• Salesforce pushtopic getting data.

Scheduled

Scheduled event sources are everything not triggered by external/internal sources. Typically these are CRON jobs, although Perigee has a few additional scheduling types to use. A few examples of using these:

• We want a data process to kick off at 4 AM.

• We need to check every hour for new data on the network drive.

• We pull an external API every 5 minutes for the status update of our request

Events callback

Now that we have a basic understanding of how event sources work, let's look at what that might look like inside Perigee.

These are just a few of the ways you can integrate different kinds of events inside Perigee. We'll cover all of them in examples and blueprints in more detail.

Every single one of the .Add ... () methods contain a callback action of when to process an event.

• For the CRON, this callback is when we're within the time and timezone to execute.

• For the IMAP Watch, it's when we receive a new email in the inbox.

• For the Directory Watch, it's when a new file is present and is no longer being written to

All of the callbacks within perigee contain a Cancellation Token and an ILogger. See the below example:

The Cancellation Token will get flagged cancelled when a shutdown event has occurred, and the process is expected to exit soon. Pay attention to this if you want to be able to gracefully shutdown, or cancel tasks in process. You can store your data or finish your process. If you're able to safely stop processing, you can respect it's request to shut down and end execution to return back to Perigee.

The ILogger is a Serilog enabled logging source, enabling you to write your logs to the console, files, external data sources, and many other sinks. , go check them out!

Perigee Application Callbacks

Line 2 - The initialization callback is how we inject and define our properties. Maybe our application requires a connection string, a custom property loader, or API credentials. This initialization block is how we inject additional configuration before the primary configuration block is called.

The init parameter is simply an IConfiguration. This allows full access to your appsettings.json file for reading any configuration values you need to set up additional components.

Line 5 - Perigee has a primary configuration callback defined every time you create an application. This block is where we define how and what the application does. More specifically, we define what event sources we use and any managed threads we want to create.

For configuring a Perigee application, we've created an easy-to-use SDK, this is why you get the taskConfig parameter, which is the .

For this section, let's walkthrough start to finish how you might design and build your companies specific application needs.

Here's the quick bullet list of every step of this process:

• Starting template

• Define Sources

• Requirements

Starting with a template

• Does our system need to respond to API requests?

  • Is data pushed to your service with POST requests?

  • Does it need to respond to a ping request?

If you answered "yes" to any of the above: lets start with this blueprint for built in hosting:

Otherwise we can create a standard .NET 6 project.

Purchase a license

To purchase a license, please visit .

Have questions, need more than 10 licenses or want to discuss volume discounts?

Teams requires a third party library and card content.

DEMO coming soon:

Perigee provides several easy methods to read Excel files into DataTables or DataSets.

//Get a memory stream of the object to read
using MemoryStream ms = new MemoryStream(File.ReadAllBytes("ExcelFile.xlsx"));

//Read "DataSheet"
var DT = ExcelReader.ReadDataTable(ms, "DataSheet", out DynamicDataTable ddt, TrimCells: true);

//Read two sheets, and return the data set
var dataSet = ExcelReader.ReadDataSet(ms, new List<string>() { "DataSheet", "Lookup" });

Resources

Our first application

Go ahead and create a new .NET 5/6+ console application. Open up Program.cs, and head to the first step to get started!

Step 1) The application starting point

Let's start with the basics. First let's create a new application in Program.cs. Depending on the version of .NET you started with, you'll see two different things when you open Program.cs

Net 6++

The .NET 6 ++ version looks like this:

If this is the case, delete Line 2 - Console.WriteLine("Hello, World!"); and start there.

Net 5 and lower

The .NET 5 and below versions looks like this:

If this is the case, start coding on Line 6.

The application code

Now that we know where to start, here's a simple Perigee application.

Let's look what what we have here in the constructor:

• Line 2 - "FirstApp" - The name of the application, this can be whatever you like!

• Line 3 - taskConfig block - This block is where we add any and all thread managed tasks.

Step 2) Adding thread managed code

This wouldn't be a hello tutorial without writing "Hello, World!", so let's add a method to Hello Perigee every 15 seconds!

The .AddCron method adds a new thread to the system. Each thread is managed by Perigee and is independent of every other thread. This is an important aspect of as it allows for individual thread management. Threads do not affect each other, and Perigee's internal thread management system has mechanisms in place to automatically restart downed threads.

• Line 4 - We use fluent syntax to .AddCRON() - This method takes a name, CRON string, and a callback(cancelToken, ILogger).

• Line 5 - We use the built in logger passed to us to log information in a templated format, passing in the name of the application to the logger.

Running the application produces a log new line every 15 seconds!

To close this application using graceful shutdown, press CTRL-C, it will start the safe shutdown procedure allowing the application to properly stop all running tasks before just exiting out.

Hello Perigee - Extended

My oh my, we logged something! 👍Exciting right? Let's take our first demo application a step further and watch for files, read them in, and report the row and data counts.

Simply replace .AddCron() with the instead, full code below:

Here's a sample.csv:

Voila! You're now watching for CSV files, reading them in, and reporting on the number of rows and columns.

For an intro to Perigee we accomplished quite a lot.

• We learned about - how they're added and independent of each other.

• We learned about information to the available sinks.

• strings and how to use a CRON thread.

Let's hop over to the next section and continue!

Logging Introduction

When building an application it is very important to have information communicated clearly about what is going on inside your application. Whether this is a success message or a critical failure that needs someones attention.

We use Serilog behind the scenes. So any configuration you can perform with Serilog, also applies here.

Log Sinks

Serilog can output it's data to all kinds of sources. The console, Azure, Amazon, Elastic, Log4Net, Loggly, NLog, SQL Server, file... just to name a few. These sources are called Log Sinks.

Any time a sink is added to the configuration, all logging within Perigee automatically sends it's log data to the configured sources. This simple configuration strategy makes it incredibly easy to send log data wherever you need it.

Configure logs

We recommend doing most of this configuration within the appsettings.json file, so let's take a look at configuring a simple logger.

In the we configured the console sink to have a different logging template. using the following section of code:

The first tag, MinimumLevel - This describes what events get emitted to the sinks. At Debug, anything logged at the Verbose level is ignored. It's great for controlling what types of messages are visible.

The seconds tag, WriteTo is the array of configured logging sinks. You can control these sinks with the various settings they accept and can be found on the Serilog Sink Configuration page for the given sink.

Configuration - With Hosting

The hosting configuration also includes a few additional keys, let's take a look here:

This time, the MinimumLevel is an object, not a string. This allows us to set the default logging level, as well as override the logging levels for several other namespaces.

The overrides supplied here will squash the incredible amount of information being logged from ASPNetCore hosting by default.

Perigee Logging

Logging is built throughout Perigee. Any thread that is managed and created is automatically sent a logger with the appropriate configurations.

Getting a logger directly

We can easily get a logger directly from the ThreadRegistry any time we want outside of a ManagedThread. This logger is tied into the Serilog system and will automatically inherit the configuration defined.

Log Scopes

As shown above, you can override the logging template with additional parameters. Here we've added the ThreadName to the logging template. The reason this works out of the box is because Perigee's internal thread management system injects this log scope before sending an ILogger back.

You can add as many custom log scopes as you need. They will appear differently for different log sinks. This is very helpful when adding a BatchID, CorrelationID, SystemID, NetworkID or other identifying information to the log request.

• If logging to a database with an additional column added to the logging table - any matching log scope will fill that column value

• If logging to something like Splunk, you'll see those log scopes present in the additional log data

• If an overriden template in the console logger includes the custom scope, it will be written when present

Scope demo

Make sure to setup your appsettings.json with the modified logging template:

Then add a simple recurring method and override the ThreadName supplied by Perigee:

The output will look like this:

Over the final few years in development we have worked closely with other companies to test, refine, and apply Perigee to their business needs. Here are a few testimonials of successfully implementing Perigee in their current production applications.

Parallel Transformation Task

With the assistance of Perigee we utilized the powerful set of parallel processing libraries to speed up our customer data transformation process by a little over %500,000 - from hours to microseconds. Perigee made this possible for us.

To register a specific RestSharp credential, just create that type and assign the appropriate Authenticator.

Any time you create a client, simply assign this credential authenticator:

The XML Converter is a useful class designed to transform a simple XML "List" into a structured list of classes. In the example provided, there are three test_scores elements nested within the data element. Utilizing the converter simplifies the process of reading and utilizing the relevant data effectively.

APIBuilder

This static class enables users to create an API specification using both custom specifications as well as specifications derived from Postman integration. Most notably, it provides a way to fully generate a complete Web API given an APISpecification.

From

A static class for serializing, compressing, and decompressing objects using JSON serialization and Brotli compression.

Compress

Serializes and compresses an object using JSON serialization and Brotli compression.

There is an included SmartSheets downloader for pulling data from SmartSheets.

The AccessToken is found in your API credentials.

To store and send email via an SMTP address, register the server at startup and use the AlertManager.Email() call anywhere in code later.

Code

Network Available

To check whether or not a network is available for use, use the NetworkUtility class. It returns true if there should be a network connectivity, and false if there should not be.

MoveToChildDirectory

This handy helper takes a file and a child directory and does the lookups, directory creation, and move operation

CreateDirectoryIfNotExists

TopoSort is a static class that provides functionality to perform a topological sort on a collection of items with dependencies.

Sort

This method takes a source collection of items and a function that returns the dependencies of each item. It returns a sorted list of items based on their dependencies.

To use the embedded discord client. Register the hook URI at start, then call anywhere else

Code

PerigeeApplication.ApplicationNoInit("Discord",  (c) => {

    //Register once
    AlertManager.RegisterDiscordWebhook("bot", "WEBHOOKURI");

    //Call anywhere else
    int IssueCount = 0;
    AlertManager.Webhook_Discord("bot", new DiscordMessage()
    {
        Embeds = new List<DiscordEmbed>() {
            new DiscordEmbed() {
                Title = $"My awesome discord enabled app {(IssueCount > 0 ? "failed" : "completed")}",
                Description = IssueCount > 0 ? "Issues exist, please check log." : "Everything completed as expected",
                Color = IssueCount > 0 ? System.Drawing.Color.Yellow : System.Drawing.Color.Green,
                Fields = new List<EmbedField> {

                    new EmbedField() { InLine = true, Name = "Server", Value = "MyServer"},
                    new EmbedField() { InLine = true, Name = "Issue Count", Value =  IssueCount.ToString() },
                    new EmbedField() { InLine = false, Name = "Location", Value = Directory.GetCurrentDirectory() },
                }
            }
        }
    });

});

There's also a builder method is you're creating the same information, it makes the creation a bit more compact and without having to remember the nested syntax:

AlertManager.Webhook_Discord("bot", DiscordMessage.Builder(
        $"My awesome discord enabled app {(IssueCount > 0 ? "failed" : "completed")}",
        IssueCount > 0 ? "Issues exist, please check log." : "Everything completed as expected", null,
        IssueCount > 0 ? System.Drawing.Color.Yellow : System.Drawing.Color.Green,
        EmbedField.Builder("Server", "MyServer"),
        EmbedField.Builder("Issue Count", IssueCount.ToString()),
        EmbedField.Builder("Location", Directory.GetCurrentDirectory())));

SalesForce Demo

To see the full demonstration of connecting to SalesForce, see the page dedicated to the SalesForce Connector

PerigeeApplication.ApplicationNoInit("SalesForce Demo", (c) => {

    //Replace the details here
    var x509 = new X509Certificate2();
    
    c.AddSalesForceWatch<SFAccount>("WatchAccounts", "Account", "ConsumerKey", "ConsumerSecret, "User", x509, "login", 
    (ct, log, updated, deleted) => {

        //updated records, bound back to an SFAccount class
        foreach (var item in updated)
        {
            Console.WriteLine($"[{item.id}] {item.AccountName}({item.Type})");
        }

        //Deleted ID's, along with the deleted date
        foreach (var item in deleted)
        {
            Console.WriteLine($"{item.id} - {item.deletedDate:G}");
        }

    });

});

In this super simple example, we'll create and submit a request to gpt-5.1, in only a few lines of code.

//Declare a client
var aic = new OpenAIClient(apiKey);

//Get a response
var response = await aic.GetResponseAsync(
   OpenAIClient.GPT51("It's this easy to message AI??")
   .WithInstructions("You're a helpful assistant, respond kindly and to the point"));

//Use the built in MessageContent Property to retrieve the latest message
if (response.IsSuccessful)
    Console.WriteLine(response.Data.MessageContent);

Perigee Licensing Agreement

SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Licensing production and development use

To obtain a valid license after the trial period, visit to purchase and view license terms and conditions.

Third Party licensing agreements

Dapper, Serilog, Serilog.AspNetCore, Google.Apis.Auth, Google.Apis.Gmail.v1, RestSharp

The packages listed are licenced under Apache 2.0: http://www.apache.org/licenses/LICENSE-2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Ude.NET

The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/

Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.

CometD.NetCore2, Bogus, BrotliSharpLib, Newtonsoft.json, Newtonsoft.json.schema, Portable.BouncyCastle, ExcelDataReader, HtmlAgilityPack, MailKit, NetCoreForce.Client, Pluralize.NET, SharpZipLib

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The Directory Watch is able to monitor a folder and report back when a new file is present. It has several key checks in place to make sure the file isn't actively being written too, or locked by another application before executing the file ready callback.

Directory Watch expects that the file will be removed from the directory after it has been processed. For this reason there are several options on the DirectoryWatchFailurePolicy on how to handle when a file hasn't been removed.

Demo Application

This demo will:

• Watch the C:\Watch folder

• It will report on any CSV files (*.csv)

• It will search AllDirectories (subDirectoties) as well as it's root

As a bonus, we'll in and report on it's load.

Dummy sample file used here:

When the above example run's against this file, it will log:

Failure Policies

At the moment, Perigee contains 3 different failure policies:

• Redelivery after a certain period of time

• Move to Failed Folder (_Failed)

• Delete removes the file from the system.

Application initialization

Both application starting points are nearly identical. The full callback has an IPC token and the initialize callback. Both are described in Hello Perigee.

Application(string appName, string IPCCancelToken, Action<IConfiguration> initialize, Action<ThreadRegistry> taskConfiguration, CancellationTokenSource cancelSource = null, Task HostTask = null, string[] CommandLineArguments = null)
ApplicationNoInit(string appName, Action<ThreadRegistry> taskConfiguration, CancellationTokenSource cancelSource = null, Task HostTask = null, string[] CommandLineArguments = null)

You may also run the ApplicationNoInit as an async method as well:

Initialize outside a PerigeeApplication

Sometimes you need to initialize outside of the scope of a PerigeeApplication. We do this for API applications, blazor applications, and other scenarios where we need to call something before Perigee.

Host Statistics

To get information about the host it's running on, there is a single method:

The response class, HostStatistics contains properties for the machine it's running on including machine name, drive info, cpu time and running directory.

Exit event

To get an exit callback, use the helper method supplied.

After return is executed and the domain/service is about to unload, OnExit has a few seconds to perform any last tasks, like persisting data.

Perigee currently supports two different SMS clients right out of the box. It's easy to register them and use them from anywhere within your application.

1. At the start of your Perigee application, register the related alert managers.

2. Anywhere from the code later on, call AlertManager.SMS() and supply the

   • Name (The name of the account you registered)

   • ToNumber (Who it is going to, if within the US, start with the digit 1)

   • Message Body (The SMS body content)

Code

Debounce Class

The Debounce class is a utility class in C# used to prevent the rapid firing of events. Particularly useful in scenarios where an action should only be performed after an event has ceased to occur for a specified duration, like key-up events on textboxes that trigger searches.

Bounce Method

The Bounce method is called whenever an event occurs. It resets the debounce timeout. If this method is not called again within that timeout, the action specified in the Debounce constructor will fire. The Bounce method can optionally take a parameter, which will be passed to the action when it fires.

Example:

In the first example, the Bounce method is called without a parameter, so when the action fires it won't receive any parameters. In the second example, the Bounce method is called with an integer parameter, so the action will receive the last integer passed when it fires.

Disposing

When disposing the Debounce class that doesn't take any generic parameters, no additional callbacks will be fired.

When disposing the Debounce<T> class, there is an immediate one time fire of the bounce callback that has the latest value supplied. This is particularly useful when you need to dispose of a resource quickly, but still need to access the latest value in the pipeline without waiting on the bounce to fire.

It's very easy to add custom providers. Remember that .NET creates a ConfigurationSource that implements and returns a ConfigurationProvider.

You can see this pattern here: .

Custom Implementation

To implement a custom source and provider only requires a single line of code.

Introduction

API Generation empowers developers to accelerate the development process of setting up a new API significantly. Whether you're creating an API for the first time or you're a seasoned veteran, crafting an API is more straightforward with the tools provided by Perigee.

API Generation refers to the creation of an API from a source document, such as Postman, or directly in code using the fluent syntax, without the in-depth knowledge required to configure a new API from scratch. It might seem too good to be true...

Thread Conditional Wrap

There are times when you want to turn off or disable something running when a condition occurs. An example of this would be to only run a Coordinator when there is a known network connection available.

This is made possible by a ThreadCondition:

Let's take a look at what's happening:

Data Load

Need to take your Excel data and load it into your database to get the analysts started?

Instead of taking weeks to develop a new data pipeline, create structure, and work with the business to load your new client data in, just drop the file in Perigee. We will take care of everything from finding and getting the data loaded, to creating a table in only a few seconds. You’ll be a hero!

A helper class when dealing with a range of data or cells

Instantiate

To create a new range, supply two initial values:

• As you can see it corrected the range by fixing the starting point to

In this simple example, We show how to load in a source file, split the name column into it's first and surnames, then re-save the file.

Block Until

This handy method takes a CRON string and is an easy way of creating an awaitable task. It returns True if the await finished without being cancelled by the optional token.

//Async wait until the 0th second of every 5th minute
await PerigeeUtil.BlockUntil("0 */5 * * * *");

//Or for non async methods
PerigeeUtil.BlockUntil("0 */5 * * * *").GetAwaiter().GetResult();

//Block until the next 5 minute mark, or 10 seconds. Whichever comes first
CancellationTokenSource CTSource = new CancellationTokenSource();
CTSource.CancelAfter(TimeSpan.FromSeconds(10));
var done = PerigeeUtil.BlockUntil("*/5 * * * *", null, CTSource.Token).GetAwaiter().GetResult();

Retry and Rety Async

One of the utility class methods is a Retryand RetryAsync. They provide an easy ability to retry on exception thrown with a delay.

The response from these classes is a Tuple where

• Item1 is a boolean indicating if it was a success(true), or failed the maximum number of times (false).

• Item2 is the exception thrown if it was not a succeed

RetryExponentialBackoff

Another retry strategy is the Exponential Backoff method. Unlike a constant delay between retries, this method progressively increases the delay (incorporating jitter) to prevent overwhelming a remote system with requests.

You can define both a base delay (in milliseconds) and a maximum delay (in milliseconds) to ensure the delay between requests scales appropriately.

Range Extraction

Range extraction allows you to split an input string into it's parts.

Fast String

These methods use the fastest possible comparisons and Boolean checks available for the various tasks.

String manipulation

Data and object or partial detection

Metrics Classes

There are two metrics classes currently, MetricDecimal and MetricInteger. They work identically and only store their respective types internally.

MetricDecimal

Metrics are set of classes that implements a metric collection for values. It provides functionality to add metric values and compute various statistics such as average, median, percentiles, minimum, maximum, and sum. In addition, it supports serialization and deserialization of the metric data to and from JSON formats.

Properties

MetricName

Represents the name of the metric. This property is used to identify the metric collection.

MetricType

Indicates the type of the metric. For this class, it is set to "Decimal".

Methods

AddMetric(decimal metric, int precision)

Adds a new metric value to the collection after rounding it to the specified number of decimal places. In addition to updating the internal bucket of values, this method updates the maximum, minimum, and sum accordingly.

Parameters:

• metric: The decimal value to add.

• precision: The number of decimal places used for rounding the metric value.

AddMetric(decimal metric)

Adds a new metric value to the collection after rounding it to 2 decimal places. This method updates the internal bucket as well as the maximum, minimum, and cumulative sum values.

Parameters:

• metric: The decimal value to add.

Average()

Calculates and returns the average of the metric values. The average is computed by dividing the cumulative sum of metric values by the total count of entries in the collection.

Max()

Returns the maximum metric value that has been added to the collection.

Median()

Computes and returns the median of the collected metric values. This method identifies the middle value by using the frequencies from the bucket of values. In cases where the cumulative frequency exactly equals the midpoint, the previous value is used.

Min()

Returns the minimum metric value recorded in the collection.

Percentile(decimal k, bool Interpolate = false)

Calculates the kth percentile of the recorded metric values. If k is 50, the method delegates to the Median() calculation. For other percentiles, it determines the appropriate index within the ordered set of unique metric values. If the computed index is a whole number, it returns the average of the two adjacent values; otherwise, it may compute an interpolated result based on the Interpolate parameter or return the ceiling value.

Parameters:

• k: The percentile value to compute (e.g., 25, 50, 75).

• Interpolate (optional): A boolean flag that, if true, forces interpolation between adjacent values. The default value is false.

PercentileRank(decimal k)

Calculates the percentile rank of a specific metric value k. The percentile rank is computed with the formula:

(cf_l + 0.5 f_i) / N * 100%

where cf_l is the cumulative frequency of values less than k, f_i is the frequency of the value k, and N is the total count of all entries.

Parameters:

• k: The metric value for which the percentile rank is to be determined.

Sum()

Returns the cumulative sum of all metric values added to the collection.

FromJson(string json)

Reconstructs the MetricDecimal instance from a JSON string representation of the metric data. This method updates the sum, minimum, maximum, and bucket of frequencies based on the JSON content.

Parameters:

• json: A JSON formatted string representing the metric data.

AsJson(bool valuesOnly = true)

Serializes the current state of metric data into a JSON string. Basic statistical information such as sum, min, max, average, median, and count are always included. If the valuesOnly parameter is false, then the full bucket containing all metric frequencies is also included.

Parameters:

• valuesOnly (optional): A boolean flag indicating whether only basic stats should be serialized (true by default).

Returns:

• A JSON formatted string representing the metric data.

AsJObject(bool valuesOnly = true)

Serializes the metric data into a JObject. Similar to AsJson, it always includes the basic statistical information. When the valuesOnly parameter is false, the complete bucket data is also inserted into the JObject.

Parameters:

• valuesOnly (optional): A boolean flag indicating whether only basic stats should be included in the JObject (true by default).

Returns:

• A JObject containing the metric data.

Example:

ParallelProcessToDictionary (DataTable)

This extension method processes a DataTable in parallel and returns a ConcurrentDictionary containing the results, with the keys generated by the provided callback.

Example:

ParallelProcessToDictionary (IEnumerable)

This extension method processes an IEnumerable in parallel and returns a ConcurrentDictionary containing the results, with the keys generated by the provided callback.

Example:

ParallelProcessToGroupProcessor

This extension method processes an IEnumerable in parallel to a new grouped processor.

Example:

ParallelProcessToSingleProcessor

Converts an IEnumerable to a SingleProcessor using a provided callback function.

Example:

ParallelProcessToBag

This extension method processes an IEnumerable in parallel and returns a ConcurrentBag containing the transformed items, with the transformation function provided by the callback.

Example:

ParallelProcessToBag (DataTable)

This extension method processes a DataTable in parallel and returns a ConcurrentBag containing the transformed items, with the transformation function provided by the callback.

Example:

ParallelProcessToBag (with ExceptionRows)

This extension method processes an IEnumerable in parallel, returns a ConcurrentBag containing the transformed items, and handles exceptions using the provided callback.

Example:

ParallelProcess (ConcurrentBag)

This extension method processes a ConcurrentBag in parallel, invoking the provided callback for each item.

Example:

ParallelProcess (ConcurrentDictionary)

This extension method processes a ConcurrentDictionary in parallel, invoking the provided callback for each key-value pair.

Example:

This class provides functionality for downloading data from a database and storing the result in a provided class. It supports multithreaded asynchronous calls, and allows for a queue of threaded download operations. It has automatic retry capability built in.

The type of DB Connection passed determines the database language it uses. You could for example provide a MySql connection instead.

Quick implementation

Downloader Extension

There is an easy way to create the DBDownloader by using the static extension method.

Assign / AssignSingle

Assigns the downloaded data to a property of the base object. Can be called multiple times using fluent syntax to fill multiple properties. The downloader automatically manages the multithreaded and asyhcnronous operations for filling each property.

It uses behind the scenes to perform the mapping and query execution. If you want to supply parameters, you can use the objectParam parameter.

If you're not assigning a list, but a single object, use AssignSingle().

Example:

Wait

Waits for all download tasks to complete. This can be called inline or later right before the downloaded data is required. It is thread blocking.

Example:

GetDownloadTime

Gets the total time taken for all download tasks to complete.

Example:

Downloads

You can get to the list of tasks created by accessing this property.

Every DownloadResult has properties for the success boolean, if an exception was thrown, the name of the property it set, and dates to indicate start and end times.

Let's take a look at how you might extend Perigee to fit your companies very specific needs. Although we ship with many different connectors, watchers, agents, and synchronization features you may want to add something to the Perigee core system. It is very easy to implement custom threads. This is one the reasons Perigee is so powerful as we don't lock you out of adding and customizing the application.

There are two types of custom threads you can add where both have different execution rules as they allow you create any additional functionality needed for your specific requirements.

Managed Thread

The first type is a managed thread. The ManagedThread calls the callback method when it is started, and doesn't expect that method to return, throw an exception, or exit until the

Let's take a look at what sources we may use and create the configuration, credentials, and refreshes we need to use.

• Do we want to put connection strings in the appsettings.json and read them?

  • Create the json sections and keys

Let's say your web app allows users to schedule reports to email off. These items are configurable and the user can add or remove them at any time.

• The front-end web app saves a database record back with all the information needed to run the specific report and it's parameters.

• Then the reads the table for those database records every few minutes and updates, removes, reschedules any changed items.

An Expression Managed Thread is called every N seconds and is expected to exit the method call. It will repeat until the application is shut down or the thread is turned off.

You can easily register hot-reloadable connection strings by registering them with the helper methods:

PerigeeApplication.ApplicationNoInit("ConnectionStrings", (c) => {

    //Register the connection string
    c.RegisterConnectionString("DB", "devServer");
    
    //Get the connection string later...
    string connectionString = CredentialStore.GetCredential_ConnectionString("DB");
    
    //Or get the credential, and cast it to retrieve
    if (CredentialStore.GetCredential("DB") is ConnectionStringCredentialStoreItem m) {
        var connectionStringFromCred = m.ConnectionString;
    }
});

The AppSettings.json file:

"ConnectionStrings": {
      "devServer": "data source=host;initial catalog=database; User Id=user; Password=abc"
}

Resources

Intro

Lets take a closer look at how the configuration works within Perigee.

To understand the configuration process we need to understand how the sections work and why we separate them. To keep it concise, sections create a logical separation of values for the different processes in our application.

Configuration sections can be registered in our application in a number of ways including but not limited to:

• Environment variables

• main(string[] args)

• appsettings.json file

Back in the we used a configuration file. A section is a key off of the root json node and as you can see there are 4 different sections in our startup file.

Demo Application

Our goal is to write an application that logs out our custom configuration sections. We will do so by reading the configuration directly at runtime as well as binding the configuration section to a custom class.

You will also see some helpful ways you can plug the configuration system into

Configuration Direct Read

To start, we need to add another section to our appsettings.json file called HelloConfig and give it some values to read.

This creates a section for us to read in our code. We will read it directly by calling taskConfig.GetValue<>().

This handy method is a quick way to reference ALL of the incoming configuration providers and it works by supplying two things:

1. The C# Type - This could be a string, or int, or whatever other value type you're reading.

2. A special format: Section:Key.

To read the Name from our configuration section the Type would be string and our Section:Key would be: HelloConfig:Name

Here's a fully working Perigee application

• Line 1 - This is a simplified Perigee start method that skips IPC tokens and initialization actions.

• Line 3 - AddRecurring simply adds a managed thread that is automatically called every n milliseconds (supplied by an optional parameter, default 5000 (5 seconds))

• Line 5 - We log out the configuration value reading it live at runtime.

You'll see below the results of running this application. We get a log every 5 seconds with the value captured from our configuration file.

Configuration Binding

Next let's look at binding the section we added to a custom class. Here's the file if you prefer to import it.

The class simply has 3 properties that are an exact case sensitive name match to the configuration section we created.

To bind a class, simply use the taskConfig and call GetConfigurationAs<>():

• The Generic T Parmater, HelloConfig, is the class we're creating and assigning. It's a type reference.

• The string, "HelloConfig", is what section to get from our configuration so that we can assign it's properties to the referenced type.

This is all that is needed to bind our configuration section to a class. Here's the full application up to this point!

Line 10 - We got the configuration as a custom class by binding to it.

Line 11 - We use the custom classes properties to write to our log.

Link to Config

Perigee will, by default, hot-reload the appsettings.json files. This is a fantastic way to give control to a configuration file to enable or disable a thread without shutting down the entire application.

Perigee has a great helper method built into it which enables the thread-to-configuration linking for enabling or disabling managed threads through a configuration boolean.

Our example config section we added (HelloConfig) which has a key called Enabled, and we're going to use that Boolean to tell Perigee whether or not to start or stop our thread.

Line 12 - We added started: false to the methods optional parameter so that with the addition of the linking command, it will be in charge of starting or stopping this thread. LinkToConfig was added and we supplied our Section:Key to it.

Now if you run the application and navigate over to the debug folder to open up the appsettings.json file, you can change that HelloConfig:Enabled value from true to false or false to true and watch Perigee start and gracefully stop that thread 🧙 🪄

Change Notification

If you'd prefer more control over how to start or stop your threads, or even make different decisions based on the configuration changes, you can always subscribe the change notification directly. .LinkToConfig() is essentially the same code as below but managed by Perigee internally and automatically, saving you from a few lines of code.

Configuring the logging

Perigee will automatically add an additional config property block to each of it's loggers that it passes to the managed threads. This property block is called ThreadName. As a fun aside, let's update our logging template to include this property block.

Open the appsettings.json file and under the Serilog section, update the console sink to include the updated outputTemplate.

Now our application's logs have the name of the managed thread they are coming from right in the log!

You can add your own properties to the loggers, we'll show you how on the page.

Secure Values

Perigee has several ways built in that allow you to encrypt your configuration values which may provide an extra layer of security/obscurity. It uses AES256 bit encryption on the values for secure encrypting and decrypting values and has built in methods for reading them.

To secure a value you must provide the encryption key, encryption IV, and the source of where to find these values.

There are 3 sources for the Encryption keys

• Environment Variable source

• Arguments Source

• Variable Source

Here's an example of setting this up all within code using the variable source. The methods are exactly the same though if you decide to pass in the encryption key from an argument, or read it from an environment variable.

With that done you should see something like this printed to the console:

These are your two encryption keys, and we will use them to decrypt values in the next step. Remember you can easily put these in the environment variables, command line arguments, or even set the variables in code. For the ease of this demonstration, we'll assign them directly:

If you're reading encrypted configuration values, there's a handy little helper than does the read and decrypt in one pass:

Summary

Hopefully you understand a bit better about how configuration, sections, and providers work as well as how to use them in code.

MatchToList

This handy method uses several de-duplication methods, as well as the Soundex and Levenshtein algorithms to produce a best-guess match between two different lists.

It will always match A => B. Meaning items that items in A will only attempt matches to items in B, not the other way around.

In the below example, Peech is matched to PeachJuice even though it's misspelled and Agave has no match. Every other item is matched with it's equivalent juice.

MatchToList (Dictionary)

This version of list matching is almost identical to what is listed above, the main difference is that your input dictionary defines the keys to match, and the value is a comma separated list of additional "hints" to match by.

In this example, we can match the key Apple with Appljuise, because of the additional hints supplied by the dictionary.

MatchToList_Quality

If you're attempting to debug your list matching and want to understand the match values being returned by this, use the _Quality function. It will return a rich object that is easy to read showing the matched value for each key.

JoinBy

Takes every item in a list and joins it with the delimiter

DistinctBy

Works identically to the classic LINQ Distinct, however it works on classes as well.

ToDataTable

Sometimes you have a list of classes, and need to convert it to a table. This uses reflection to perform the conversion.

WhereSubclassOfType<T>

Enable you to filter a list of classes that conform to the subclass of another class

Flatten<T,R>

Recursively flatten a data structure and return the new type

FromHierarchy<T>

Recursively select down a tree until something is null. Great for returning a list of all inner exceptions in order.

RemoveDuplication

Our first example was a bunch of fruit juices. The remove duplication method attempts to find any common pattern in the list of items and remove them.

Multi-Threaded Processor

Multi-Threaded Processor allows you to efficiently process data in parallel across multiple threads, using both simple and advanced approaches for scatter-gather operations.

Unlike a traditional "scatter-gather", MTP allows you to separate the loading and processing across threads and then await the results. This is incredibly powerful when the loading is done from a remote source where latency is involved.

An Example

SDK

ParallelProcessMultiThread

Processes a collection of items in parallel using a specified processing function.

Parameters:

• items (IEnumerable<TIn>): The collection of items to process.

• Process (Func<TIn, TOut>): The function to process each item.

• cancelToken (CancellationToken?

Returns:

• List<MultiProcessResult<TIn, TOut>>: A list of processing results.

Example:

Enqueue

Enqueue an item to be processed.

Parameters:

• item (TIn): The item to be processed.

Example:

AwaitProcessed

Await until all pending executions have been processed. Optionally accepts a cancellation token or a time span to limit the wait.

Example:

Class Reference

MultiProcessResult<TIn, TOut>

A class that represents the result of processing an individual item.

InData

The input data that was processed.

OutData

The output data produced by the process.

Exception

An optional exception that occurred during processing.

ProcessTime

An optional timespan representing the processing time.

Grab the code from the blueprint section for API hosting with Perigee

Admin controller example

In this example we create a controller that allows us to turn threads on and off.

1. Simply add a new empty "API Controller" to the Controllers folder.

2. Call it AdminController.cs

3. Paste the code below in to replace the content

In this example we've created a simple endpoint we can call to turn anything on or off by simply checking the thread registry for the requested thread, then turning it on or off.

The port may be different - but you can see how easy it would be to turn something on or off!

[GET] https://localhost:5001/api/admin?task=ProcessAgent&start=true

Resources

Sync Agents

Sync agents are a more powerful and configurable ways of defining tasks that must perform under a given sequence, dependency tree, or schedule. They can be configured in a number of ways that allow for very complex and fine grain control over how and when they execute.

Nested Sets provide a fantastic way of traversing hierarchical data. Once processed, they are able to very rapidly walk large data trees.

Suppose you're given a list of products and are asked:

• "What products does the clothing category contain?"

• "What products are my siblings?"

The Directory Notifier is able to monitor a folder and report back when a new file is present. It has several key checks in place to make sure the file isn't actively being written too, or locked by another application before executing the file ready callback.

Unlike the that expects the file to be processed out of the folder, notifier is intended to only signal that a file has been changed, added, or removed. It does not contain a failure policy because of this reason.

If you are having issues with the credential store, please verify a couple of things. More than likely, it's one of the following issues:

Pathing

Verify that you are not setting the current working directory to something outside of the executable while the Perigee is initializing.

A typical load cycle for a Perigee Application looks like this:

This verifies that the current working directory and pathing is correct. This will load the credential file under $WorkingDirectory/credentialStore/credentials.[pcb|pce]

This example uses a file watcher to scan for CSV's and parse them into a DataTable. It prints the info about it.

The IMAP bot will scan an email box and respond with the text you sent it, as well as info about attachments.

For more information about the imap client, see it's page linked below!

Demo Code

To run the demo, you would need to authenticate it properly with your email server. This demo is using SASL-OAUTH for Google GMail. If you want to supply direct username/password authentication you can do that as well.

PerigeeApplication.ApplicationNoInit("SASL Google", (c) => {

    var SASLGoogle = MailWatcher.SASL_GoogleAPIS("[email protected]", "mailbot_google_auth.json");
    c.AddIMAPWatcher("EmailBot", "[email protected]", "FromMe", "smtp.gmail.com", 587, "smtp.gmail.com", 993, () => SASLGoogle, (ct, l, mail) => {

        try
        {
            if (!mail.IsAnswered)
            {
                var From = mail.FromAddresses().FirstOrDefault()?.Name ?? "";
                var SaidWhat = mail.GetOnlyInputTextBody();
                var attachments = mail.Message.Attachments.Count();

                mail.Reply(false, (b) => {

                    b.HtmlBody = $"Hello <b>{From}</b><br />  You said: {SaidWhat}<br />Attachments: {attachments}";
                    b.TextBody = $"Hello {From}\r\n  You said: {SaidWhat}\r\nAttachments: {attachments}";

                });

                mail.AddFlags(MailKit.MessageFlags.Answered | MailKit.MessageFlags.Seen);
                mail.AddLabels("bot");
            }
        }
        catch (Exception ex)
        {
            l.LogError(ex, "Uncaught exception in mail processor");
            try
            {
                mail.AddFlags(MessageFlags.Answered | MailKit.MessageFlags.Seen);
                mail.AddLabels("error");
            } catch (Exception) { }
        }

    });
});

Another integration pattern is limiting the number of calls that can be made during a given period of time. This is important when rate limiting or billing may be an issue and is a perfect stop gap for a critical failure causing an erroneous number of API calls or running up an unwanted bill!

The Limiter auto resets it's count on the defined TimeSpan, and will not allow additional executions past the defined limit.

using Perigee;
using Perigee.Extensions;

PerigeeApplication.ApplicationNoInit("Limiter", (c) =>
{

    c.AddRecurring("Limiter", (ct, l) => {
    using var APILimit = new RateLimiter("RateLimiter.json", 2, TimeSpan.FromMinutes(1), ct);

    if (APILimit.CanExecute())
    {
        //Perform action
        l.LogInformation("Calling API {n}/{x} executions", APILimit.GetCount(), APILimit.GetLimit());
    }
    else
    {
        l.LogInformation("Out of executions...");
    }
});
});

/*
 [16:35:07 INF] Calling API 1/2 executions
 [16:35:13 INF] Calling API 2/2 executions
 [16:35:18 INF] Out of executions...
 */