Anatomy of a Perigee Application

Perigee Applications are created around two main concepts: Threads and Events.

Events are the source for starting a process:

• An API call

• An email received

• A scheduled task at a specific date/time

• A file dropped on disk

• A new message from a queue

• A task recurring every minute

Events live and execute their code within Managed Threads.

We use the Perigee Application Callbacks to tell our application what events we want to subscribe to, and Perigee automatically manages the relevant threads for you.

Managed Threads

Threads are managed independent code blocks, completely separate from other threads. This is a key part of the application design process, as one thread can be shut down without affecting another thread. One thread can crash without taking down the application or other running threads.

Threads are managed by Perigee internally. This allows for the application to track its running state, restart them if they're down, and allows programmatic access to manage them.

Threads provide a clear separation of running code that does not allow one running block to cancel or crash another. By designing applications that use this separation of logic, we can create highly customizable, scalable, and fault-tolerant applications where a fault in one process doesn't take down other processes.

Threads are configured during the Perigee Application Callbacks phase. We have some options during this startup, and we aren't forced to start threads immediately. They may be configured to start later or from a different event.

"Real World" example - PDF Document Processor

Imagine for a moment you have a directory watcher to read all the account PDFs from a folder and securely upload them to a third party for processing. Let's take a look at the power of managed threads and how they can help you and your business.

Inherently, you've got several points of failure and things to consider in this scenario:

• The drive the documents live on may fail

• The third party may be offline, or down

• We may need to perform maintenance on the server

• There may be an emergency call to stop processing documents

All of the above scenarios are easily handled within Perigee. Let's create a watcher to notify us of a new PDF on our drive.

taskConfig.AddDirectoryWatch("DocumentProcessor", "C:\\PDFs", "*.pdf", SearchOption.TopDirectoryOnly, 
(ct, log, path) => {
    log.LogInformation("New file ready for process {file}", Path.GetFileName(path));
});

Much of the difficult code is handled within the Directory Watcher. As we progress through the "Hello" series, we will be covering all of the fault-tolerant design patterns to follow. Let's take a look at a few of the mentioned issues above:

Have an emergency call to stop processing?

Maybe you need to take that processor down and perform maintenance. You can just shut down that thread, and leave the remainder of the application running and untouched!

//Is it running?
taskConfig.IsRunning("DocumentProcessor");

//Ask it to stop
taskConfig.QueueStop("DocumentProcessor");

The way you implement this logic is up to you. Maybe you decide to expose an internal, authorized endpoint to shut down or restart that process:

[HttpGet]
[Authorize]
public IActionResult Admin_StopDocumentProcessing()
{
    if (Perigee.IsRunning("DocumentProcessor")) Perigee.QueueStop("DocumentProcessor");
    return Ok();
}

Have a critical, unhandled exception that crashes the entire thread?

Maybe the drive had a fault and is now offline, which caused the thread itself to go offline. Perigee implements several powerful ways to assist in the restart of this process. Another powerful feature of threads is that we're able to set up and configure how threads should react when they crash. Below is an example of adding the optional callback for restarting a down thread.

taskConfig.AddDirectoryWatch("DocumentProcessor", 
(ct, log, path) => { },

//Restart function and paramaters //
() => { 
    //Check the hard drive?
    
    //Check the connectivity to the third party processor?
    
    //All look good? return true
    return true; 
}, 
restartTimerDelayMS: 5000, 
shutDownWaitMS: 300000,
// ----------------------------- //

started: true);

• Line 5 The first callback () => {} allows you to perform logic before telling Perigee to restart that thread. If you return true, the thread will be started again.

• Line 13 restartTimerDelay is to tell Perigee how long to wait until checking the callback again, if it returns false.

• Line 14 shutdownWaitMS is how long Perigee will wait until dropping the thread when attempting to cancel it.

• Line 17 started is an optional way to tell Perigee only to configure and store the thread, but do not start it yet. You have control over how to start this later.

Need to shut down the whole application?

Simply press ctrl-c if you're within a console application and wait while Perigee gracefully cancels all running tasks. No data will be lost, and everything will be persisted to disk. You'll see the logs indicate what is currently running, and what is awaiting shutdown.

You may want to write custom logic to shut the app down gracefully as well. Check out the Thread Registry section to see how to cancel the parent token.

Events

Whether you are writing an integration, data pipeline, worker bot, synchronization agent, data process, or even an order transaction coordinator, they all have an event source. There are only two types of event sources. Let's look at each of them:

Triggered

Triggered event sources are used in most types of applications. They can be new accounts created in Salesforce, an email arriving in an inbox, or a new message in a queue. They act as a starting point for additional logic and are events created outside of our control. Here are a few more examples:

• A directory watcher receiving a new file.

• An email client getting an email.

• A Kafka topic receiving a new message.

• Salesforce pushtopic getting data.

• An API with JSON data posted to it.

Scheduled

Scheduled event sources are everything not triggered by external/internal sources. Typically these are CRON jobs, although Perigee has a few additional scheduling types to use. A few examples of using these:

• We want a data process to kick off at 4 AM.

• We need to check every hour for new data on the network drive.

• We pull an external API every 5 minutes for the status update of our request

Events callback

Now that we have a basic understanding of how event sources work, let's look at what that might look like inside Perigee.

PerigeeApplication.Application("Sync Agent", "IPCSingleRunToken", (init) => {}, 
            (taskConfig) => 
            { 
                taskConfig.AddDirectoryWatch("C:\WatchMeMaybe") //Watch a directory
                .AddCRON("0 */2 * * *")  //Run every other hour
                .AddIMAPWatcher() //Watch an IMAP inbox
                .AddSalesForceClientPush() //Watch a salesforce pushtopic
                .Add() //Add your own thread managed method
            });

These are just a few of the ways you can integrate different kinds of events inside Perigee. We'll cover all of them in examples and blueprints in more detail.

Every single one of the .Add ... () methods contain a callback action of when to process an event.

• For the CRON, this callback is when we're within the time and timezone to execute.

• For the IMAP Watch, it's when we receive a new email in the inbox.

• For the Directory Watch, it's when a new file is present and is no longer being written to

All of the callbacks within perigee contain a Cancellation Token and an ILogger. See the below example:

taskConfig.Add("ManagedThreadA",
(cancelToken, logger) => { 

});

The Cancellation Token will get flagged cancelled when a shutdown event has occurred, and the process is expected to exit soon. Pay attention to this if you want to be able to gracefully shutdown, or cancel tasks in process. You can store your data or finish your process. If you're able to safely stop processing, you can respect it's request to shut down and end execution to return back to Perigee.

The ILogger is a Serilog enabled logging source, enabling you to write your logs to the console, files, external data sources, and many other sinks. Serilog is awesome, go check them out!

Perigee Application Callbacks

PerigeeApplication.Application("FirstApp", "IPCSingleRunToken", 
(init) => { 
   
}, 
(taskConfig) => { 
       
});#

Line 2 - The initialization callback is how we inject and define our properties. Maybe our application requires a connection string, a custom property loader, or API credentials. This initialization block is how we inject additional configuration before the primary configuration block is called.

The init parameter is simply an IConfiguration. This allows full access to your appsettings.json file for reading any configuration values you need to set up additional components.

Line 5 - Perigee has a primary configuration callback defined every time you create an application. This block is where we define how and what the application does. More specifically, we define what event sources we use and any managed threads we want to create.

For configuring a Perigee application, we've created an easy-to-use SDK, this is why you get the taskConfig parameter, which is the ThreadRegistry.

Logging Introduction

When building an application it is very important to have information communicated clearly about what is going on inside your application. Whether this is a success message or a critical failure that needs someones attention.

We use Serilog behind the scenes. So any configuration you can perform with Serilog, also applies here.

Log Sinks

Serilog can output it's data to all kinds of sources. The console, Azure, Amazon, Elastic, Log4Net, Loggly, NLog, SQL Server, file... just to name a few. These sources are called Log Sinks.

Any time a sink is added to the configuration, all logging within Perigee automatically sends it's log data to the configured sources. This simple configuration strategy makes it incredibly easy to send log data wherever you need it.

Configure logs

We recommend doing most of this configuration within the appsettings.json file, so let's take a look at configuring a simple logger.

In the Hello Configuration Section we configured the console sink to have a different logging template. using the following section of code:

{
    "Serilog": {
        "MinimumLevel": "Debug",
        "WriteTo": [
            {
                "Name": "Console",
                "Args": {
                    "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}]({ThreadName}) {Message:lj}{NewLine}{Exception}"
                }
            }
        ]
    }
}

The first tag, MinimumLevel - This describes what events get emitted to the sinks. At Debug, anything logged at the Verbose level is ignored. It's great for controlling what types of messages are visible.

The seconds tag, WriteTo is the array of configured logging sinks. You can control these sinks with the various settings they accept and can be found on the Serilog Sink Configuration page for the given sink.

{ "Name": "MSSqlServer", ...}

Configuration - With Hosting

The hosting configuration also includes a few additional keys, let's take a look here:

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning",
        "Microsoft.AspNetCore": "Warning"
      }
    },
    "WriteTo": [
      { "Name": "Console" }
    ]
  }
}

This time, the MinimumLevel is an object, not a string. This allows us to set the default logging level, as well as override the logging levels for several other namespaces.

The overrides supplied here will squash the incredible amount of information being logged from ASPNetCore hosting by default.

Perigee Logging

Logging is built throughout Perigee. Any thread that is managed and created is automatically sent a logger with the appropriate configurations.

PerigeeApplication.ApplicationNoInit("DemoApp", (c) =>
{
    c.AddRecurring("Recurring Task" (ct, l) => {
    
        l.LogInformation("See how this Thread is automatically sent a logger?");
    
    });
});

Getting a logger directly

We can easily get a logger directly from the ThreadRegistry any time we want outside of a ManagedThread. This logger is tied into the Serilog system and will automatically inherit the configuration defined.

PerigeeApplication.ApplicationNoInit("DemoApp", (c) =>
{
    //Get a logger from the ThreadRegistry
    c.GetLogger<Program>();
    
});

Log Scopes

As shown above, you can override the logging template with additional parameters. Here we've added the ThreadName to the logging template. The reason this works out of the box is because Perigee's internal thread management system injects this log scope before sending an ILogger back.

You can add as many custom log scopes as you need. They will appear differently for different log sinks. This is very helpful when adding a BatchID, CorrelationID, SystemID, NetworkID or other identifying information to the log request.

• If logging to a database with an additional column added to the logging table - any matching log scope will fill that column value

• If logging to something like Splunk, you'll see those log scopes present in the additional log data

• If an overriden template in the console logger includes the custom scope, it will be written when present

Scope demo

Make sure to setup your appsettings.json with the modified logging template:

{
    "Serilog": {
        "MinimumLevel": "Debug",
        "WriteTo": [
            {
                "Name": "Console",
                "Args": {
                    "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}]({ThreadName}) {Message:lj}{NewLine}{Exception}"
                }
            }
        ]
    }
}

Then add a simple recurring method and override the ThreadName supplied by Perigee:

PerigeeApplication.ApplicationNoInit("Scopes", (c) =>
{
    c.AddRecurring("RecurringLogger", (ct, l) => {

        //Begin a new log scope on this logger, overriding the ThreadName
        using var scopes = l.BeginScope(new Dictionary<string, object> { { "ThreadName", "CUSTOMIZED" } });

        //Anything now logged from here will have it's "ThreadName" set to "CUSTOMIZED"
        l.LogInformation("See the overriden thread name?");

    });
});

The output will look like this:

[15:09:16 INF](CUSTOMIZED) See the overriden thread name?

Over the final few years in development we have worked closely with other companies to test, refine, and apply Perigee to their business needs. Here are a few testimonials of successfully implementing Perigee in their current production applications.

Parallel Transformation Task

With the assistance of Perigee we utilized the powerful set of parallel processing libraries to speed up our customer data transformation process by a little over %500,000 - from hours to microseconds. Perigee made this possible for us.

Multi-Agent Daily Refresh Tasks & Scheduling

Perigee allowed us to schedule our complicated daily refresh processes with ease. The “Sync Agents” allowed us to build up a tiered and structured refresh system that follows our requirements. We get reports sent to our team with the status of every job after they complete so we can easily monitor the process. It’s been running for almost a year now and through the logging has helped us identify multiple outages on our third party system.

Daily Integration Process With Notifications

Perigee serves as the foundation for our daily integration and data mapping between source systems. It enables us to schedule synchronizations daily, with the flexibility to initiate ad-hoc syncs as needed. The powerful logging allows us to query our database for each step of the refresh process on a per-transaction basis. In case of any issues, we receive instant notifications on our devices, making us aware of system outages within minutes.

Azure/Graph - Custom Sharepoint Workflow

We used Perigee to work directly with Microsoft Graph and Azure Service Bus to enable a custom Sharepoint solution for our analysts. Perigee handles the daily Authentication requirements, and provides us with an easy way to temporarily turn off the service bus when applying patches to our code base. We saved several hundred development hours by using Perigee.

Email Data Processor

Our email bot has processed thousands of files over the past 8 months, all thanks to Perigee's IMAP connector. It features a robust SDK that allows us to receive files, respond with dynamic HTML, and process the results as we get new requests. Perigee enabled our team to finish this task with minimal effort and saved weeks of development time.

Empowered Our Client Website's Query Functionality

Perigee has empowered our company to efficiently process and prepare large hierarchical documents, owing to its parallel processing and NestedSet helpers. We can effortlessly query the document hierarchy and pose complex questions from the UI, which would be impossible without the data preparation carried out by Perigee.

Transformation Engine

We build a powerful transformation engine powered by the incredible CSV reader provided by Perigee. We're able to load, process, and transform customer data at lightning speed - far better than anything we tried previously.

Resources

Our first application

Go ahead and create a new .NET 5/6+ console application. Open up Program.cs, and head to the first step to get started!

Step 1) The application starting point

Let's start with the basics. First let's create a new application in Program.cs. Depending on the version of .NET you started with, you'll see two different things when you open Program.cs

Net 6++

The .NET 6 ++ version looks like this:

// See https://aka.ms/new-console-template for more information
Console.WriteLine("Hello, World!");

//YOUR CODE HERE

If this is the case, delete Line 2 - Console.WriteLine("Hello, World!"); and start there.

Net 5 and lower

The .NET 5 and below versions looks like this:

class Program
{

    static void Main(string[] args)
    {
        //YOUR CODE HERE
    }
}

If this is the case, start coding on Line 6.

The application code

Now that we know where to start, here's a simple Perigee application.

//A fully managed perigee application! 
PerigeeApplication.ApplicationNoInit("FirstApp", 
(taskConfig) => { 

});

Let's look what what we have here in the constructor:

• Line 2 - "FirstApp" - The name of the application, this can be whatever you like!

• Line 3 - taskConfig block - This block is where we add any and all thread managed tasks.

using Microsoft.Extensions.Logging;
using Perigee;

Step 2) Adding thread managed code

This wouldn't be a hello tutorial without writing "Hello, World!", so let's add a CRON method to log Hello Perigee every 15 seconds!

The .AddCron method adds a new thread to the system. Each thread is managed by Perigee and is independent of every other thread. This is an important aspect of Perigee Application Design as it allows for individual thread management. Threads do not affect each other, and Perigee's internal thread management system has mechanisms in place to automatically restart downed threads.

//A fully managed perigee application! 
using Microsoft.Extensions.Logging;
using Perigee;

PerigeeApplication.ApplicationNoInit("FirstApp", (taskConfig) => {

    taskConfig.AddCRON("HelloWorld", "*/15 * * * * *", (ct, log) => {
        log.LogInformation("Hello Perigee from {appName}", taskConfig.AppName);
    });

});

• Line 4 - We use fluent syntax to .AddCRON() - This method takes a name, CRON string, and a callback(cancelToken, ILogger).

• Line 5 - We use the built in logger passed to us to log information in a templated format, passing in the name of the application to the logger.

Running the application produces a log new line every 15 seconds!

To close this application using graceful shutdown, press CTRL-C, it will start the safe shutdown procedure allowing the application to properly stop all running tasks before just exiting out.

Hello Perigee - Extended

Simply replace .AddCron() with the directory watcher instead, full code below:

PerigeeApplication.ApplicationNoInit("FirstApp", (taskConfig) => {

    taskConfig.AddDirectoryWatch("CSV", "C:\\Watch", "*.csv", SearchOption.TopDirectoryOnly, (ct, l, path) => {

        //Read the CSV
        var CSVData = CSVReader.ToDataTable(path, out var rRes);

        //Reprt on it
        l.LogInformation("Read CSV {file}[{encoding}]. Columns/Rows: {col}/{row}; Delimiter: {delChar}; Jagged? {jagged}",
            Path.GetFileName(path), rRes.FileEncoding.EncodingName, rRes.ColumnCount,
            CSVData.Rows.Count, rRes.FinalDelimiter, rRes.RowShifts.Count > 0 ? "YES" : "NO");

        //Remove it - OR let the failed folder policy kick in
        //File.Delete(path);

    }, policy: ThreadRegistry.DirectoryWatchFailurePolicy.MoveToFailedFolder);

});

Here's a sample.csv:

Name, Age, Email, Phone
John Doe, 30, john.doe@example.com, 123-456-7890
Jane Smith, 25, jane.smith@example.com, 987-654-3210
Bob Johnson, 40, bob.johnson@example.com, 555-555-5555
Mary Lee, 28, mary.lee@example.com, 111-222-3333
Hash Brown, 35, hash.brown@example.com, 444-444-4444

using Microsoft.Extensions.Logging;
using Perigee;
using Perigee.FileFormats.CSV;
using System.Data;

Voila! You're now watching for CSV files, reading them in, and reporting on the number of rows and columns.

For an intro to Perigee we accomplished quite a lot.

• We learned about threads - how they're added and independent of each other.

• We learned about logging information to the available sinks.

• CRON strings and how to use a CRON thread.

• How to configure Perigee.

• What "Graceful Shutdown" looks like.

• We even got to see the CSV reader in action.

Let's hop over to the next section and continue!

Installation

Find the package manager and type install-package Perigee. It's available on NuGet.

License Installation

To install your .license file, simply add it to the projects root folder and set it to "Copy Always".

Installation on a deployed application is exactly the same! Make sure the .license is in the executable directory.

Basic Configuration

To actually use the logging, we need a basic appsettings.json file in the project.

A) Either drop the included sample configuration into the project

B) OR - Create the file (appsettings.json) in the root of the project with the code below:

{
  "ConnectionStrings": {
  
  },
  "AppSettings": {
  
  },
  "Perigee": { "HideConsole": false },
  "Serilog": {
    "MinimumLevel": "Debug",
    "WriteTo": [
      { "Name": "Console" }
    ]
  }
}

We will cover additional logging and configuration in /HelloLogs later!

A quick word on "Using" Statements

Since we use Serilog, you may be inclined to use those namespaces at the top of your code. Typically speaking, we import the Microsoft.Extensions.Logging package instead. Since Serilog extends that generic functionality, we can do everything we need to with its inclusion. To get access to the logging functions, put this include at the top of the project:

using Microsoft.Extensions.Logging;
using Perigee;

That's it! Perigee is installed and ready to go. Head to the next section, let's write our first "Hello Perigee!" application

Resources

Intro

Lets take a closer look at how the configuration works within Perigee.

To understand the configuration process we need to understand how the sections work and why we separate them. To keep it concise, sections create a logical separation of values for the different processes in our application.

Configuration sections can be registered in our application in a number of ways including but not limited to:

• Environment variables

• main(string[] args)

• appsettings.json file

• Custom providers

Back in the Installation and Project Setup we used a configuration file. A section is a key off of the root json node and as you can see there are 4 different sections in our startup file.

{
  "ConnectionStrings": {
  
  },
  "AppSettings": {
  
  },
  "Perigee": { "HideConsole": false },
  "Serilog": {
    "MinimumLevel": "Debug",
    "WriteTo": [
      { "Name": "Console" }
    ]
  }
}

Demo Application

Our goal is to write an application that logs out our custom configuration sections. We will do so by reading the configuration directly at runtime as well as binding the configuration section to a custom class.

You will also see some helpful ways you can plug the configuration system into managed threads

Configuration Direct Read

To start, we need to add another section to our appsettings.json file called HelloConfig and give it some values to read.

{
  "ConnectionStrings": {
  
  },
  "AppSettings": {
  
  },
  "HelloConfig": {
    "Enabled": true,
    "Name": "HAL 9000",
    "Year": 2001,
    "Tags": [ "Heuristic", "Algorithmic" ]
  },
  "Perigee": { "HideConsole": false },
  "Serilog": {
    "MinimumLevel": "Debug",
    "WriteTo": [
      { "Name": "Console" }
    ]
  }
}

This creates a section for us to read in our code. We will read it directly by calling taskConfig.GetValue<>().

This handy method is a quick way to reference ALL of the incoming configuration providers and it works by supplying two things:

1. The C# Type - This could be a string, or int, or whatever other value type you're reading.

2. A special format: Section:Key.

To read the Name from our configuration section the Type would be string and our Section:Key would be: HelloConfig:Name

Here's a fully working Perigee application

PerigeeApplication.ApplicationNoInit("HelloConfig", (taskConfig) => {

    taskConfig.AddRecurring("TestMethod", (ct, log) => {

        log.LogInformation("What is my name? It is {name}", taskConfig.GetValue<string>("HelloConfig:Name"));

    });
});

• Line 1 - This is a simplified Perigee start method that skips IPC tokens and initialization actions.

• Line 3 - AddRecurring simply adds a managed thread that is automatically called every n milliseconds (supplied by an optional parameter, default 5000 (5 seconds))

• Line 5 - We log out the configuration value reading it live at runtime.

while (PerigeeApplication.delayOrCancel(delay, cancelToken)) { 
  // This is only called after delay, and if it's not cancelled
  // Otherwise, the loop block ends
}

You'll see below the results of running this application. We get a log every 5 seconds with the value captured from our configuration file.

Configuration Binding

Next let's look at binding the section we added to a custom class. Here's the file if you prefer to import it.

The class simply has 3 properties that are an exact case sensitive name match to the configuration section we created.

public class HelloConfig
{
    public string Name { get; set; } // HelloConfig:Name
    public int Year { get; set; } // HelloConfig:Year
    public List<string> Tags { get; set; } // HelloConfig:Tags
}

To bind a class, simply use the taskConfig and call GetConfigurationAs<>():

taskConfig.GetConfigurationAs<HelloConfig>("HelloConfig");

• The Generic T Parmater, HelloConfig, is the class we're creating and assigning. It's a type reference.

• The string, "HelloConfig", is what section to get from our configuration so that we can assign it's properties to the referenced type.

This is all that is needed to bind our configuration section to a class. Here's the full application up to this point!

PerigeeApplication.ApplicationNoInit("HelloConfig", (taskConfig) => {
   
    taskConfig.AddRecurring("TestMethod", (ct, log) => {

        //Directly by reading
        log.LogInformation("What is my name? It is {name}", taskConfig.GetValue<string>("HelloConfig:Name"));

        //Binding a class
        HelloConfig config = taskConfig.GetConfigurationAs<HelloConfig>("HelloConfig");
        log.LogInformation("{name} first appeared in {year:N0} and was {@tagged}", config.Name, config.Year, config.Tags);

    });
});

Line 10 - We got the configuration as a custom class by binding to it.

Line 11 - We use the custom classes properties to write to our log.

Link to Config

Perigee will, by default, hot-reload the appsettings.json files. This is a fantastic way to give control to a configuration file to enable or disable a thread without shutting down the entire application.

Perigee has a great helper method built into it which enables the thread-to-configuration linking for enabling or disabling managed threads through a configuration boolean.

Our example config section we added (HelloConfig) which has a key called Enabled, and we're going to use that Boolean to tell Perigee whether or not to start or stop our thread.

PerigeeApplication.ApplicationNoInit("HelloConfig", (taskConfig) => {
   
    taskConfig.AddRecurring("TestMethod", (ct, log) => {

        //Directly by reading
        log.LogInformation("What is my name? It is {name}", taskConfig.GetValue<string>("HelloConfig:Name"));

        //Binding a class
        HelloConfig config = taskConfig.GetConfigurationAs<HelloConfig>("HelloConfig");
        log.LogInformation("{name} first appeared in {year:N0} and was {@tagged}", config.Name, config.Year, config.Tags);

    }, started: false).LinkToConfig("HelloConfig:Enabled");
});

Line 12 - We added started: false to the methods optional parameter so that with the addition of the linking command, it will be in charge of starting or stopping this thread. LinkToConfig was added and we supplied our Section:Key to it.

Change Notification

If you'd prefer more control over how to start or stop your threads, or even make different decisions based on the configuration changes, you can always subscribe the change notification directly. .LinkToConfig() is essentially the same code as below but managed by Perigee internally and automatically, saving you from a few lines of code.

taskConfig.Event_ConfigurationUpdated += (sender, Configuration) =>
{
    bool enabled = Configuration.GetValue<bool>("HelloConfig:Enabled", false);
    if (enabled) taskConfig.StartIfNotRunning("TestMethod");
    else taskConfig.QueueStop("TestMethod", true);
};

Configuring the logging

Perigee will automatically add an additional config property block to each of it's loggers that it passes to the managed threads. This property block is called ThreadName. As a fun aside, let's update our logging template to include this property block.

Open the appsettings.json file and under the Serilog section, update the console sink to include the updated outputTemplate.

{
  "Name": "Console",
  "Args": {
    "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}]({ThreadName}) {Message:lj}{NewLine}{Exception}"
  }
}

Now our application's logs have the name of the managed thread they are coming from right in the log!

You can add your own properties to the loggers, we'll show you how on the Hello Logs page.

Secure Values

Perigee has several ways built in that allow you to encrypt your configuration values which may provide an extra layer of security/obscurity. It uses AES256 bit encryption on the values for secure encrypting and decrypting values and has built in methods for reading them.

To secure a value you must provide the encryption key, encryption IV, and the source of where to find these values.

There are 3 sources for the Encryption keys

• Environment Variable source

• Arguments Source

• Variable Source

Here's an example of setting this up all within code using the variable source. The methods are exactly the same though if you decide to pass in the encryption key from an argument, or read it from an environment variable.

PerigeeApplication.ApplicationNoInit("DemoApp", (c) => {

//Set the value source as variables, to which we assign next
c.SecureValueSource = ThreadRegistry.SecureKeyStorage.Variable;

//Create two new randoms, one AES 256 and the IV as AES 128
c.SecureValueKey = AesCrypto.GetNewRandom();
c.SecureValueIV = AesCrypto.GetNewRandom(16);

//Write out the two new generated values, so we can use them to decrypt values with later
c.GetLogger<Program>().LogInformation($"Copy these values out!!{Environment.NewLine}Key: {{key}}{Environment.NewLine}iv: {{iv}}", c.SecureValueKey, c.SecureValueIV);

});

With that done you should see something like this printed to the console:

These are your two encryption keys, and we will use them to decrypt values in the next step. Remember you can easily put these in the environment variables, command line arguments, or even set the variables in code. For the ease of this demonstration, we'll assign them directly:

PerigeeApplication.ApplicationNoInit("DemoApp", (c) => {

//Assign our keys
c.SecureValueSource = ThreadRegistry.SecureKeyStorage.Variable;
c.SecureValueKey = "e7zP37543Rbvn5a6NnN3FGlhPVAsdTmljcXZoTLOlkw=";
c.SecureValueIV = "UFn5LH+RLLCVkxt+qfjWKQ==";

//Encrypt and decrypt the same string
string encrypted = c.EncryptSecureValue("Hello World");
string decrypted = c.DecryptSecureValue(encrypted);
c.GetLogger<Program>().LogInformation($"Encrypted: {{Encrypted}}{Environment.NewLine}Decrypted: {{Decrypted}}", encrypted, decrypted);

});

If you're reading encrypted configuration values, there's a handy little helper than does the read and decrypt in one pass:

PerigeeApplication.ApplicationNoInit("DemoApp", (c) => {

    c.GetSecureConfigurationValue("HelloConfig:EncryptedValue")
    
});

Summary

Hopefully you understand a bit better about how configuration, sections, and providers work as well as how to use them in code.

If you're having issues getting a Perigee application to start or load properly, check the below items:

License check

The license check run's immediately on application startup, and if a valid license is not detected, it will immediately shut down the application.

1. Make sure you have a valid license, and your trial period is not expired

2. Make sure you're copying that license file to the output directory, or it is installed on the server.

Serilog configuration

If a Serilog sink is configured to startup on a remote source (like a database), Please verify that it has an open connection and there are no firewalls preventing Serilog from initializing.

It's very easy to add custom providers. Remember that .NET creates a ConfigurationSource that implements and returns a ConfigurationProvider.

You can see this pattern here: Implement a custom configuration provider.

Custom Implementation

To implement a custom source and provider only requires a single line of code.

1. Call the ConfigureConfiguration method.

2. This callback sends you two parameters:

   1. IConfigurationBuilder This builder allows you to add a custom source.

   2. EnivronmentString the string variable that is the current environment.

3. Simply add a new instance of the source. (See line 5)

PerigeeApplication.ApplicationNoInit("DemoApp", (c) => {
    
    //Add ConfigurationSource to the builder
    c.ConfigureConfiguration((builder, EnvStr) => {
        builder.Add(new SQLConfigurationSource("connectionString", "select [name], [value] from dbo.configs", "sql"));
    });

    //Reload all providers
    c.ReloadProviders();

    //Reload the specific provider
    c.ReloadProvidersOfType(typeof(SQLConfigurationProvider));
    
    string MyValueFromSQL = c.GetValue<string>("sql:MyValue");
    
});

You can see how easy it is in the future to reload the providers. You can reload all of them or specific types by supplying that type information.

Using the built in SQL Provider

If you're trying to load values from MSSQL then Perigee ships with a loader ready to go in only a single line of code. Simply call the configuration method and supply the connection string, query, and prefix.

PerigeeApplication.ApplicationNoInit("DemoApp", (c) => {

    //Configure the SQL Property loader
    c.ConfigurePropertyLoader_SQL("connectionString", "select [name], [value] from dbo.configs", "sql");
    
    string MyValueFromSQL = c.GetValue<string>("sql:MyValue");
    
});

Perigee Licensing Agreement

SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Licensing production and development use

To obtain a valid license after the trial period, visit https://Perigee.software to purchase and view license terms and conditions.

Third Party licensing agreements

Dapper, Serilog, Serilog.AspNetCore, Google.Apis.Auth, Google.Apis.Gmail.v1, RestSharp

The packages listed are licenced under Apache 2.0: http://www.apache.org/licenses/LICENSE-2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Ude.NET

The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/

Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.

CometD.NetCore2, Bogus, BrotliSharpLib, Newtonsoft.json, Newtonsoft.json.schema, Portable.BouncyCastle, ExcelDataReader, HtmlAgilityPack, MailKit, NetCoreForce.Client, Pluralize.NET, SharpZipLib

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Let's take a look at how you might extend Perigee to fit your companies very specific needs. Although we ship with many different connectors, watchers, agents, and synchronization features you may want to add something to the Perigee core system. It is very easy to implement custom threads. This is one the reasons Perigee is so powerful as we don't lock you out of adding and customizing the application.

There are two types of custom threads you can add where both have different execution rules as they allow you create any additional functionality needed for your specific requirements.

Managed Thread

The first type is a managed thread. The ManagedThread calls the callback method when it is started, and doesn't expect that method to return, throw an exception, or exit until the cancellation token is canceled.

When the callback does exit before token cancelation, it will restart the thread a second later expecting the process to start again. If an exception is thrown and is uncaught within the callback, it will be caught in the ManagedThread, and the ExceptionRestartTime property of the manager will be awaited before restarting.

Below is a fully working extension that adds a new method to call from TaskConfig.

public static ThreadRegistry AddCustomXYZFunction(this ThreadRegistry tr)
{   
    //Create a regular managed thread (not an expression, CRON)
    var TM = new ManagedThread("CustomXYZFunction", (ct, l) => { 
        
        //The callback method is here...

        do
        {
            //An example of repeating something every second, and exiting the thread when the token is cancelled.


            //Doing a long process that can be stopped safely? Pay attention to the cancellation token, and kindly exit when a graceful shutdown is requested.
            if (ct.IsCancellationRequested)
            {
                //Save or finish or persist information.
                return; 
            }

        }
        while (PerigeeApplication.delayOrCancel(1000, ct));

    }, tr.CTS, tr.GetLogger<Program>(), started: true);

    //Set additional options
    TM.ExceptionRestartTime = TimeSpan.FromSeconds(30);

    //Add the thread to the management system
    tr.AddManagedThread(TM);
    
    return tr;
}

Then to use this custom method on startup.

PerigeeApplication.ApplicationNoInit("DemoApp", (c) => {

    c.AddCustomXYZFunction();
    
});

Expression Thread

This is nearly identical to the managed thread, however it's execution is built on a CRON expression. This changes the way the callback functions slightly.

The Callback Method is ONLY called the expression is at execution time. This means the process is expected to kick off, and end by returning out of the method.

If an exception is thrown, it will be caught in ManagedThread, but no additional delay is added as the next execution period of the CRON expression will simply call the callback again.

Purchase a license

To purchase a license, please visit https://perigee.software.

Have questions, need more than 10 licenses or want to discuss volume discounts?

Contact us at Sales@perigee.software. We are happy to hear from you!

This is the last of the hello series. We're going to cover the final few topics you'll need to really get started. Let's jump in!

Watermarking

Watermarking is simply storing a key/value pair from last execution and recalling it again next time it is needed. We use this when interfacing with another system we need to synchronize data to or from. A few examples of this are:

• A Mail sync client needs to know what the UIDNEXT, or MODSEQ integers are.

• If you're synchronizing data changes to SharePoint, you'll want to store the delta link as a string.

• If you're pulling data by a last modified date, you'll want to make sure you store a DateTime(Offset).

These are just a few examples of how you would use watermarking in a real world production application.

The SDK

We make working with watermarks quite easy and they are very powerful.

• Our watermarks are persisted locally to disk automatically and recalled on application startup.

• They are debounced (We cover this in the updating section).

• They are thread safe.

• You can register as many subscribers as you need to the data feed.

• You may push that data externally on update, and recall it on initialization for further backup.

//Register a watermark
Watermarking.Register("IntegrationOffset", () => Watermark.FromDateTimeOffset(DateTimeOffset.UtcNow), (nVal) => {
    taskConfig.GetLogger<Watermark>().LogInformation("New value for {name} ({value})", nVal.Name, nVal.GetDateTimeOffset());
});

//Register additional handlers anywhere else, they receive updates to that watermark
Watermarking.RegisterAdditionalEventHandler("IntegrationOffset", (s, nVal) => {
    taskConfig.GetLogger<Watermark>().LogInformation("AddlHandler: New value for {name} ({value})", nVal.Name, nVal.GetDateTimeOffset());
});

//Push updates to this watermark anywhere else
Watermarking.UpdateWatermark("IntegrationOffset", Watermark.FromDateTimeOffset(DateTimeOffset.UtcNow));

//And when you need the value, refer to it again later
Watermarking.GetWatermark("IntegrationOffset").GetDateTimeOffset()

PerigeeApplication.ApplicationNoInit("HelloConfig",
(taskConfig) =>
{
    //Register me first!
    
    
    //Then .Add other methods
});

Registering

Registering a watermark is an easy one liner. Let's look at the parameters:

Watermarking.Register(

//Name 
"IntegrationOffset", 

//Initialization Callback
() => Watermark.FromDateTimeOffset(DateTimeOffset.UtcNow), 

//OnUpdate (optional)
(nVal) => {
    taskConfig.GetLogger<Watermark>().LogInformation("New value for {name} ({value})", nVal.Name, nVal.GetDateTimeOffset());
});

• Line 4 - Name - The name of the watermark

• Line 7 - Func<Watermark> Initialization callback - If the value does not exist, the initialization callback is called and an initial value is set.

• Line 10 - Action<Watermark> onUpdate? - An optional callback for when the value is updated.

Watermark.FromString("");
Watermark.FromInt(1);
//etc

That's it. The underlying system does the heavy lifting of persisting this locally under the /watermarks folder beside it's running application.

Registering Handlers

You're able to register as many "subscribers" to the data changes you desire in other parts of the code. These will be called on a background thread with the updated value.

The RegisterAdditionalEventHandler simply takes two parameters:

Watermarking.RegisterAdditionalEventHandler(

//Name of the watermark to register
"IntegrationOffset", 

//Callback function
(s, nVal) => {
    taskConfig.GetLogger<Watermark>().LogInformation("AddlHandler: New value for {name} ({value})", nVal.Name, nVal.GetDateTimeOffset());
});

• Line 4 - Name - The name of the watermark you want to update.

• Line 7 - EventHandler<Watermark> - This sends you the (sender, watermark) back in the callback and you're able to read the value from there

Updating

The UpdateWatermark call takes two parameters:

Watermarking.UpdateWatermark(

//Name
"IntegrationOffset", 

//New value
Watermark.FromDateTimeOffset(DateTimeOffset.UtcNow));

• Line 4 - Name - The name of the watermark you want to update.

• Line 7 - Watermark - The new watermark and value.

A lot happens behind the scenes you should be aware of:

• Updates are debounced which means that rapid updates don't all push their values at once to any subscribers of the data updates.

  1. Let's say you processed 10 records in a matter of 50 milliseconds and every record you updated the watermark so it went from the initial value of 0 to 10.

  2. After about a second, all of the subscribers would get a single updated watermark event with the value of 10.

  3. Debouncing reduces the number of calls both to the filesystem and to your subscribers

• Updates are thread safe as they perform internal item locks before writing to disk and notifying subscribers.

Get

Getting the watermark only requires the Name of the watermark to retrieve.

Watermarking.GetWatermark("IntegrationOffset").GetDateTimeOffset()

If there are inflight updates to the watermark, this call may block the thread for a second or two while those debounces get processed. If there are no inflight updates, it returns immediately.

Credential Store

Another primary concept of writing integrations is using credentials to get access tokens and authorization for other systems. The most common example of this would be using OAuth 2.0 to retrieve an access or bearer token that expires in the future.

Credentials are all registered on startup and are usually given an expiration date. This allows for any process later in the application to ask the manager to retrieve a credential. The below example shows how to register SuperCredential, and two possible return values:

CredentialStore.RegisterRefresh("SuperCredential", (o) => {

        //Call third party API
        //Reach out to database
        //Request keys from AWS
        //Do anything you need to get authorization details

        //Then return either a good credential:
        return new CredentialStoreItem() { 
            Expiration = DateTimeOffset.Now.AddMinutes(60),
            Authorization = "ABCDEFG",
            Scope = "offline.access",
            StoreA = "Custom ValueStore"
        };

        //Or a faulted one:
        return new FaultedCredentialStoreItem("External ERROR", new Exception("Exception thrown while trying to get credential!!!"), retry: true);
    });

When the application loads up or a new credential is created, the local drive is checked and all previously non expired credentials are restored immediately. This allows for applications to be safely shut down and restarted without interrupting or over-retrieving from an external API.

During the retrieval process, the CredentialStore checks for existence of the credential, it's expiration and then checks if the expiration is within N number of minutes(user defined) before returning to the caller.

//Use default settings
CredentialStore.GetCredential("SuperCredential"); 

//Set all settings on retrieval 
CredentialStore.GetCredential("SuperCredential", maxRetries: 3, retryMS: 1000, expireTimeBufferSeconds: 600);

In Summary: the CredentialStore does several important things:

• It automatically caches the authorization details to disk and retrieves them again on application reload.

• When supplying expiration dates to a credential, it's able to renew itself when it's required before sending back the old authorization parameters.

• The retrieval call is automatically retried to prevent a blip in connectivity from stopping a process.

• It will automatically renew the token a defined number of minutes before expiration, allowing long running processes not to fail while in the middle of execution.

• It integrates seamlessly with RestSharp, allowing all HTTP(S) traffic to automatically and internally pull credentials

Using Credentials - The SDK

RestSharp for HTTP(S)

To register a specific RestSharp credential, just create that type and assign the appropriate Authenticator.

CredentialStore.RegisterRefresh("ClientAPI", (o) => {

        return new RestSharpCredentialStoreItem(
                new RestSharp.Authenticators.JwtAuthenticator("MyToken"), 
                DateTimeOffset.UtcNow.AddMinutes(58));
        
});

Any time you create a client, simply assign this credential authenticator:

var rco = new RestClientOptions("https://localhost.com")
{
    //Assign the CredentialAuthenticator with the name tied to the refresh
    Authenticator = new CredentialAuthenticator("ClientAPI")
};

//Create a new client
using var client = new RestClient(rco);

//Anytime you execute any requests, it's automatically maintained. 

There's a whole page on this content, read more here:

Limiting

Another integration pattern is limiting the number of calls that can be made during a given period of time. This is important when rate limiting or billing may be an issue and is a perfect stop gap for a critical failure causing an erroneous number of API calls or running up an unwanted bill!

The Limiter auto resets it's count every calendar day and will not allow additional executions past the defined limit.

using Perigee;
using Perigee.Extensions;

PerigeeApplication.ApplicationNoInit("Limiter", (c) =>
{

    c.AddRecurring("Limiter", (ct, l) => {
        using var APILimit = new Limiter("APILimiter.json", 2, ct);

        if (APILimit.CanExecute())
        {
            //Perform action
            l.LogInformation("Calling API {n}/{x} executions", APILimit.GetCount(), APILimit.GetLimit());
        }
        else
        {
            l.LogInformation("Out of executions...");
        }
    });
});

/*
 [16:35:07 INF] Calling API 1/2 executions
 [16:35:13 INF] Calling API 2/2 executions
 [16:35:18 INF] Out of executions...
 */

Fault Tolerance

A common integration issue is that systems outside of the immediate domain will have faults, issues, disconnects, exceptions, availability or downtime. All of these issues are usually easily resolved with a fault tolerance approach to building your application.

Perigee has several classes built into it to make this a common practice within your application.

The Cancellation Token

The Cancellation Token passed alongside every ManagedThread is an important fault tolerant key to be aware of. They allow us to gracefully shut down or stop processing data, and give us an opportunity to save state before the application closes.

If you can successfully pay attention to them, they give you the opportunity you need to end your process and save state.

PerigeeApplication.ApplicationNoInit("CancelTokens", (c) => {

    c.AddRecurring("Recurring", (ct, l) => {

        //Since we haven't started yet, can we stop and resume later?
        if (ct.IsCancellationRequested) {
            //Save state or info first?
            return;
        }

        //Process stuff
        // ...

        //Are we at a safe place to stop and resume later? Check the cancellation
        if (ct.IsCancellationRequested) return;
    
    });

});

Exit Event

Use the Exit Event alongside the cancellation tokens to store and finalize state on shut down, this allows for an easy application reload, update, or server restart. Remember Exit is called immediately before the application closes, and in some cases only allows for a few seconds to pass before ending execution. This is a last step process where data should be persisted.

PerigeeApplication.OnExit(() => {
    Console.WriteLine("App is closing in a few seconds");
});

Retry and Rety Async

One of the utility class methods is a Retry and RetryAsync. They provide an easy ability to retry on exception thrown with a delay.

The response from these classes is a Tuple where

• Item1 is a boolean indicating if it was a success(true), or failed the maximum number of times (false).

• Item2 is the exception thrown if it was not a succeed

PerigeeApplication.ApplicationNoInit("CancelTokens", (c) => {

    c.AddRecurring("Recurring", (ct, l) => {

        //This will be tried 5 times, with 10 seconds between retries. 
        Tuple<bool, Exception> Retry = PerigeeUtil.Retry(5, (i) => {

            bool Success = false;
            if (Success == false)
            {
                throw new Exception("Do or do not, there is a retry");
            }

        }, 10000);

        if (!Retry.Item1)
        {
            l.LogError(Retry.Item2, "Retries exceeded");
        }

    
    });
});

REST Retry

A lot of cases network connectivity issues can easily be dealt with by simply retrying a moment later. These handy extensions are built right alongside RestSharp to use in place. If they don't suit your needs, you can always wrap the call chain in the utility retry methods instead!

In the example below: ExecuteRetry(request, count) will execute the request at least once.

• If a success response code is sent back, it will return and not execute the retry.

• If a failure code is returned it will retry as many times as supplied with a 5 second delay between requests.

  • If using the CredentialAuthenticator and a 401 (unauthorized) status is returned, it will detect and invalidate the credential. This will refresh the authorization token/mechanism and retry again with the new token.

PerigeeApplication.ApplicationNoInit("CancelTokens", (c) => {

    CredentialStore.RegisterRefresh("TokenAuth", (o) => { 
        // Usually this would include making a request or generating a new token,
        //  for this simple demo, it's creating a basic authentication header
        return new RestSharpCredentialStoreItem(new HttpBasicAuthenticator("user", "pass"), DateTimeOffset.Now.AddMinutes(45)); 
    });

    c.AddRecurring("Recurring", (ct, l) => {

        var rco = new RestClientOptions("https://postman-echo.com") { 
            Authenticator = new CredentialAuthenticator("TokenAuth") 
        };
        using RestClient rc = new RestClient(rco);
        RestRequest rq = new RestRequest("/status/401", Method.Get);
        
        //Execute retry will retry 2(n) times, and if the response code is a 401 (unauthorized)
        //  it will automatically invalidate the credential and refresh the token
        rc.ExecuteRetry(rq, 2);

    });
});

FileSyncWrite

FSW for short is the primary class suited for synchronizing data to the local drive. It's explained in detail here. In short, you supply the CancellationToken from the thread and it safely handles the file locking and writing of newly updated data on a regularly scheduled basis and before the application closes.

This provides a fantastic way to keep state or progress on any tasks ongoing.

If you run the below demo multiple times, you'll see the count being restored, incremented and shut down. There's a new file with the data located at bin\debug\sync\local.json.

//Declare a cancellation token source to control the FSW. 
//If using FSW within a ManagedThread, use the passed cancellation token
CancellationTokenSource CTS = new CancellationTokenSource();

//Local persist data
LocalPersistData localData = new LocalPersistData();

//Declare a new FileSyncWrite, it's generic T type can be any class that supports new()
FileSyncWrite<LocalPersistData> _FSW = new FileSyncWrite<LocalPersistData>($"sync{Path.DirectorySeparatorChar}local.json", 
    CTS.Token,              //The token mentioned above

    rollingDelay: 1000,     //Rolling delay is bumped every time an Update() is pushed in. 
    maximumDelay: 10000,    // Maximum delay is if rolling delay is never expired
                            //  When either event is satisfied, the data is written to disk

    (s,data) => { }         //This is an updated event callback. Any time the data is written, this is called

    );


//If initialized, then a value was loaded back in from the disk.
//If this is FALSE, no value was loaded in
if (_FSW.InitializedValue)
{
    localData = _FSW.Get();
    Console.WriteLine($"FSW Initialized: {localData.Count}, {localData.Offset}");
}

//Register an additional callback
_FSW.UpdatedEvent += (object s, LocalPersistData e) =>
{
    if (e != null)
        Console.WriteLine($"FSW Updated: {e.Count}, {e.Offset}");
};

//Push debounced updates to it:
localData.Count++;
_FSW.Update(localData);

//Tell the FSW to end all pending update queues and sync data back now
CTS.Cancel();

Task.Delay(8000).Wait();
return;

The class used above is simply two properties with getter/setters:

public class LocalPersistData
{
    public int Offset { get; set; }
    public int Count { get; set; }
}

Thread Conditions

Thread Conditions provide a way to cancel processes if a condition is not met. It's a great way to turn off a service is the network is down or the hard drive is too full. They're fully customizable and allow you to set your own logic for how they should be run.

Transaction Coordination

Transaction Coordinator is the big boss of all of the above methods. It implements cancellation tokens, exit events, retries, the FSW modal, and local+remote two way synchronization.

Think of the coordinator as a way of defining the steps of a process in logical tiny blocks, and always verifying you move through those blocks in the correct order while never missing a step, or data in the process.

It's so fault tolerant you can terminate the running application, and restart it at any point and it will pick up where it left off. Impressive eh?

To see this better in action, check out it's page!

APIBuilder

This static class enables users to create an API specification using both custom specifications as well as specifications derived from Postman integration. Most notably, it provides a way to fully generate a complete Web API given an APISpecification.

From

This method allows users to manually build an API Specification using a fluent syntax.

You can easily add endpoints after declaring the specification like so:

Example:

//Declare spec
var spec = APIBuilder.From("ProjectName", AuthorizationType.Bearer, true, true);

//Add as many endpoints, in this example, an echo endpoint
spec.AddEndpoints(
  APIGeneration.Endpoint.From(spec, "Echo", "/echo", APIGeneration.Method.GET)
    .WithRequestJSON(@"{""echo"": ""message""}")
    .WithAIAssistance("Echo the message back to the use in json"));

FromPostman

This method allows users to build an API Specification from Postman. This is useful for integrating existing Postman collections into new or existing projects.

Example:

APIBuilder.FromPostman("ProjectName", "PostmanCollection.json", "env.json");

GenerateAPI

This method generates a full web API given an APISpecification. It requires the specification object, the project path, as well as optional parameters for IIS, HTTP and HTTPS Ports, additional installs, additional App Settings, a GPT Token and Model for generative AI, and boolean flags for generating Controller and Method summaries.

Example:

APIBuilder.GenerateAPI(spec, "ProjectPath", 60978, 7181, 7182, additionalInstalls, additionalAppSettings, "GPTToken", "gpt-4", true, false);

Introduction

API Generation empowers developers to accelerate the development process of setting up a new API significantly. Whether you're creating an API for the first time or you're a seasoned veteran, crafting an API is more straightforward with the tools provided by Perigee.

API Generation refers to the creation of an API from a source document, such as Postman, or directly in code using the fluent syntax, without the in-depth knowledge required to configure a new API from scratch. It might seem too good to be true...

Benefits of API Generation

When contemplating your next API project, here are some points to consider:

Mocking

Mocking is the procedure of emulating a production API and creating a "shadow copy" for testing purposes. This ensures tests are conducted without the repercussions of calling the actual API. Such capability grants developers complete control over integrations. They can alter API responses, introduce delays, and disable parts of the API, facilitating better integration development from the outset.

A New API?

Embarking on a new API for testing or production? Feed the generator a set of JSON documents and gain a substantial advantage in your new venture. Sidestep the tedious hours spent on configuring authorization, settings, or Swagger examples—let the code handle it!

Typo Free

Eliminate concerns over model typos or overlooked code. We generate everything based on a model registry that ensures the output is pristine C# code.

Time Savings

In our demo, we replicated the DropBox API, resulting in approximately 35,000 lines of code, encompassing:

• Class Models

• Documentation

• Controllers and Actions

• Swagger Samples

• Comments

• Authorization

AI Power

Curious about how OpenAI's ChatGPT API hinting operates? Here's an insight:

Consider an API method you've set up named EchoMessage. The model defined for this is a class named Echo that comprises a message string.

Your AI hint suggests: "Take my echo message method and return a JSON object that sends the user's message back to them."

In response, the API Generator composes a specific message for GPT, containing your input/output models and the organized method, providing GPT with the complete context to fulfill your request. It appears like:

public class Echo { public string message {get; set; } }

[HttpPost]
public IActionResult EchoMessage([FromBody] Echo request) {
    //{Take my echo message method and return a json object that returns the users message back to them}
}

GPT could suggest the following improved code (this is an actual response from GPT!):

[HttpPost("EchoMessage")]
public IActionResult EchoMessage([FromBody] Echo request)
{
    if (request == null)
    {
        return BadRequest("Request body is null");
    }
            
    return Ok(new { echoedMessage = request.message });
}

Impressive, isn't it?

Application initialization

Both application starting points are nearly identical. The full callback has an IPC token and the initialize callback. Both are described in Hello Perigee.

Application(string appName, string IPCCancelToken, Action<IConfiguration> initialize, Action<ThreadRegistry> taskConfiguration, CancellationTokenSource cancelSource = null, Task HostTask = null, string[] CommandLineArguments = null)
ApplicationNoInit(string appName, Action<ThreadRegistry> taskConfiguration, CancellationTokenSource cancelSource = null, Task HostTask = null, string[] CommandLineArguments = null)

You may also run the ApplicationNoInit as an async method as well:

PerigeeApplication.ApplicationNoInit("async", async (c) => { 
    await Task.Delay(1000);
});

Initialize outside a PerigeeApplication

Sometimes you need to initialize outside of the scope of a PerigeeApplication. We do this for API applications, blazor applications, and other scenarios where we need to call something before Perigee.

static void Main(string[] args) {

   //To initialize perigee and the thread system:
   ThreadRegistry tr = ThreadRegistry.InstanceWithArgs(args);
   
   
   //Initializing credentials is not usually required unless changing working directories.
   //If you must initialize them, use either A or B:
   
   //A) Call instance
   var cinstance = CredentialStore.Instance;
   
   //B) Congiure
   CredentialStore.Configure()
}

Host Statistics

To get information about the host it's running on, there is a single method:

GetHostStatistics()

The response class, HostStatistics contains properties for the machine it's running on including machine name, drive info, cpu time and running directory.

Exit event

To get an exit callback, use the helper method supplied.

After return is executed and the domain/service is about to unload, OnExit has a few seconds to perform any last tasks, like persisting data.

static void Main(string[] args)
{
    PerigeeApplication.OnExit(() => {
        Console.WriteLine("Exiting");
    });

    return;
}

Let's take a look at what sources we may use and create the configuration, credentials, and refreshes we need to use.

• Do we want to put connection strings in the appsettings.json and read them?

  • Create the json sections and keys

  • Optionally, encrypt them

• Do we want to implement a property loader?

  • See Property Loaders

• Do we communicate with anything on the web?

  • See Restsharp Authenticators and Credential Management.

  • Read all about the HTTP(S) communication and extensions here.

• Do we need to define additional services to dependency inject?

  • See the Core Module: Application.

• Do we have any important information we need to keep in synchronized state like pull dates, offsets or delta keys?

  • Define a Watermark for all basic data types

  • Or define a custom FileSync to track a rich data object

  • Look at the Limiter for resource restricted calls.

• Do you need to have things run on a timer?

  • Check out the CRON thread.

  • Have a scheduler pick up items from a remote source, like a database.

  • Have a network of Sync Agent's communicating and performing refresh tasks.

• Do you need to coordinate a list of steps and NEVER have a failure cause the process to break?

  • Create a Transaction Coordinator.

• Talk with third parties:

  • Salesforce

  • Microsoft Graph

  • DataVerse

  • SmartSheets

  • Azure Service Bus

• Do you have any instances of:

  • Testing Network Connectivity?

  • Sorting dependencies?

  • Working in Parallel?

  • Sending notifications?

  • Run logic through a behavior tree?

  • Sort and query hierarchical data?

For this section, let's walkthrough start to finish how you might design and build your companies specific application needs.

Here's the quick bullet list of every step of this process:

• Starting template

• Define Sources

• Requirements

Starting with a template

• Does our system need to respond to API requests?

  • Is data pushed to your service with POST requests?

  • Does it need to respond to a ping request?

  • Does it provide on-demand data through restful calls?

• Do you plan on implementing administrative requests for your system?

  • Do you want to be able to shutdown threads with an administrative login?

  • Do you want to check the status, runtime, or other system metrics?

If you answered "yes" to any of the above: lets start with this blueprint for built in hosting:

Otherwise we can create a standard .NET 6 project.

What about a CRON?

CRON strings are a concise way of telling the application when to schedule or run something. Perigee uses CRON string data throughout for advanced process scheduling and events.

CRON Data Explanation

Most CRON strings contain five bits. Perigee's CRON can contain 6 if you use the optional seconds bit.

Second   Minute      Hour      Day      Month      Day-Of-Week
   *        *         *         *         *            *

• The ASTERISK character means "All Available Values"

  • For hour, this is 0-23

  • For Minute, this is 0-59

  • For Day this is 1-31

  • For Month this is 1-12

  • For Day-Of-Week this is 0-6 (Sunday being 0)

• The COMMA character is a list of values

  • If the Month bit is 0,1,2 - Then the CRON is only run in Jan, Feb and March.

• The DASH character means a range from a-z.

  • If the day bit is 0-6 - Then the CRON is only run the first 7 days of the month

• The SLASH character is a divider of all available values.

  • If the second bit is */5 - Then the CRON is executed on every fifth second.

• There are several special characters that can only be used in certain situations.

  • The F character is for FIRST OF:

    • This can be used in the Day or Day-Of-Week bits

    • In the day bit, F - The CRON would run the first day of the month

  • The L character is for LAST OF:

    • This can be used in the Day or Day-Of-Week bits

    • In the day bit, L-2 - The CRON would run the second from last day of the month

  • The # is for day-of-week selector

    • This is only used in the Day-Of-Week bit:

    • 0#2 would be the second Sunday (0 Sunday #2)

    • 1#3 would be the third Monday (1 Monday #3)

Examples

How would I run a CRON every morning at 5AM ?

• Let's leave out the seconds since we don't need that here.

• If we don't specify 0 in the minute bit the CRON it will run every minute of 5AM.

• 5AM is 5 in the hour bit.

• The key request is "every day", so let's set the day bit to an asterisk.

• Every other

0 5 * * *

How would I run a CRON at 1AM on the last Friday of every month?

• This is where that special L character comes into play.

  • Day-Of-Week starts as 0 Sunday, so Friday would be 5.

  • Asking for the last friday of the month would be L5 in day-of-week bit.

• If we don't specify 0 in the minute bit the CRON it will run every minute of 1AM.

• 1AM in the hour bit is 1

• Asterisks for the other fields

0 1 * * L5

How would I run a CRON at 1AM on the second to last day of the month?

• This is where that special L character comes into play again.

  • Asking for the second to last day of of the month would be L-2 in day bit.

• If we don't specify 0 in the minute bit the CRON it will run every minute of 1AM.

• 1AM is the hour bit is 1

• Asterisks for the other bits

0 1 L-2 * *

The SDK

Parse

Call Parse on any CRON string. It will detect if seconds are present.

var cparsed = Cron.Parse("0 1 * * 0#1");

Occurrence

Get the next occurrence from a given date or supply a different time zone to find the next occurrence in that time zone.

var nOcc = cparsed.GetNextOccurrence(DateTimeOffset.Now);

//You can also supply a timezone:
var nOcc = cparsed.GetNextOccurrence(DateTimeOffset.Now, TimeZoneInfo.Utc);

Print info

You can use that parsed CRON to display info about it.

Console.WriteLine(cparsed.ToString());
//Prints: on minute 0 on hour 1 on the 1st Sunday

Inverse Period

The only other method you may care about is the InversePeriod. InversePeriod is interesting as it takes the CRON string and determines how consecutively it runs.

var inv = Cron.Parse("* 1-2 * * *").GetInversePeriod(DateTimeOffset.Now);
Console.WriteLine($"From {inv.Item1:G} to {inv.Item2:G}");

//Prints: From 1/1/2023 1:00:00 AM to 1/1/2023 2:59:00 AM

Notice how this CRON would run from 1AM consistently at every interval (minute) inclusively to 2:59AM? This method is an interesting way to use CRON data to specify ranges of time.

CRON in Perigee

There are bunch of ways to use CRON strings in Perigee. Let's take a look at a few:

Agents

SyncAgents use CRON strings to determine when to run, and even when to create blackout periods. You can see more in the linked section. Here's a quick preview!

taskConfig.AddAgent("DailySync", "DailySync", "DailyDB", 
new SyncAgentSourceMemory("SA.json"), AgentRunCondition.RunIfPastDue,

(configAgent) => configAgent.SetSyncLimitPerDay(2).SetSync(null, "55 6 * * *"),

(ct, l, executeHandler) => 
    DailySync.sync(taskConfig, taskConfig.CTS.Token, log, executeHandler, "DailyDB"),

(ct, l, treeHandler) => { },

TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(10), false)
    .LinkToConfig("Agents:DailySyncEnabled");

BlockUntil

This is a handy way of creating an awaitable task, or thread blocking mechanism using a CRON string.

//Async wait until the 0th second of every 5th minute
await PerigeeUtil.BlockUntil("0 */5 * * * *");

//Or for non async methods
PerigeeUtil.BlockUntil("0 */5 * * * *").GetAwaiter().GetResult();

Methods using CRON

As part of the core set of events in Perigee, you can easily add CRON methods.

PerigeeApplication.ApplicationNoInit("Crons", (c) =>
{
    c.AddCRON("CronMethod", "0 */5 * * * *", (ct, l) => {
        //Run's every 5 minutes at second 0
    });

});

CRON Testing Resources

A method to parse and print

A handy little method to print the definition of a CRON string, as well as print the next occurrence:

void PrintCRON(string inputStr)
{
    Console.WriteLine(inputStr);
    var cparsed = Cron.Parse(inputStr);
    Console.WriteLine(cparsed.ToString());
    var nOcc = cparsed.GetNextOccurrence(DateTimeOffset.Now);
    Console.WriteLine(nOcc.Value.ToString("G") + Environment.NewLine);
}

A site to test CRON data

This is an amazing site to visually and instantly see the results of a CRON.

Every application design has requirements, and before starting a new project, or convert an old project you're going to want to define what those requirements are.

This is different for every organization and every process so in this walkthrough we'll take a few common examples and show how we might architect those scenarios.

Common Requirements

We need an API

Whether JSON, plaintext, xml, or file content, the built in .NET hosting with Perigee has you covered. Check out the blueprint here:

We have an important mission critical daily process that needs to succeed.

For this type of process we have several options.

• The first option is our Sync Agent:

  • This handy agent is highly configurable and can use everything from recurring timers, CRON strings, blackout periods, behavior trees, inter agent communications and more.

  • You the designer have complete control over the system and how/when it executes the processes

  • They can be easily turned on or off with the SDK.

  • They can communicate with other defined agents and use them as dependencies to execute.

  • They know how to execute behavior trees to determine run conditions

• The second option is a simple CRON method:

  • CRON methods are very lightweight and they're only job is to fire on on time.

  • All of the implementation details, retry logic, and execution is up to you.

• The third option is the Transaction Coordinator:

  • The coordinator is highly fault tolerant and is typically used with a queue of requests, however they can be used to perform a singular daily task with very high tolerance.

  • The coordinator tracks EVERY step of the process from initialization data to final completion. It stores a highly compressed file at every step of the process and can resume locally and remotely when the application or server is stopped and restarted.

  • If you need to replay the same transaction later, you're able to inherit the initial request and replay it again later

We use Salesforce and need to trigger a process on new records

If you need direct API access or to use push topics and have data streamed to your application, we have that covered here:

We have CSV files dropped in a folder every day and need to import that data

Use the DirectoryWatcher:

• It checks file locks automatically

• Verifies file size over time to prevent callbacks on actively writing content

• Asynchronous file operations

• Options for failures

We have scheduled reports that need to be run

Use the built in Scheduler:

• It allows for individual rows in a database to control when a function is in code executed.

• It tracks changes to the rows in real time and will adjust scheduled tasks accordingly when the rules change.

• Accepts a CRON string, a timezone, and a runtime type with arguments.

We need to process customer orders

This is where the Transaction Coordinator comes in.

• The coordinator is highly fault tolerant and has a built in multi-threaded processor to handle a defined number of queued transactions using the FIFO ruleset.

• The coordinator tracks EVERY step of the process from the initial JSON payload to every single operation that is performed afterwards.

• Immediately upon queuing the transaction, the system can persist that item to disk and it is tracked and maintained in a highly compressed format that is updated as the transaction progresses.

• The Coordinator tracks the responses, requests, timing and codes as part of this transactional process automatically. No more pawing through logs trying to figure out what happened.

• Retry logic, automatic transaction requeuing and a rich SDK to fine tune the steps are all available to you.

• If the downstream processor loses a day worth of data to faulty server, you're able able to simply replay an entire transaction process automatically by inheriting the initial JSON payload and walking through the same steps.

• NEVER lose an order again.

We need to import customer created Excel files

Maybe start with the API template if you want to accept a file through an API call:

Then let's use the ExcelReader class after we get the file posted to us:

• It's very fault tolerant, especially with customer created data.

• It uses very advanced data detection algorithms to scan for the actual table within a tab which can bypass headers and notes left by a user.

We need to synchronize data from a third party with a DeltaKey

We want to use three different methods to make this process easy:

1. Let's create a new Watermark that stores our DeltaKey.

   • This will automatically sync this delta key to the local disk, and restore it upon application restart.

   • It has hooks to initialize the source, and provides event handlers when the data is updated

   • It's very easy to synchronize the watermark to a remote source, like a database or the cloud.

2. Let's register our RestSharp credentials so we don't have to worry about authentication issues:

   1. This takes care of the reauthoring the token and making sure it's ready to authorize when needed.

   2. Anytime we create a new RestClient, we supply this to the Authenticator of the RestClientOptions.

3. Let's use a CRON scheduled thread to perform this operation on our schedule.

   1. We now have the DeltaKey stored, maintained and refreshed on application load and shutdown.

   2. We have the authentication token and parameters taken care of by the RestSharp Authenticators.

   3. Run the logic:

      • Simply call the third party with the token and auth ->

      • Process any data if it exists ->

      • If you processed data, update the watermark ->

      • Let the CRON fire again and repeat.

An Expression Managed Thread is called every N seconds and is expected to exit the method call. It will repeat until the application is shut down or the thread is turned off.

Most of the Demos have the property named c when registering a new PerigeeApplication. The c simply stands for config. Let's take a look at how you can access and control the managed threads within an application.

Properties

The Cancellation Token

There is a single "global" token that all other tokens are joined to. This is a great place to listen for application shutdown as it's cancelled the moment a SIG_ABORT or CONTROL-C event is received.

To get the globally registered cancellation token, it's available as such:

Names, configurations

AppName - is available for the registered application name, if used for logging or custom logic.

Args[] - is available if passed into the constructor, and is the application start parameters.

Configuration - is the IConfiguration that was loaded and built after it's Configuration phase.

ServiceProvider - The IServiceProvider after service build configuration

Event_ConfigurationUpdated - Is the event handler for subscribing to hot-reloads. You may also use the helper method c.ConfigUpdatedEvent((nconfig) => {}).

RegisteredTaskCount - The number of registered threads

Hosted

Hosted application task is respected and shut down alongside Perigee.

HostTask - When hosting an API application, supply the host task and it is stored here. The demo for this functionality can be seen here:

Secure properties

SecureValueKey - The secure key

SecureValueIV - The secure IV

SecureValueSource - The secure source, either args, environment, or direct.

To see the demo for this, check out:

Configuration Methods

IConfiguration

Configure

To override the IConfiguration, call ConfigureConfiguration(). This builds the IConfiguration with the default options and whatever you've added to the builder.

Reload Configuration Event

Get a configuration

IServiceProvider

If you need to configure the service provider, do so with the ConfigureServices() callback.

Configure and get service

Windows Startup

Registering

If you want to register an autostart for the windows platform, it does so by adding values to the registry. The app MUST be run with administrative privileges the first time this key is set.

Thread Methods

GetThreads

Get all threads from the management system

Example:

RestartAllThreads

Restarts all threads in the system. This will start stopped threads, it's intended to perform a full reboot of the application and all of the threads, regardless of their current state.

Example:

You could use the RunTime property to only restart threads that haven't been restarted in a certain period of time:

Example:

StartOrRestart

Starts or restarts a thread with the specified name.

Example:

Start

Starts a thread with the specified name.

Example:

Stop

NOTE We highly recommend using QueueStop() over Stop(). The reason is that Stop is not thread safe, and may cause issues. It exists only for niche, same thread operation. Use with caution!

Example:

QueueStop

Queues the stop of a thread, waiting for the task to complete.

Example:

IsRunning

Checks whether a thread with the specified name is running.

Example:

StartIfNotRunning

Starts a thread with the specified name if it is not already running.

Example:

GetThread

Gets a thread with the specified name, if it exists. Use with caution.

Example:

ContainsThreadByName

Checks if the collection contains a thread with the specified name.

Example:

ExecuteCRONNow

Executes a CRON thread immediately by its name. Optionally, you can provide a callback to be invoked when the CRON operation is completed and the number of seconds to wait for a start operation

Example:

Add a custom managed thread

The best example of this is shown here:

Link to Configuration

You can link managed threads to configuration values in one of two ways:

1. Use the handy .LinkToConfig() method immediately following the thread

2. Use the .LinkConfigToThread() method somewhere later

Agent Methods

Agents are explained fully over here:

Get agents

To get an agent after it's been added to the system, call GetAgent().

Configure agents

To reconfigure an agent after it's been added to the system:

Queue Remote Refresh

Queues an agent for remote refresh. The parameters are:

• The agent name.

• A boolean to say whether or not to queue if the current agent is executing.

  • If TRUE, it will wait for the current execution to finish and then trigger another exeucution.

  • If FALSE, it will not trigger another execution, but return.

• An optional wait time before giving up. If the wait time is not supplied, it will wait indefinitely.

Example:

Resources

Agents

Sync agents are a more powerful and configurable ways of defining tasks that must perform under a given sequence, dependency tree, or schedule. They can be configured in a number of ways that allow for very complex and fine grain control over how and when they execute.

A few of examples of why we would use a sync agent:

• A task or job needs to be fired at a given time

• We need the ability to schedule something in a different time zone

• We want to request that an agent performs it's refresh from a remote source (like a database)

• The task is dependant on other tasks completing

• We have data that may expire if not kept current

• We need to supply multiple CRON strings to specify which times the task runs

• We need to supply a blackout range where the task should not be executed

Agent callbacks

Agents have several main blocks to them, they are as follows:

1. The first callback is the configuration section. We can set all kinds of settings including:

   • Maximum executions per day.

   • A timespan of how often to execute

   • An array of CRON strings to specifiy when to execute

   • Blackout periods in the form of Timespan and CRON strings.

     • Setting "* 5 * * *" as a blackout CRON would disallow the agent to run during the entire fifth hour of the day (from 5:00AM to 5:59AM inclusive)

2. The execution callback

   • This callback is only ever called when the agent is in an active refresh/sync state.

   • You can perform whatever logic you need here, and simply return exec.Complete() or exec.Failure() depending on the results of your process

3. Tree check callback

   • This callback is only for late binding trees, in the below example you can see how it's used to setup a behavior tree for checking previous agent runs

Example - Three agents configuration

In the below example we configure 3 agents.

1. The first is run every 5 seconds, once a day.

2. The second is run on the minute 0 mark, once a day.

3. The level 2 agent has a late binding dependency tree to check to determine whether the first two succeeded and the data is not expired. If this is true, then it runs

Code for sources:

MSSQL

The code for the sync agent MSSQL source can be found here. If you're writing a connector to another database engine, this is a template and a guide on how to do so:

Memory

The code for the memory source is linked here. A great example of how to implement the source interface

The Scheduler is a way of declaring tasks to execute on a given schedule and time zone. It provides an easy to use SDK for scheduling, automatic de-scheduling and re-scheduling for event tasks.

Demo Code

• Line 5 - Declare a new event source. This demo uses the memory scheduler.

• Line 9-10 - Add the two scheduled items, A, and B, to schedule and execute

• Line 13 - Add a scheduler using the source we defined, and declare the callback.

  • You're given a CancellationToken for respecting graceful shutdown event.

  • An ILogger for logging to the system and defined sink sources.

  • And the GenericScheduledItem<ushort> Which is the interfaced item that allows you to access it's definition values.

• Line 14 - You can execute or perform any tasks you need. Like generating a report with parameters.

  • You can see the call to GetRunType(). There's also GetRunArgs(), GetLastRunDate(), GetName(), etc.

SDK

Memory Source

The Memory source stores all records, run times, and event descriptions on the path specified. It's a great way to test, debug, or play with the scheduler without having to stand up a database. Typically speaking for production scenarios, you would likely use a remote source, like a database or file to maintain the scheduled items.

MSSQL Source

MSSQL Source is the example remote source included with the Scheduler. This source pulls records from a table in MSSQL which allows for 1 database record to create and maintain 1 scheduled task.

Using the MSSQL source is very easy, it only required a registered connection string credential.

Generic Scheduled Item <ushort>

This The included scheduled item interfaced class. It's suitable for most scenarios, but as the whole callback process is interfaced, you're able to implement your own class to use if needed.

Below are the interfaced methods for a scheduled item. These methods provide an easy way to retrieve the relevant properties on a scheduled task regardless of where that event description came from (in a database, a file, remotely, etc).

Code for Sources

Memory:

This memory source shows the most simplified way possible of implementing the interface and can be used to learn how it works.

MSSQL:

If you're writing a custom connector to another database engine, this is a great template and guide to do so:

The Directory Notifier is able to monitor a folder and report back when a new file is present. It has several key checks in place to make sure the file isn't actively being written too, or locked by another application before executing the file ready callback.

Unlike the Directory Watch that expects the file to be processed out of the folder, notifier is intended to only signal that a file has been changed, added, or removed. It does not contain a failure policy because of this reason.

Demo Application

This demo will:

• Watch the C:\Watch folder.

• It will report on any JSON files (.*\.json$) - This uses and supports full Regex patterns.

• It will search TopDirectoryOnly (no subDirectories).

• NotifyInitial is true which tells the system to send notifications on all files currently in the path.

  • If this was false, it would not call the callback with the pre-existing files.

PerigeeApplication.ApplicationNoInit("HotReloadExample", (c) => {

    c.AddDirectoryNotifier("ReloadConfigs", @"C:\Watch", @".*\.json$", SearchOption.TopDirectoryOnly,
        (ct, l, path) => {
            //path is the full file path of the file that was modified or added, or removed.
            
            //Before loading or reading, verify it's existance:
            if (File.Exists(path))
            {
                //Added / Modified and no longer being written to
            }
            else
            {
                //Removed
            }
        },
        true, null, null, NotifyInitial: true, started: true);

});

SharePoint watcher is easy enough to setup and only requires the Graph authorization details.

The Watcher automatically polls for folder changes within a sharepoint library and gives you an SDK as well as a callback any time a new file is added or changed. You will also get the item in the callback if a field has been updated.

This allows for automated processes to be kicked off any time something within the watched SharePoint directory is modified.

Example Configuration

Here's the fully configured and ready to use snippet of setting this up within Perigee.

PerigeeApplication.ApplicationNoInit("Sharepoint Demo", (taskConfig) =>
    {
        
        taskConfig.AddSharepointWatch("Sharepoint",
            taskConfig.GetValue<string>("Sharepoint:tenant"), 
            taskConfig.GetValue<string>("Sharepoint:appID"),
            taskConfig.GetValue<string>("Sharepoint:appSecret"), 
            taskConfig.GetValue<string>("Sharepoint:site"),
            taskConfig.GetValue<string>("Sharepoint:drivePath"), 
            taskConfig.GetValue<string>("Sharepoint:listName"), 
            (ct, log, api, sync, items) =>
            {
                //Process file here
                return true;
                
            }).LinkToConfig("Sharepoint:enabled");

    });

All of the above are referencing a node in the appsettings.json file, you would need to fill this out with your own authorization details.

"Sharepoint": {
    "tenant": "guid",
    "appID": "guid",
    "appSecret": "secret",
    "drivePath": "",
    "site": "https://mydomain.sharepoint.com/sites/demo/",
    "listName": "Documents",
    "enabled": true
  },

• tenant/appID/appSecret - These will be pulled from the Azure portal app registration.

• The drivePath supplied may be blank if you're pulling files from the root of the drive. Otherwise supply the name of the folder to watch.

• The site key is the full address of the sharepoint site. It's used to perform a graph lookup of the ID records required.

• The listName is the name of the list behind the drive. If using the default document library, this will be "Documents". If you've created a custom library, something like "Input", then you will need to set the listName to that instead.

• The enabled flag is to turn the process on or off and is tied to the .LinkToConfig() line above.

SDK & Callbacks

Sync Class - Callback Property

The sync property sent back by the SharePoint watcher contains all of the relevant ID's needed to perform requests within SharePoint. Most importantly:

• ListID

• DriveID

• SiteID

• SiteName

Items callback

When new items arrive to be processed, you'll get a list of those items in that callback. Each item in the list contains all of the properties you'd need to check the item that was changed.

List<GraphAPIModel.Drive.Children> items;


item.Id;                            // The ID of the item
item.MicrosoftGraphDownloadUrl;     // The URL to download the item
item.Name;                          // The name of the item
item.CreatedBy;                     // The user referencee class of who created it
item.File;                          // If it is a file, this has the file details
item.Folder;                        // If it is a folder, this has the folder details

Change Notification

Want to check if the changed notification is a file?

foreach (var item in items)
{
    //Guard change notifications for folders
    if (item.Folder != null || item.File == null) continue;
    
    //Only process file changes
    log?.LogInformation("New file to process from {name} - {item} [{size}]", 
        item.CreatedBy.User.DisplayName, 
        item.Name, 
        item.Size);
}

Get Item + Fields

This call retrieves the item from a drive, and expands the response to include the list detail and fields as well

var details = api.GetItemWithList(sync.siteID, sync.driveID, item.Id);

The list fields are important as they allow you to get the custom field values. In this example, we have a custom field called "Status" that we can update along the process:

var status = details.listItem.fields["Status"];
//Status = "Ready"

Get Item by Path

To get an item if you already have the path.

If expand is true, you'll also receive the list item details along with the response.

api.GetItemByPath(sync.siteID, "Folder/File.txt", expandListItem = true)

Get Item by Path In Drive

To get an item if you already have the path for a given DriveID.

If expand is true, you'll also receive the list item details along with the response.

api.GetItemByPath(DriveID, "Folder/File.txt", expandListItem = true)

Generate SharePoint Download Link

You can generate a direct download link to a SharePoint item by giving the Site, library, and path to the item.

The resulting string is generated and uses the SharePoint /Download.aspx page.

api.GenerateSharepointDownloadLink(Site, "Shared Documents", "Folder/File.txt");

Delete an Item

To remove an item by ID:

api.DeleteItem(sync.siteID, ItemID);

Patch a Field

After retrieving the List Details, you can patch any of the fields with the PatchField command:

api.PatchField(sync.siteID, sync.listID, details.listItem.id, 
new { 
    Status = "Success",
    Notes = "We are done!"
    });

Get Another Drive

If you need to get a different drive, say a Processed drive, you can do so by querying for the drive:

var Drives = api.GetSiteDrives(sync.siteID);
var ProcessedDriveID = Drives.Where(f => f.name.Equals("Processed")).FirstOrDefault()?.id ?? "";

Uploading an item

To upload an item to a drive you must supply the DriveID, Name, Content, and MIME Type.

We'll upload a document to the Processed drive as shown here.

var uploaded = api.UploadFile(ProcessedDriveID, 
    "MyZip.zip", new byte[] {0x0}, "application/zip");

//Get the details from the newly uploaded item
var itemDetails = api.GetItemWithList(sync.siteID, ProcessedDriveID, uploaded.Id);

Download an Item

To download an item from SharePoint we highly recommend first getting the List Details as well as it can provide a backup download link. Then simply call download:

if (item.MicrosoftGraphDownloadUrl == null)
{
    //Try resetting it from the details list
    item.MicrosoftGraphDownloadUrl = new Uri(details.DownloadURL);
}

byte[] File = api.DownloadItem(item);

Implementing your own

To call any other method not supplied here use the internal graph call client and supply the right path, the rest will be taken care of automatically.

var response = api.RestGraphCall<GraphAPIModel.Generic.Response>(
    $"/sites/{sync.siteID}/lists/{sync.ListName}", Method.Get);

IMAP watcher is able to watch a single inbox and give a callback any time a new message is available to process.

There's a rich SDK that enables the IMAP client to respond to messages, attach images or HTML, get the body body, create reply messages, etc.

Authentication

There are two basic ways to authenticate with your IMAP server:

User/Pass

This is the "direct" method. If the server supports it, you can supply a username and password.

PerigeeApplication.ApplicationNoInit("EmailWatcher",  (c) => {
    c.AddIMAPWatcher("MailWatch", 
        "email@address.com", "MailBot", "password", 
        "hostImap", 993, 
        "smtphost", 587, 
        (ct, l, mail) => { 
            //Mail handler here!
    });
});

SASL

Simple Authentication Security Layer: This is for the authentication methods that require OAUTH2 or other methods.

//Direct creation
var saslDirect = new SaslMechanismOAuth2("", "");

//If using GMAIL, we built in the google auth flow
var googleSASL = MailWatcher.SASL_GoogleAPIS("email", "CredentialPath.json");

An example of the google SASL callback:

 PerigeeApplication.ApplicationNoInit("EmailWatcher",  (c) => {

    c.AddIMAPWatcher("MailWatch",
     "email@address.com", "MailBot",
     "hostImap", 993,
     "smtphost", 587, 
     () => MailWatcher.SASL_GoogleAPIS("email@address.com", "CredentialPath.json"),
     (ct, l, mail) => {
         //Mail handler here!
     });
});

The SDK

All of the SDK references below are available when a callback occurs for new mail messages to be processed.

Addresses

To get the various addresses:

mail.CCAddresses();
mail.FromAddresses();
mail.ToAddresses();

//Or to get to the message envelope
var env = mail.Message.Envelope;

Labels and Flags

//Add
mail.AddFlags(MailKit.MessageFlags.Answered | MailKit.MessageFlags.Seen);
mail.AddLabels("bot");

//Get
var labels = mail.GetLabels();
var flags = mail.GetFlags();

Replying

The BodyBuilder callback is how you configure the outgoing message. You can see an example here of adding an image with CIDs.

The Reply method automatically configures the method reply parameters including the correct message headers, subject, response addresses, and if includeReplyText is true, it will also quote the original message back as any normal client would do.

//Generate a reply with the correct message contents
 var mReply = mail.Reply(false, (bb) =>
 {
     //Set HTML, and text fallback content
     bb.TextBody = "text is supported";
     bb.HtmlBody = "<b>html</b> is supported";
     
     //Add an attachment
     //bb.Attachments.Add("MyPDF", File.ReadAllBytes("MyPDF.pdf"));
 }, includeReplyText: true);

 //Send
 mail.SendMessage(mReply);

Is ?

To see if the message is <FLAG>:

var isAnswered = mail.IsAnswered;
var isSeen = mail.IsSeen;
var isFlagged = mail.IsFlagged;
var isDeleted = mail.IsDeleted;
var isDraft = mail.IsDraft;

Delete Mail

To delete a message, issue the delete command. The parameter is for expunging the message from the server as well as issuing the deleted flag.

mail.DeleteMail(true);

Attachments

You can get a list of attachment parts, and iterate over them to get the actual content, mime type, name, etc.

//This does not pull the body or content, and is a fast way of checking how many attachments a message has
var attachcount = mail.Message.Attachments.Count();

//This iterates over the bodyparts to get additional information, it is slower and should be used after there are known messages
var attachments = mail.GetAttachments();
if (attachments.Any())
{
    var attach = attachments.ElementAt(0);
    var name = attach.FileName;
    using MemoryStream attachBytes = mail.GetAttachment(attach);
}

Querying

To query the inbox:

var uidsNotAnswered = mail.GetNotAnsweredUIDs();
if (uidsNotAnswered.Any())
{
   var ListMailSummaries = mail.FetchMessages(uidsNotAnswered);
}

var uidsNotSeen = mail.GetNotSeenUIDs();
if (uidsNotSeen.Any())
{
   var ListMailSummaries = mail.FetchMessages(uidsNotSeen);
}

//Or direct access querying:
var uids = mail.Folder.Search(MailKit.Search.SearchQuery.DeliveredAfter(DateTime.UtcNow.AddDays(-1)));

Sent Box

If you need to access the sent box, we provide an easy way to retrieve and open the sent mailbox.

var sentBox = mail.GetAndOpenSentFolder();
if (sentBox != null)
{
 var uids = sentBox.Search(MailKit.Search.SearchQuery.DeliveredAfter(DateTime.UtcNow.AddDays(-1)));

 //Fetch using the sentbox, as mail.FetchMessages uses the inbox.
 var ListMailSummaries = sentBox.FetchAsync(uids, mail.MessagePullItems).GetAwaiter().GetResult(); //make sure you're "using MailKit;"
}

Input Text

Sometimes you just need whatever the person said, excluding their signature and reply content. This method takes several passes at retrieving just the top level user input and skipping the rest:

string textOnlyPart = mail.GetOnlyInputTextBody();

Low Level Access

There are a bunch of prebuilt methods to help with a mail client. If you want to do something specific, you can get the client and folder access as so, and use any of the available methods from MailKit:

var imapClient = mail.Client;
var imapInbox = mail.Folder;

Best practice watching

We highly recommend:

1. Putting a stop gap on the mail receive handler, something like IsAnswered, as a way of preventing unwanted reprocessing of messages.

2. Catching the exception and attempting to mark it answered and label it error if the client supports labelling.

PerigeeApplication.ApplicationNoInit("MailDemo",  (c) => {

    c.AddIMAPWatcher("MailWatch",
        "email@address.com", "MailBot",
        "hostImap", 993,
        "smtphost", 587,
        () => MailWatcher.SASL_GoogleAPIS("email@address.com", "CredentialPath.json"),
        (ct, l, mail) => 
        {
            try
            {
                if (!mail.IsAnswered)
                {
                    //Do stuff here!
                    
                    
                    //Mark it on success
                    mail.AddFlags(MessageFlags.Answered | MailKit.MessageFlags.Seen);
                    mail.AddLabels("success");
                }
            }
            catch (Exception ex)
            {
                l.LogError(ex, "Uncaught exception in mail processor");
                try
                {
                    mail.AddFlags(MessageFlags.Answered | MailKit.MessageFlags.Seen);
                    mail.AddLabels("error");
                }
                catch (Exception) { }
            }
        });
});