Perigee Application Design

Anatomy of a Perigee Application

Perigee Applications are created around two main concepts: Threads and Events.

Events are the source for starting a process:

• An API call

• An email received

• A scheduled task at a specific date/time

• A file dropped on disk

• A new message from a queue

• A task recurring every minute

Events live and execute their code within Managed Threads.

We use the Perigee Application Callbacks to tell our application what events we want to subscribe to, and Perigee automatically manages the relevant threads for you.

Managed Threads

Threads are managed independent code blocks, completely separate from other threads. This is a key part of the application design process, as one thread can be shut down without affecting another thread. One thread can crash without taking down the application or other running threads.

Threads are managed by Perigee internally. This allows for the application to track its running state, restart them if they're down, and allows programmatic access to manage them.

Threads provide a clear separation of running code that does not allow one running block to cancel or crash another. By designing applications that use this separation of logic, we can create highly customizable, scalable, and fault-tolerant applications where a fault in one process doesn't take down other processes.

Threads are configured during the Perigee Application Callbacks phase. We have some options during this startup, and we aren't forced to start threads immediately. They may be configured to start later or from a different event.

"Real World" example - PDF Document Processor

Imagine for a moment you have a directory watcher to read all the account PDFs from a folder and securely upload them to a third party for processing. Let's take a look at the power of managed threads and how they can help you and your business.

Inherently, you've got several points of failure and things to consider in this scenario:

• The drive the documents live on may fail

• The third party may be offline, or down

• We may need to perform maintenance on the server

• There may be an emergency call to stop processing documents

All of the above scenarios are easily handled within Perigee. Let's create a watcher to notify us of a new PDF on our drive.

taskConfig.AddDirectoryWatch("DocumentProcessor", "C:\\PDFs", "*.pdf", SearchOption.TopDirectoryOnly, 
(ct, log, path) => {
    log.LogInformation("New file ready for process {file}", Path.GetFileName(path));
});

Much of the difficult code is handled within the Directory Watcher. As we progress through the "Hello" series, we will be covering all of the fault-tolerant design patterns to follow. Let's take a look at a few of the mentioned issues above:

Have an emergency call to stop processing?

Maybe you need to take that processor down and perform maintenance. You can just shut down that thread, and leave the remainder of the application running and untouched!

//Is it running?
taskConfig.IsRunning("DocumentProcessor");

//Ask it to stop
taskConfig.QueueStop("DocumentProcessor");

The way you implement this logic is up to you. Maybe you decide to expose an internal, authorized endpoint to shut down or restart that process:

[HttpGet]
[Authorize]
public IActionResult Admin_StopDocumentProcessing()
{
    if (Perigee.IsRunning("DocumentProcessor")) Perigee.QueueStop("DocumentProcessor");
    return Ok();
}

You can find this setup with .NET hosting in the Blueprints section, here.

The specific demo for the admin controller here.

Have a critical, unhandled exception that crashes the entire thread?

Maybe the drive had a fault and is now offline, which caused the thread itself to go offline. Perigee implements several powerful ways to assist in the restart of this process. Another powerful feature of threads is that we're able to set up and configure how threads should react when they crash. Below is an example of adding the optional callback for restarting a down thread.

taskConfig.AddDirectoryWatch("DocumentProcessor", 
(ct, log, path) => { },

//Restart function and paramaters //
() => { 
    //Check the hard drive?
    
    //Check the connectivity to the third party processor?
    
    //All look good? return true
    return true; 
}, 
restartTimerDelayMS: 5000, 
shutDownWaitMS: 300000,
// ----------------------------- //

started: true);

• Line 5 The first callback () => {} allows you to perform logic before telling Perigee to restart that thread. If you return true, the thread will be started again.

• Line 13 restartTimerDelay is to tell Perigee how long to wait until checking the callback again, if it returns false.

• Line 14 shutdownWaitMS is how long Perigee will wait until dropping the thread when attempting to cancel it.

• Line 17 started is an optional way to tell Perigee only to configure and store the thread, but do not start it yet. You have control over how to start this later.

Need to shut down the whole application?

Simply press ctrl-c if you're within a console application and wait while Perigee gracefully cancels all running tasks. No data will be lost, and everything will be persisted to disk. You'll see the logs indicate what is currently running, and what is awaiting shutdown.

You may want to write custom logic to shut the app down gracefully as well. Check out the Thread Registry section to see how to cancel the parent token.

Events

Whether you are writing an integration, data pipeline, worker bot, synchronization agent, data process, or even an order transaction coordinator, they all have an event source. There are only two types of event sources. Let's look at each of them:

Triggered

Triggered event sources are used in most types of applications. They can be new accounts created in Salesforce, an email arriving in an inbox, or a new message in a queue. They act as a starting point for additional logic and are events created outside of our control. Here are a few more examples:

• A directory watcher receiving a new file.

• An email client getting an email.

• A Kafka topic receiving a new message.

• Salesforce pushtopic getting data.

• An API with JSON data posted to it.

Scheduled

Scheduled event sources are everything not triggered by external/internal sources. Typically these are CRON jobs, although Perigee has a few additional scheduling types to use. A few examples of using these:

• We want a data process to kick off at 4 AM.

• We need to check every hour for new data on the network drive.

• We pull an external API every 5 minutes for the status update of our request

Events callback

Now that we have a basic understanding of how event sources work, let's look at what that might look like inside Perigee.

PerigeeApplication.Application("Sync Agent", "IPCSingleRunToken", (init) => {}, 
            (taskConfig) => 
            { 
                taskConfig.AddDirectoryWatch("C:\WatchMeMaybe") //Watch a directory
                .AddCRON("0 */2 * * *")  //Run every other hour
                .AddIMAPWatcher() //Watch an IMAP inbox
                .AddSalesForceClientPush() //Watch a salesforce pushtopic
                .Add() //Add your own thread managed method
            });

These are just a few of the ways you can integrate different kinds of events inside Perigee. We'll cover all of them in examples and blueprints in more detail.

Every single one of the .Add ... () methods contain a callback action of when to process an event.

• For the CRON, this callback is when we're within the time and timezone to execute.

• For the IMAP Watch, it's when we receive a new email in the inbox.

• For the Directory Watch, it's when a new file is present and is no longer being written to

All of the callbacks within perigee contain a Cancellation Token and an ILogger. See the below example:

taskConfig.Add("ManagedThreadA",
(cancelToken, logger) => { 

});

The Cancellation Token will get flagged cancelled when a shutdown event has occurred, and the process is expected to exit soon. Pay attention to this if you want to be able to gracefully shutdown, or cancel tasks in process. You can store your data or finish your process. If you're able to safely stop processing, you can respect it's request to shut down and end execution to return back to Perigee.

The ILogger is a Serilog enabled logging source, enabling you to write your logs to the console, files, external data sources, and many other sinks. Serilog is awesome, go check them out!

Perigee Application Callbacks

PerigeeApplication.Application("FirstApp", "IPCSingleRunToken", 
(init) => { 
   
}, 
(taskConfig) => { 
       
});#

Line 2 - The initialization callback is how we inject and define our properties. Maybe our application requires a connection string, a custom property loader, or API credentials. This initialization block is how we inject additional configuration before the primary configuration block is called.

The init parameter is simply an IConfiguration. This allows full access to your appsettings.json file for reading any configuration values you need to set up additional components.

Line 5 - Perigee has a primary configuration callback defined every time you create an application. This block is where we define how and what the application does. More specifically, we define what event sources we use and any managed threads we want to create.

For configuring a Perigee application, we've created an easy-to-use SDK, this is why you get the taskConfig parameter, which is the ThreadRegistry.