The Directory Watch is able to monitor a folder and report back when a new file is present. It has several key checks in place to make sure the file isn't actively being written too, or locked by another application before executing the file ready callback.
Directory Watch expects that the file will be removed from the directory after it has been processed. For this reason there are several options on the DirectoryWatchFailurePolicy on how to handle when a file hasn't been removed.
Directory Watch is great for file processing where the file is expected to be removed.
is great for supporting hot-reload at runtime.
This demo will:
Watch the C:\Watch folder
It will report on any CSV files (*.csv)
It will search AllDirectories (subDirectoties) as well as it's root
As a bonus, we'll in and report on it's load.
When the above example run's against this file, it will log:
At the moment, Perigee contains 3 different failure policies:
Redelivery after a certain period of time
Move to Failed Folder (_Failed)
Delete removes the file from the system.
The DirectoryWatchFailurePolicy is set to move the file to a _Failed folder if the file has not been moved or deleted after the callback has completed.
PerigeeApplication.ApplicationNoInit("Watcher Demo", (c) =>
{
c.AddDirectoryWatch("CSV", "C:\\Watch", "*.csv", SearchOption.AllDirectories, (ct, l, path) => {
//Read the CSV
var CSVData = CSVReader.ToDataTable(path, out var rRes);
//Reprt on it
l.LogInformation("Read CSV {file}[{encoding}]. Columns/Rows: {col}/{row}; Delimiter: {delChar}; Jagged? {jagged}",
Path.GetFileName(path), rRes.FileEncoding.EncodingName, rRes.ColumnCount,
CSVData.Rows.Count, rRes.FinalDelimiter, rRes.RowShifts.Count > 0 ? "YES" : "NO");
//You'll notice the file gets moved to the _Failed Folder (Due to DirectoryWatchFailurePolicy supplied below)
// Watcher expects the file to be removed after it's processed to prevent infinite loops
}, policy: ThreadRegistry.DirectoryWatchFailurePolicy.MoveToFailedFolder);
});Read CSV NewCSV.csv[US-ASCII]. Columns/Rows: 2/2; Delimiter: ','; Jagged? NOIMAP watcher is able to watch a single inbox and give a callback any time a new message is available to process.
There's a rich SDK that enables the IMAP client to respond to messages, attach images or HTML, get the body body, create reply messages, etc.
There are two basic ways to authenticate with your IMAP server:
This is the "direct" method. If the server supports it, you can supply a username and password.
Simple Authentication Security Layer: This is for the authentication methods that require OAUTH2 or other methods.
An example of the google SASL callback:
All of the SDK references below are available when a callback occurs for new mail messages to be processed.
To get the various addresses:
The BodyBuilder callback is how you configure the outgoing message. You can see an of adding an image with CIDs.
The Reply method automatically configures the method reply parameters including the correct message headers, subject, response addresses, and if includeReplyText is true, it will also quote the original message back as any normal client would do.
To see if the message is <FLAG>:
To delete a message, issue the delete command.
The parameter is for expunging the message from the server as well as issuing the deleted flag.
You can get a list of attachment parts, and iterate over them to get the actual content, mime type, name, etc.
To query the inbox:
If you need to access the sent box, we provide an easy way to retrieve and open the sent mailbox.
IMAP only allows a single mailbox to be open at once. Don't forget to call:
mail.OpenInbox();.
This verifies and prevents future issues with any subsequent calls to the mail client.
Sometimes you just need whatever the person said, excluding their signature and reply content. This method takes several passes at retrieving just the top level user input and skipping the rest:
There are a bunch of prebuilt methods to help with a mail client. If you want to do something specific, you can get the client and folder access as so, and use any of the available methods from :
We highly recommend:
Putting a stop gap on the mail receive handler, something like IsAnswered, as a way of preventing unwanted reprocessing of messages.
Catching the exception and attempting to mark it answered and label it error if the client supports labelling.
PerigeeApplication.ApplicationNoInit("EmailWatcher", (c) => {
c.AddIMAPWatcher("MailWatch",
"[email protected]", "MailBot", "password",
"hostImap", 993,
"smtphost", 587,
(ct, l, mail) => {
//Mail handler here!
});
});//Direct creation
var saslDirect = new SaslMechanismOAuth2("", "");
//If using GMAIL, we built in the google auth flow
var googleSASL = MailWatcher.SASL_GoogleAPIS("email", "CredentialPath.json"); PerigeeApplication.ApplicationNoInit("EmailWatcher", (c) => {
c.AddIMAPWatcher("MailWatch",
"[email protected]", "MailBot",
"hostImap", 993,
"smtphost", 587,
() => MailWatcher.SASL_GoogleAPIS("[email protected]", "CredentialPath.json"),
(ct, l, mail) => {
//Mail handler here!
});
});mail.CCAddresses();
mail.FromAddresses();
mail.ToAddresses();
//Or to get to the message envelope
var env = mail.Message.Envelope;//Add
mail.AddFlags(MailKit.MessageFlags.Answered | MailKit.MessageFlags.Seen);
mail.AddLabels("bot");
//Get
var labels = mail.GetLabels();
var flags = mail.GetFlags();//Generate a reply with the correct message contents
var mReply = mail.Reply(false, (bb) =>
{
//Set HTML, and text fallback content
bb.TextBody = "text is supported";
bb.HtmlBody = "<b>html</b> is supported";
//Add an attachment
//bb.Attachments.Add("MyPDF", File.ReadAllBytes("MyPDF.pdf"));
}, includeReplyText: true);
//Send
mail.SendMessage(mReply);var isAnswered = mail.IsAnswered;
var isSeen = mail.IsSeen;
var isFlagged = mail.IsFlagged;
var isDeleted = mail.IsDeleted;
var isDraft = mail.IsDraft;mail.DeleteMail(true);//This does not pull the body or content, and is a fast way of checking how many attachments a message has
var attachcount = mail.Message.Attachments.Count();
//This iterates over the bodyparts to get additional information, it is slower and should be used after there are known messages
var attachments = mail.GetAttachments();
if (attachments.Any())
{
var attach = attachments.ElementAt(0);
var name = attach.FileName;
using MemoryStream attachBytes = mail.GetAttachment(attach);
}var uidsNotAnswered = mail.GetNotAnsweredUIDs();
if (uidsNotAnswered.Any())
{
var ListMailSummaries = mail.FetchMessages(uidsNotAnswered);
}
var uidsNotSeen = mail.GetNotSeenUIDs();
if (uidsNotSeen.Any())
{
var ListMailSummaries = mail.FetchMessages(uidsNotSeen);
}
//Or direct access querying:
var uids = mail.Folder.Search(MailKit.Search.SearchQuery.DeliveredAfter(DateTime.UtcNow.AddDays(-1)));var sentBox = mail.GetAndOpenSentFolder();
if (sentBox != null)
{
var uids = sentBox.Search(MailKit.Search.SearchQuery.DeliveredAfter(DateTime.UtcNow.AddDays(-1)));
//Fetch using the sentbox, as mail.FetchMessages uses the inbox.
var ListMailSummaries = sentBox.FetchAsync(uids, mail.MessagePullItems).GetAwaiter().GetResult(); //make sure you're "using MailKit;"
}string textOnlyPart = mail.GetOnlyInputTextBody();var imapClient = mail.Client;
var imapInbox = mail.Folder;PerigeeApplication.ApplicationNoInit("MailDemo", (c) => {
c.AddIMAPWatcher("MailWatch",
"[email protected]", "MailBot",
"hostImap", 993,
"smtphost", 587,
() => MailWatcher.SASL_GoogleAPIS("[email protected]", "CredentialPath.json"),
(ct, l, mail) =>
{
try
{
if (!mail.IsAnswered)
{
//Do stuff here!
//Mark it on success
mail.AddFlags(MessageFlags.Answered | MailKit.MessageFlags.Seen);
mail.AddLabels("success");
}
}
catch (Exception ex)
{
l.LogError(ex, "Uncaught exception in mail processor");
try
{
mail.AddFlags(MessageFlags.Answered | MailKit.MessageFlags.Seen);
mail.AddLabels("error");
}
catch (Exception) { }
}
});
});The PerigeeSalesForceClient is an easy to use implementation on top of NetCoreForce that allows you to connect and authorize easily. It manages the token refreshes and local disk persistence as well as provides several additional methods for easy communication to SalesForce.
We have used and implemented custom logic into NetCoreForce. To get more information about it's functionality the client documentation can be
There are two ways to authorize and keep the connection automatically active.
The Consumer Key and Consumer Secret are from the connected application
Username (or password if using UserPass flow) are the login user details
If using JWT, you'll supply the Certificate and optional password that was uploaded to the connected application
Once connected with either of the two methods mentioned above, you'll have full access to the available commands!
To get to the connected applications in SalesForce, go to "Setup => Build => Create => Apps"
Under the "Connected Apps" Create or Edit one
Enter a Name and Contact Email
Click the box: Enable OAuth Settings
Our little demo model class is available here. As you can see, we've used Newtonsoft to remap "Name" to "AccountName".
To perform a single describe call:
There are a bunch of ways to get objects, including for them. You may also supply direct IDS in a few ways as well:
If you're going to be calling the auto-mapped version frequently, please cache the map results and supply them to the GetObjectsAsync call as shown:
We provide an asynchronous block that automatically does all the date time handling, offsets, delays, authentication paramaters, etc. You can easily start or stop this process and it will pick up from the last set of records that were sent through.
To add a watch directly from Perigee Startup as a Managed Thread:
Make sure to configure your own:
Certificate
There are two easy ways to update a single record
To execute a query, simply supply the query along with a class to map back to:
For the full list of methods and other ways of working with SalesForce, please visit the !
https://login.salesforce.com/services/oauth2/callbackClick Use digital signatures
Upload a certificate (.cer file) (it can be self-signed)
One option for creating a certificate is the Java Key Tool.
Perigee ships with a CertGen as well, code is linked below
Add Selected OAuth Scopes.
An example of these scopes might be:
full
api
refresh_token,offline_access
Click Save
There's a button called: "Manage Consumer Details" - This will show your consumer key and consumer secret. Store them.
Click the "Manage" button at the top of the connected app, and then "Edit policies"
Setup the polices as you need.
An example of this would be:
Permitted Users: "Admin Approved users are pre-authorized"
IP Relaxation: "Relax IP restrictions"
Consumer key
Consumer secret
User
Domain
//Username Password
var client = PerigeeSalesForceClient.UsernamePassword(
consumerKey, consumerSecret,
user, pass);
//JWT
var client = PerigeeSalesForceClient.JWT(
consumerKey, consumerSecret,
user, new X509Certificate2("SF.pfx", "ABCD123"),
"login");var x5 = CertGen2.SelfSigned();
CertGen2.SaveX509ToPath(x5, "C:\\SF", "SF", "ABCDEFG12345");public class SFAccount
{
public string id { get; set; }
[Newtonsoft.Json.JsonProperty("Name")]
public string AccountName { get; set; }
public string Type { get; set; }
}var res = client.GetObjectDescribe("Account").GetAwaiter().GetResult();//Get a single object by ID, mapped back, supplying the fields
SFAccount byID = client.GetObjectById<SFAccount>("Account", "001f400000Mk8lPAAR", new List<string>() { "id", "name" }).GetAwaiter().GetResult();
//Get a list of objects by ID, dynamically automapped from your class to SalesForce
List<SFAccount> objectsByID = client.GetObjectsAsync<SFAccount>("Account", new List<string>() { "001f400000Mk8lPAAR", "001f400001N4NXqAAN" }).GetAwaiter().GetResult();//Store the cache somewhere static
var CachedProperties = client.MapProperties<SFAccount>("Account").Values.ToList();
//Supply cache map on frequent and subsequent calls
List<SFAccount> objectsByID = client.GetObjectsAsync<SFAccount>("Account", new List<string>() { "001f400000Mk8lPAAR", "001f400001N4NXqAAN" }, CachedProperties).GetAwaiter().GetResult();client.WatchAsync<SFAccount>("Account", CTS.Token, (ct, updated, deleted) => {
//updated records, bound back to an SFAccount class
foreach (var item in updated) {
Console.WriteLine($"[{item.id}] {item.AccountName}({item.Type})"); }
//Deleted ID's, along with the deleted date
foreach (var item in deleted) {
Console.WriteLine($"{item.id} - {item.deletedDate:G}"); }
}).GetAwaiter().GetResult();PerigeeApplication.ApplicationNoInit("SalesForce Demo", (c) => {
c.AddSalesForceWatch<SFAccount>("WatchAccounts", "Account", cKey, cSec, sUser, x5092Cert, "login",
(ct, log, updated, deleted) => {
//updated records, bound back to an SFAccount class
foreach (var item in updated)
{
Console.WriteLine($"[{item.id}] {item.AccountName}({item.Type})");
}
//Deleted ID's, along with the deleted date
foreach (var item in deleted)
{
Console.WriteLine($"{item.id} - {item.deletedDate:G}");
}
});
});//Update by bound object properties, serialized through newtonsoft
client.UpdateRecord<SFAccount>("Account", "001f400000Mk8lPAAR", new SFAccount() { Type = "Prospect" }).GetAwaiter().GetResult();
//Supply a dyanamic object or class, unbound
client.UpdatePatch("Account", "001f400000Mk8lPAAR", new { Rating = "Warm" }).GetAwaiter().GetResult();var accounts = client.Query<SFAccount>("SELECT id,name from account limit 10").GetAwaiter().GetResult();SharePoint watcher is easy enough to setup and only requires the Graph authorization details.
The Watcher automatically polls for folder changes within a sharepoint library and gives you an SDK as well as a callback any time a new file is added or changed. You will also get the item in the callback if a field has been updated.
This allows for automated processes to be kicked off any time something within the watched SharePoint directory is modified.
Here's the fully configured and ready to use snippet of setting this up within Perigee.
All of the above are referencing a node in the appsettings.json file, you would need to fill this out with your own authorization details.
tenant/appID/appSecret - These will be pulled from the Azure portal app registration.
The drivePath supplied may be blank if you're pulling files from the root of the drive. Otherwise supply the name of the folder to watch.
The site key is the full address of the sharepoint site. It's used to perform a graph lookup of the ID records required.
The sync property sent back by the SharePoint watcher contains all of the relevant ID's needed to perform requests within SharePoint. Most importantly:
ListID
DriveID
SiteID
SiteName
Items callbackWhen new items arrive to be processed, you'll get a list of those items in that callback. Each item in the list contains all of the properties you'd need to check the item that was changed.
To see the full list and response please check out the
Want to check if the changed notification is a file?
This call retrieves the item from a drive, and expands the response to include the list detail and fields as well
The list fields are important as they allow you to get the custom field values. In this example, we have a custom field called "Status" that we can update along the process:
To get an item if you already have the path.
If expand is true, you'll also receive the list item details along with the response.
To get an item if you already have the path for a given DriveID.
If expand is true, you'll also receive the list item details along with the response.
You can generate a direct download link to a SharePoint item by giving the Site, library, and path to the item.
The resulting string is generated and uses the SharePoint /Download.aspx page.
To remove an item by ID:
After retrieving the , you can patch any of the fields with the PatchField command:
If you need to get a different drive, say a Processed drive, you can do so by querying for the drive:
To upload an item to a drive you must supply the DriveID, Name, Content, and MIME Type.
We'll upload a document to the .
To download an item from SharePoint we highly recommend first getting the as well as it can provide a backup download link. Then simply call download:
To call any other method not supplied here use the internal graph call client and supply the right path, the rest will be taken care of automatically.
The Directory Notifier is able to monitor a folder and report back when a new file is present. It has several key checks in place to make sure the file isn't actively being written too, or locked by another application before executing the file ready callback.
Unlike the Directory Watch that expects the file to be processed out of the folder, notifier is intended to only signal that a file has been changed, added, or removed. It does not contain a failure policy because of this reason.
Directory Notifier is great for supporting hot-reload at runtime.
is great for file processing where the file is expected to be removed.
This demo will:
Watch the C:\Watch folder.
It will report on any JSON files (.*\.json$) - This uses and supports full Regex patterns.
It will search TopDirectoryOnly (no subDirectories).
PerigeeApplication.ApplicationNoInit("Sharepoint Demo", (taskConfig) =>
{
taskConfig.AddSharepointWatch("Sharepoint",
taskConfig.GetValue<string>("Sharepoint:tenant"),
taskConfig.GetValue<string>("Sharepoint:appID"),
taskConfig.GetValue<string>("Sharepoint:appSecret"),
taskConfig.GetValue<string>("Sharepoint:site"),
taskConfig.GetValue<string>("Sharepoint:drivePath"),
taskConfig.GetValue<string>("Sharepoint:listName"),
(ct, log, api, sync, items) =>
{
//Process file here
return true;
}).LinkToConfig("Sharepoint:enabled");
});
Documents". If you've created a custom library, something like "Input", then you will need to set the listName to that instead.The enabled flag is to turn the process on or off and is tied to the .LinkToConfig() line above.
NotifyInitial is true which tells the system to send notifications on all files currently in the path.
If this was false, it would not call the callback with the pre-existing files.
"Sharepoint": {
"tenant": "guid",
"appID": "guid",
"appSecret": "secret",
"drivePath": "",
"site": "https://mydomain.sharepoint.com/sites/demo/",
"listName": "Documents",
"enabled": true
},List<GraphAPIModel.Drive.Children> items;
item.Id; // The ID of the item
item.MicrosoftGraphDownloadUrl; // The URL to download the item
item.Name; // The name of the item
item.CreatedBy; // The user referencee class of who created it
item.File; // If it is a file, this has the file details
item.Folder; // If it is a folder, this has the folder detailsforeach (var item in items)
{
//Guard change notifications for folders
if (item.Folder != null || item.File == null) continue;
//Only process file changes
log?.LogInformation("New file to process from {name} - {item} [{size}]",
item.CreatedBy.User.DisplayName,
item.Name,
item.Size);
}var details = api.GetItemWithList(sync.siteID, sync.driveID, item.Id);var status = details.listItem.fields["Status"];
//Status = "Ready"api.GetItemByPath(sync.siteID, "Folder/File.txt", expandListItem = true)api.GetItemByPath(DriveID, "Folder/File.txt", expandListItem = true)api.GenerateSharepointDownloadLink(Site, "Shared Documents", "Folder/File.txt");api.DeleteItem(sync.siteID, ItemID);api.PatchField(sync.siteID, sync.listID, details.listItem.id,
new {
Status = "Success",
Notes = "We are done!"
});var Drives = api.GetSiteDrives(sync.siteID);
var ProcessedDriveID = Drives.Where(f => f.name.Equals("Processed")).FirstOrDefault()?.id ?? "";var uploaded = api.UploadFile(ProcessedDriveID,
"MyZip.zip", new byte[] {0x0}, "application/zip");
//Get the details from the newly uploaded item
var itemDetails = api.GetItemWithList(sync.siteID, ProcessedDriveID, uploaded.Id);if (item.MicrosoftGraphDownloadUrl == null)
{
//Try resetting it from the details list
item.MicrosoftGraphDownloadUrl = new Uri(details.DownloadURL);
}
byte[] File = api.DownloadItem(item);var response = api.RestGraphCall<GraphAPIModel.Generic.Response>(
$"/sites/{sync.siteID}/lists/{sync.ListName}", Method.Get);PerigeeApplication.ApplicationNoInit("HotReloadExample", (c) => {
c.AddDirectoryNotifier("ReloadConfigs", @"C:\Watch", @".*\.json$", SearchOption.TopDirectoryOnly,
(ct, l, path) => {
//path is the full file path of the file that was modified or added, or removed.
//Before loading or reading, verify it's existance:
if (File.Exists(path))
{
//Added / Modified and no longer being written to
}
else
{
//Removed
}
},
true, null, null, NotifyInitial: true, started: true);
});