LogoLogo
HomePricingDocumentation
  • 💿Getting Started
    • Installation and Project Setup
    • Hello Perigee!
    • Perigee Application Design
    • Hello Configuration
    • Hello Logs
    • Hello Integration
    • Troubleshooting
    • Case Studies
  • 📃License + Notice
    • 📂Licensing
    • Notice of Third Party Agreements
  • 🚀Perigee and Beyond
    • Extending - Threads
    • Extending - Loaders
    • ⏳All about CRON
  • 🔮API Generation
    • What is API Generation?
    • API Builder
  • 🗺️Architecting YOUR App
    • Design and Requirements
    • Define Sources
    • Requirements
  • 🧩Core Modules
    • 🌐PerigeeApplication
    • 🪡Thread Registry
    • Event Sources
      • Scheduled/Logic
        • CRON Thread
        • Scheduler
        • Sync Agent
      • Watchers
        • SalesForce
        • Sharepoint
        • Directory Watch
        • Directory Notifier
        • IMAP
    • Credential Management
      • Connection Strings
      • Custom Refresh Logic
      • RestSharp Authenticator
      • Credential Store SDK
      • ⁉️Troubleshooting Credentials
    • Integration Utilities
      • HTTP(S) - RestSharp
      • Transaction Coordinator
      • Limiter
      • Watermarking
    • Alert Managers
      • SMS
      • Email
      • Discord
      • Teams
    • File Formats
      • Excel
      • CSV
    • 📁File System Storage
      • File Revision Store
      • Concurrent File Store
      • FileSync + Cache
    • Third Party
      • SmartSheets
      • Microsoft Graph
    • Perigee In Parallel
      • Parallel Processing Reference
      • Extensions
      • GroupProcessor
      • SingleProcessor
    • 🧱Utility Classes
      • Metrics
      • F(x) Expressions
      • Multi-Threaded Processor (Scatter Gather)
      • OpenAI - GPT
      • XML Converter
      • Dynamic Data Table
      • Debounce
      • Thread Conditions
      • Perigee Utility Class
      • Network Utility
      • Lists
      • FileUtil
      • Inclusive2DRange
      • Strings, Numbers, Dates
      • Nested Sets
      • Behavior Trees
      • JsonCompress
      • Topological Sorting
      • DBDownloader
    • 🈁Bit Serializer
  • 📣Examples and Demos
    • API + Perigee
    • 📰Excel Quick Load
    • SalesForce Watcher
    • Report Scheduler
    • Agent Data Synchronization
    • 📩IMAP Echo bot
    • Watch and load CSVs
    • Graph Delegated Authorization + DataVerse
    • Coordinator Demo
    • Azure Service Bus
    • QuickBooks Online
  • 📘Blueprints
    • Perigee With .NET Hosting
    • Web Host Utilities
    • 🔌Plugin Load Context
  • 🎞️Transforms
    • 🌟What is Transforms?
    • 📘Terminology
    • 🦾The Mapping Document
    • 👾Transformation Process
    • 😎Profile
    • 🎒Automation
      • 🕓Package Options
      • 🔳Configuration
    • 🔧Utilities
      • 🧹Clean
      • 📑Map File
      • 🔎File Identification
      • 🗺️Map Generation
      • 🪅Insert Statement Generation
  • 🗃️Transform SDK
    • 👋Quick Start Guide
    • 🥳MapTo
    • 🔌Authoring Plugins
      • 🔘File IO Process
      • 📢Data Quality
      • 🟢Transform Process
    • SDK Reference
      • 🔘FileIOProcessData
      • 📢DataQualityContext
      • 🎛️TransformDataContext
      • 🏅TransformResult
Powered by GitBook
On this page
  • Use Cases
  • Creating a plugin project
  • Authoring the plugin
  • Attribute
  • Interface
  • SDK
  • Running FIOPS Manually (SDK)
  • Installation in Client App
Export as PDF
  1. Transform SDK
  2. Authoring Plugins

File IO Process

PreviousAuthoring PluginsNextData Quality

Last updated 11 months ago

The File IO Process is the most versatile process of the plugins as it isn't restricted to performing a single action (like checking data quality). The File IO Process is completely open to you on how you modify the incoming or outgoing data,

  • Need to modify the data before it's ever touched by the Transforms? Use .

  • Need to modify the data, maps, options, etc after Transforms has successfully loaded the data into a table? Use the

  • Need to generate data quality reporting? Use .

Use Cases

File IO is perfect when:

  • The incoming data is not recognizable from the file read methods.

  • The file is plaintext, a flat file, XML, or other format that requires a pre-transform to generate an appropriate table.

  • The outgoing data must be modified in a unique way.

Creating a plugin project

If you would like to actually create a plugin library (dll project), follow these steps first and we'll put our code here. Otherwise, , and create the code directly within your project.

  1. Create a new DLL project, and for the time being, set the framework to net6.0.

  2. Install the latest version of Perigee using install-package perigee - OR use Nuget Package Manager.

  3. Open the .proj file by double clicking on the DLL project in your code editor. You should see the XML for the project below.

  4. The two changes you need to make are:

    • Add the <EnableDynamicLoading>true</EnableDynamicLoading> to the PropertyGroup tag

    • For the PackageReferences, add <Private>false</Private and <ExcludeAssets>runtime</ExcludeAssets>

<Project Sdk="Microsoft.NET.Sdk">

	<PropertyGroup>
		<EnableDynamicLoading>true</EnableDynamicLoading>
		<TargetFramework>net6.0</TargetFramework>
		<ImplicitUsings>enable</ImplicitUsings>
		<Nullable>enable</Nullable>
	</PropertyGroup>

	<ItemGroup>
		<PackageReference Include="perigee" Version="24.6.1.1">
			<Private>false</Private>
			<ExcludeAssets>runtime</ExcludeAssets>
		</PackageReference>
	</ItemGroup>

</Project>

That's it! You've created a new DLL Project that when built, will produce a plugin.dll that Transforms is able hot reload and run dynamically at runtime.

Authoring the plugin

The plugin can contain many file IO processes. Each process is defined by a method, and an attribute. Here's what a new process for FlatFileSplitter looks like:

[FileIOProcess(true, "Sample: FlatFileSplitter")]
public class FlatFileSplitter : IFileIOProcess
{
    public void ProcessData(FileIOProcessData data)
    {
        //Process!
    }
}

Attribute

The [attribute] tells the system two important things, in the order shown above, they are:

  1. Active? - Should the plugin loader use this plugin, is it active? Or is this in development or unavailable.

  2. Name - What name is this plugin given? Although it may not be shown anywhere immediately when running locally, the name is used for debugging and shown in certain log messages.

Other optional attribute values you can supply are:

  • AutoRun (false|true) - If this is true, this IO process is an auto-start process, meaning it doesn't require an explicit reference to trigger. These types of processes are not as common, but allow you to ALWAYS run this process any time a file is being processed and you can optionally modify it

  • IsPreTransform (false|true) - This is typically true, meaning this process is run before the transformation occurs. If you're writing a process to modify the transformed results, then set this to false.

  • SortOrder (int) - When multiple IO steps are defined and activated, which order (ascending) are they run in?

Interface

The IFileIOProcess interface gives the method all of the required data it needs to process the file.

Here's a quick example of the powerful toolset provided by this interface. This will split a flat file with two columns into a data table that can be read by the transformer

[FileIOProcess(true, "Sample: FlatFileSplitter")]
public class FlatFileSplitter : IFileIOProcess
{
    public void ProcessData(FileIOProcessData data)
    {
        if (!data.IsTextMime) return;

        //Convert a flat file to a CSV
        DynamicDataTable DDT = new();
        data.BytesToString().Split(new char[] { '\r', '\n' },
            StringSplitOptions.RemoveEmptyEntries).ForEach((uRow, r) =>
            {
                DDT.AddRowValues((uint)uRow, r.Substring(0, 30).Trim(), r.Substring(30, 30).Trim());
            });

        //Write
        data.SetAsDataTable(DDT.FinishDataLoad().ToDataTable());
    }
}

SDK

To see all of the available methods, properties, and helpers, check out the SDK page:

Running FIOPS Manually (SDK)

Running the processes is up to you and what processes to even run. Here's a quick snippet of running all of the defined modules in the current assembly.

//Assign FIOData
var FIOData = new FileIOProcessData()
{
    FileData = File.ReadAllBytes("file.csv"),
    FileMime = TransformUtil.ToMimeType(Path.GetExtension("file.csv")),
    FileName = "file",
    SignificantDigits = 5
};

//Iterate modules and run in order
foreach (var fi in FileIOProcess.GetModules().OrderBy(f => f.Attribute.SortOrder))
    fi.RunForInstance(fi.Instance, FIOData);

//FIOData convert to table (or read direct table if it exists)
var dt = FIOData.TableFromFile();

Installation in Client App

If you created a plugin.dll project: Compile the project and drop the .dll into the Plugins/IO folder.

If you wrote the process in the same project as you're running, the plugin loader will automatically take a scan of the assembly and the plugin is available for use.

🗃️
🔌
🔘
🔘FileIOProcessData
Transform Process
Data Quality
File IO
skip this step