🔘File IO Process
The File IO Process is the most versatile process of the plugins as it isn't restricted to performing a single action (like checking data quality). The File IO Process is completely open to you on how you modify the incoming or outgoing data,
Need to modify the data before it's ever touched by the Transforms? Use File IO.
Need to modify the data, maps, options, etc after Transforms has successfully loaded the data into a table? Use the Transform Process
Need to generate data quality reporting? Use Data Quality.
Use Cases
File IO is perfect when:
The incoming data is not recognizable from the file read methods.
The file is plaintext, a flat file, XML, or other format that requires a pre-transform to generate an appropriate table.
The outgoing data must be modified in a unique way.
Creating a plugin project
If you would like to actually create a plugin library (dll
project), follow these steps first and we'll put our code here. Otherwise, skip this step, and create the code directly within your project.
Create a new DLL project, and for the time being, set the framework to
net6.0
.Install the latest version of Perigee using
install-package perigee
- OR use Nuget Package Manager.Open the
.proj
file by double clicking on the DLL project in your code editor. You should see the XML for the project below.The two changes you need to make are:
Add the
<EnableDynamicLoading>true</EnableDynamicLoading>
to thePropertyGroup
tagFor the
PackageReferences
, add<Private>false</Private
and<ExcludeAssets>runtime</ExcludeAssets>
That's it! You've created a new DLL Project that when built, will produce a plugin.dll
that Transforms is able hot reload and run dynamically at runtime.
Authoring the plugin
The plugin can contain many file IO processes. Each process is defined by a method, and an attribute. Here's what a new process for FlatFileSplitter
looks like:
Attribute
The [attribute]
tells the system two important things, in the order shown above, they are:
Active? - Should the plugin loader use this plugin, is it active? Or is this in development or unavailable.
Name - What name is this plugin given? Although it may not be shown anywhere immediately when running locally, the name is used for debugging and shown in certain log messages.
Other optional attribute values you can supply are:
AutoRun (false|true) - If this is
true
, this IO process is an auto-start process, meaning it doesn't require an explicit reference to trigger. These types of processes are not as common, but allow you to ALWAYS run this process any time a file is being processed and you can optionally modify itIsPreTransform (false|true) - This is typically
true
, meaning this process is run before the transformation occurs. If you're writing a process to modify the transformed results, then set this tofalse
.SortOrder (int) - When multiple IO steps are defined and activated, which order (ascending) are they run in?
Interface
The IFileIOProcess
interface gives the method all of the required data it needs to process the file.
Here's a quick example of the powerful toolset provided by this interface. This will split a flat file with two columns into a data table that can be read by the transformer
SDK
To see all of the available methods, properties, and helpers, check out the SDK page:
Running FIOPS Manually (SDK)
Running the processes is up to you and what processes to even run. Here's a quick snippet of running all of the defined modules in the current assembly.
Installation in Client App
If you created a plugin.dll
project: Compile the project and drop the .dll
into the Plugins/IO
folder.
If you wrote the process in the same project as you're running, the plugin loader will automatically take a scan of the assembly and the plugin is available for use.
Last updated