LogoLogo
HomePricingDocumentation
  • 💿Getting Started
    • Installation and Project Setup
    • Hello Perigee!
    • Perigee Application Design
    • Hello Configuration
    • Hello Logs
    • Hello Integration
    • Troubleshooting
    • Case Studies
  • 📃License + Notice
    • 📂Licensing
    • Notice of Third Party Agreements
  • 🚀Perigee and Beyond
    • Extending - Threads
    • Extending - Loaders
    • ⏳All about CRON
  • 🔮API Generation
    • What is API Generation?
    • API Builder
  • 🗺️Architecting YOUR App
    • Design and Requirements
    • Define Sources
    • Requirements
  • 🧩Core Modules
    • 🌐PerigeeApplication
    • 🪡Thread Registry
    • Event Sources
      • Scheduled/Logic
        • CRON Thread
        • Scheduler
        • Sync Agent
      • Watchers
        • SalesForce
        • Sharepoint
        • Directory Watch
        • Directory Notifier
        • IMAP
    • Credential Management
      • Connection Strings
      • Custom Refresh Logic
      • RestSharp Authenticator
      • Credential Store SDK
      • ⁉️Troubleshooting Credentials
    • Integration Utilities
      • HTTP(S) - RestSharp
      • Transaction Coordinator
      • Limiter
      • Watermarking
    • Alert Managers
      • SMS
      • Email
      • Discord
      • Teams
    • File Formats
      • Excel
      • CSV
    • 📁File System Storage
      • File Revision Store
      • Concurrent File Store
      • FileSync + Cache
    • Third Party
      • SmartSheets
      • Microsoft Graph
    • Perigee In Parallel
      • Parallel Processing Reference
      • Extensions
      • GroupProcessor
      • SingleProcessor
    • 🧱Utility Classes
      • Metrics
      • F(x) Expressions
      • Multi-Threaded Processor (Scatter Gather)
      • OpenAI - GPT
      • XML Converter
      • Dynamic Data Table
      • Debounce
      • Thread Conditions
      • Perigee Utility Class
      • Network Utility
      • Lists
      • FileUtil
      • Inclusive2DRange
      • Strings, Numbers, Dates
      • Nested Sets
      • Behavior Trees
      • JsonCompress
      • Topological Sorting
      • DBDownloader
    • 🈁Bit Serializer
  • 📣Examples and Demos
    • API + Perigee
    • 📰Excel Quick Load
    • SalesForce Watcher
    • Report Scheduler
    • Agent Data Synchronization
    • 📩IMAP Echo bot
    • Watch and load CSVs
    • Graph Delegated Authorization + DataVerse
    • Coordinator Demo
    • Azure Service Bus
    • QuickBooks Online
  • 📘Blueprints
    • Perigee With .NET Hosting
    • Web Host Utilities
    • 🔌Plugin Load Context
  • 🎞️Transforms
    • 🌟What is Transforms?
    • 📘Terminology
    • 🦾The Mapping Document
    • 👾Transformation Process
    • 😎Profile
    • 🎒Automation
      • 🕓Package Options
      • 🔳Configuration
    • 🔧Utilities
      • 🧹Clean
      • 📑Map File
      • 🔎File Identification
      • 🗺️Map Generation
      • 🪅Insert Statement Generation
  • 🗃️Transform SDK
    • 👋Quick Start Guide
    • 🥳MapTo
    • 🔌Authoring Plugins
      • 🔘File IO Process
      • 📢Data Quality
      • 🟢Transform Process
    • SDK Reference
      • 🔘FileIOProcessData
      • 📢DataQualityContext
      • 🎛️TransformDataContext
      • 🏅TransformResult
Powered by GitBook
On this page
  • Read
  • Write
  • Clean
  • Sample Data
Export as PDF
  1. Core Modules
  2. File Formats

CSV

Read

Perigee ships with several powerful CSV tools that make reading even complicated CSV's very easy and fast with memory efficient span based data reads.

//Get a memory stream to the file
using MemoryStream ms = new MemoryStream(File.ReadAllBytes("Products.csv"));

//Read the CSV as a list of T classes
List<Product> products = CSVReader.ReadAs<Product>(ms);

//Read the csv as a DataTable
DataTable ProductTable =  CSVReader.ToDataTable(ms, out var res);

When using the .ToDataTable() methods, you get the resulting parse parameters sent back as an out variable. This allows you to check it's encoding, formatting, delimiters, header row location, etc.

Write

To write CSV data supply a DataTable and any additional handlers to override specifics on a data type.

  • We'll read in the Products.CSV as our source.

  • Let's then declare a writer, and set it to BAR delimited

  • Register a decimal handler that write's any decimals with 3 digits after the period

    • The handler sends you back the object (in this case a decimal)

    • The Row Index.

    • The Column Name.

    • It expects a return value of the string converted version, as well as a boolean indicating if the transform was successful.

    • Any NON-SUCCESSFUL transformed handlers are added to the writers LoadLog property and can be viewed after conversion.

  • Then simply call .Write().

//Read products CSV from above reader demo
using MemoryStream ms = new MemoryStream(File.ReadAllBytes("Products.csv"));
DataTable ProductTable = CSVReader.ToDataTable(ms, out var res);

//Write this table back out to CSV using a custom decimal format and BAR delimited
CSVWriter writer = new CSVWriter(ProductTable, (d) => "Products").WithDelimiter("|");

//You can register booleans, strings, datetimes, decimals... etc.
writer.RegisterHandler<decimal>((dec, rowIndex, colName) => new Tuple<string, bool>(dec.ToString("N3"), true));    

//Write it out
string content = writer.Write();
writerResult.csv
Products
ID|PRODUCT|PRICE
1|MILK|8.990
2|55-INCH TV|899.990
3|BREAD|1.990

Clean

using MemoryStream ms = new MemoryStream(File.ReadAllBytes("Products.csv"));

string cleanContent = CSVReader.Clean(ms);

This transforms the very badly formatted CSV Sample Data into this:

ID,PRODUCT,PRICE
1,MILK,8.99
2,55-INCH TV @ aisle 4,899.99
3,BREAD,1.99

Sample Data

Included is the sample data and classes used above.

Products.csv
CSVData as of 1/1/2001
By: Yours Truly
ID@PRODUCT@PRICE
1@MILK@8.99
2@'55-INCH TV @ aisle 4'@899.99
3@BREAD@1.99
public class Product
{
    public int id { get; set; } = 0;
    public string product { get; set; }
    public decimal price { get; set; }
}
PreviousExcelNextFile System Storage

Last updated 2 years ago

Maybe the only thing you need to do is take the absolutely horrendous CSV data in the and just create a CSV that can be cleanly loaded into another system.

Sample Data section
🧩
Page cover image