Import Definitions

Import Definitions are a part of our data integration technologies. It is frequently the case that you are rarely importing a single file, rather you are importing a set of related files in order for a particular process. Import Definitions are designed to address this problem by providing functionality that enables you to process importing data from a variety of sources in an ordered manner with advanced capabilities.

This provides an extremely powerful mechanism to configure a single process that can automate interfacing with any 3rd party product. If you have LemonEdge sitting on top of another accounting system, or interfacing with a client/investor system, or feeding a data warehouse, you can make sure that with one click (or on a schedule) you can always import/export all of that data seamlessly onto the platform.

Import Definition Features

Importing a single file works for ad-hoc requirements, but isn't a scalable approach when migrating a system or when integrating with other 3rd party systems on a daily basis. Import Definitions breaks everything down into steps that are executed sequentially, each step can acquire data from a variety of different sources, parse it however you like (using our Data Mappings), transform it and import it before moving onto the next step of the process.

This allows you to automate a lot of the manual work around migrations and data integrations, and alleviates the need to rely on development teams while doing so.

Steps

Import Definition Steps are designed to work in isolation, or in tandem with each other, to ensure the data is imported into the system in the manner you require. Each step can import/validate the data or run custom calculations/processes using the same functionality as our Importing routines. But by having multiple steps we enable the following additional functionality:

  • The ability to validate multiple sets of data
  • The ability to create multiple steps and batch them together as one process
    • This ensures the data/calculations are saved in one batch transaction after several steps have been processed
    • Provides a mechanism to easily deal with hierarchical dependent data
  • The ability to apply different data mappings to different import files for each step
  • The ability to use different sources for the data
    • Poll for file from ftp/cloud/etc paths
    • Retrieve data from stored procedures or other sql queries
    • Import hard coded data associated with each step (for use with testing/modelling/etc see below)
  • The ability to import csv or xml files

Custom Processes

Import Definitions allow you to create complex definitions to import data from 3rd party systems on a regular schedule. You can use our Server Tasks functionality to ensure this process is automatically run nightly (or whatever your schedule including on a custom trigger) to import data seamlessly into the platform.

As well as being able to import data from different sources you can re-purpose this functionality to regularly run hard coded import/processes. Using our Importing technology not only can you import data that will automatically update/create/delete records, but you can also run custom calculations on data should you wish too [more info]. By hardcoding data to be imported into a definition you can specify data processes and custom calculations that you wish to regularly re-run. This can be configured to automatically be triggered rather than having to actually run the definition manually.

This can go a long way to automating a lot of manual, and error prone, steps that are regularly being done by team members.

Testing

In addition to custom process you can also use Import Definitions to address the common problems with testing your solution and any new version releases. Typically month long processes are involved in testing a new product version across teams in an isolated test environment. This is often a manual and time intensive operation that involves users from every team including how the product integrates with other systems.

Import Definitions can alleviate a lot of these pain points by using the following process:

  1. Map out areas you want to test of the system using test cases that model all the data needed in the system. This can include:
    1. Creating new database entities;
    2. Create history of transaction operations;
    3. Run calculations, algorithms, and custom processes;
    4. Run Reports;
    5. Exporting data;
    6. Etc...
  2. Create a blank system.
  3. Build your test system from step 1 in the system.
  4. Use our exporting functionality (see Standard Grids and Reporting) to export all the data out of the system, and then create an import definition from all these files including steps to run any calculations, algorithms and custom processes.

Now whenever you want to test a new version of the product you simply need follow these steps:

  1. Create a new blank system.
  2. Import the Import Definition you created in step 4 above
  3. Run the import definition
  4. Use our Server Tasks to run the task that compares databases to compare this one with the one you created in step 3 above

If the system reports no differences then you know all you data, calculations, algorithms, processes, reports, etc are all still behaving consistently.

You can even take the above 4 steps and create an admin script to automate it using our Console application in so in one click you can automatically test new version releases on your test models.

Views

Import Definitions can be edited using the following views: