Loading data into LUSID

There are several ways to load data into LUSID's numerous storage options. Some are more suited to bulk loading data than others.

Note: When you load data into LUSID there may be a small propagation delay before it can be retrieved unless you specify the recorded asAt datetime of the upsert operation when you make a List* API call (or equivalent).

LUSID REST API

The LUSID REST API has an upsert model for entities. For example, the UpsertTransactions API:

  • Updates an existing transaction if it exists, or

  • Creates a new one if it does not.

LUSID does not generally have separate update and insert APIs.

Note that the REST API has restrictions on:

The alternatives to the REST API below might be better suited to bulk loading data into LUSID.

LUSID SDK

The LUSID SDK is available in 5 programming languages and frameworks (Python, C#, Java, JavaScript and Angular) and provides a convenient wrapper around the REST API.

You may prefer to use an SDK over Luminesce (see below) if your environment is suited to manipulating data using one of the languages/frameworks listed above rather than SQL.

LUSID Python Tools

LUSID Python Tools (LPT) is a wrapper around the LUSID Python SDK (above), with tooling for loading of Pandas DataFrames, CSV files and Excel files.

Reasons to prefer using LPT over the LUSID Python SDK to bulk load data include:

  • Your preferred language for manipulating data is Python.

  • LPT handles batching automatically.

  • LPT makes asynchronous calls to improve performance.

  • LPT streamlines the process of creating properties for entities if they do not already exist.

Luminesce

Luminesce is LUSID's data virtualisation engine and is optimised for bulk loading data in the following formats and storage locations into LUSID:

  • CSV, Excel, SQLite, XML, Parquet and plain text files stored in Drive, LUSID's file repository and manager. More information.

  • Relational data stored in MySql, Odbc, Oracle, PostgreSql, SqLite, SqlServer, GoogleBigQuery, Db2 or AwsAthena SQL-compatible databases. More information.

  • CSV, Excel, SQLite, XML, Parquet and plain text files stored locally, in a NAS or similar device. More information.

  • CSV, Excel, SQLite, XML, Parquet and plain text files hosted on a SFTP server. More information.

  • CSV, Excel, SQLite, XML, Parquet and plain text files stored in AWS S3. More information coming soon.

  • Snowflake. More information coming soon.

Reasons to use Luminesce include:

  • Your data exists in the storage locations and data formats listed above.

  • Your preferred language for manipulating data is SQL.

  • Your payloads exceed 50MB.

  • Luminesce handles batching automatically.

For a demonstration of Luminesce please reach out to your FINBOURNE representative.

Horizon

Horizon is FINBOURNE's partnership program for industry-leading data vendors and service providers. It consists of utilities optimised for the bulk loading of subscription data into LUSID.

Utilities for Copp Clark market holiday data and SIX Flex instrument and quote data are available now, with many more coming soon.

For more information please reach out to your FINBOURNE representative.