Loading data into LUSID

There are several ways to load data into LUSID's numerous storage options. Some are more suited to bulk loading data than others.

Note: When you load data into LUSID there may be a small propagation delay before it can be retrieved unless you specify the recorded asAt datetime of the upsert operation when you make a List* API call (or equivalent).

LUSID REST API

The LUSID REST API has an upsert model for entities. For example, the UpsertTransactions API:

• Updates an existing transaction if it exists, or

• Creates a new one if it does not.

LUSID does not generally have separate update and insert APIs.

Note that the REST API has restrictions on:

• The number of entities you can upsert per API call. You must batch calls if you need to upsert more than an endpoint's maximum:

  Upsert API	Maximum # entities per call
  BatchUpsertTransactions, UpsertTransactions, BatchUpsertCorporateActions, UpsertInstrumentEvents	10,000
  UpsertInstruments, UpsertQuotes, UpsertComplexMarketData, UpsertLegalEntities, UpsertCustodianAccounts, UpsertCustomEntities	2,000
  All other upsert APIs	1

• The quality and quantity of data you can supply for entity fields.

• The size of the payload. By default, this is 50MB per API call.

• The number of API calls you can make in total per minute. By default, you are rate-limited to 1000 API calls per 60 second window.

The alternatives to the REST API below might be better suited to bulk loading data into LUSID.

LUSID SDK

The LUSID SDK is available in 5 programming languages and frameworks (Python, C#, Java, JavaScript and Angular) and provides a convenient wrapper around the REST API.

You may prefer to use an SDK over Luminesce (see below) if your environment is suited to manipulating data using one of the languages/frameworks listed above rather than SQL.

LUSID Python Tools

LUSID Python Tools (LPT) is a wrapper around the LUSID Python SDK (above), with tooling for loading of Pandas DataFrames, CSV files and Excel files.

Reasons to prefer using LPT over the LUSID Python SDK to bulk load data include:

• Your preferred language for manipulating data is Python.

• LPT handles batching automatically.

• LPT makes asynchronous calls to improve performance.

• LPT streamlines the process of creating properties for entities if they do not already exist.

Luminesce

Luminesce is LUSID's data virtualisation engine and is optimised for bulk loading data in the following formats and storage locations into LUSID:

• CSV, Excel, SQLite, XML, Parquet and plain text files stored in Drive, LUSID's file repository and manager. More information.

• Relational data stored in MySql, Odbc, Oracle, PostgreSql, SqLite, SqlServer, GoogleBigQuery, Db2 or AwsAthena SQL-compatible databases. More information.

• CSV, Excel, SQLite, XML, Parquet and plain text files stored locally, in a NAS or similar device. More information.

• CSV, Excel, SQLite, XML, Parquet and plain text files hosted on a SFTP server. More information.

• CSV, Excel, SQLite, XML, Parquet and plain text files stored in AWS S3. More information coming soon.

• Snowflake. More information coming soon.

Reasons to use Luminesce include:

• Your data exists in the storage locations and data formats listed above.

• Your preferred language for manipulating data is SQL.

• Your payloads exceed 50MB.

• Luminesce handles batching automatically.

For a demonstration of Luminesce please reach out to your FINBOURNE representative.

Partnerships and integrations

FINBOURNE has a partnership program for industry-leading data vendors and service providers. It consists of utilities optimised for the bulk loading of subscription data into LUSID.

Utilities for Copp Clark market holiday data and SIX Flex instrument and quote data are available now, with many more coming soon.

For more information please reach out to your FINBOURNE representative.

LUSID web app

The LUSID web app is a graphical user interface that provides a simplified way to load and interact with your data in LUSID.

You can use the Import Data dashboard to download an Excel or CSV template for the type of data you want to load into LUSID, populate it with your data, and upload it to LUSID. More information.

You can also use the LUSID web app to create or update entities on an ad hoc basis, for example to create a transaction portfolio.