Understanding the Luminesce platform architecture

At the simplest level, the Luminesce platform consists of four components:

Component

Summary (see this section for more detail)

Client app

An app, script or tool that enables you to write a Luminesce SQL query for one or more providers, send it to the query engine, and display the aggregated results. You can use:

  • A client app supplied by us that you either install or access via a browser.

  • A custom app, script or batch process you write yourself to call our APIs.

  • A 3rd party BI tool such as Excel, DBeaver, Power BI or Tableau that you integrate using our ODBC/JDBC drivers.

Luminesce query engine

Conducts entitlement checks, passes your query to appropriate providers, and assembles and filters results to return to the client app.

Provider

Performs a particular operation on a particular data source. A provider can:

  • Read a data source and return some or all records.

  • Analyse a data source to perform calculation, valuation or other operations, and return results.

  • Write to a data source (data must be sent as part of the query), and return a summary of operations performed, including any errors.

We host providers for the LUSID platform and for other common data sources. Alternatively, you can take some of our providers and host them yourself, or create and host your own providers.

Data source

This can be:

  • Investment data in LUSID, files stored in Drive, or system information about Luminesce itself, hosted by us.

  • Your own data stored on-premise or in the cloud.

The flow of data can be summarised as follows:

  1. You write a query for one or more providers using your choice of client app, which sends it to the query engine.

  2. The query engine federates responsibility to appropriate providers (assuming you are licensed and entitled correctly).

  3. Each provider accesses its data source and performs an appropriate read, analyse or write operation, returning results to the query engine.

  4. The query engine joins and further filters results where necessary, returning aggregated results to the client app.

For example, you could write a single Luminesce SQL query that:

  1. Reads transaction data from a file stored in Drive.

  2. Upserts transactions to a portfolio stored in LUSID.

  3. Calculates a valuation for the portfolio based on the new holdings.

  4. Extracts the valuation in tabular form for storage in your own SQL database.

The Luminesce platform in more detail

For explanation, see the notes below the diagram.

Client apps

  • If you want to query LUSID investment data or Drive files and analyse results in a GUI, sign in to the LUSID web app from your browser. A Luminesce query builder and simple charting facility are built in under the Data Virtualisation menu. Take a tour.

  • To schedule queries to run at particular or regular times, you can either access the Job Scheduling GUI built in to the LUSID web app, or call the Scheduler REST API from your custom app.

  • You can write a custom app to call your choice of the Luminesce REST API or the query engine directly. Alternatively, you can install the Python SDK or C# SDK, which calls the Luminesce REST API under-the-hood. Note calling the query engine directly requires you to open port 5671 in your firewall, but has the benefit of being more performant.

  • You can install the Luminesce CLI, a .NET application, and use it to call your choice of the Luminesce REST API or the query engine directly. Note the latter scenario requires both OAuth2 authentication and PKI certification, and so is the most secure option supplied by FINBOURNE.

  • You can install Lumipy, a Python library that makes it easy to use Luminesce as part of the Python data science ecosystem.

  • For more on using 3rd party business intelligence tools such as PowerBI, Excel or Tableau, start with our guide to setting up the ODBC/JDBC drivers.

Luminesce hub

  • The query engine:

    • Performs license and access control checks.

    • Decomposes queries received from authorised client apps and sends appropriate portions to relevant providers in parallel, with instructions to filter as much as possible.

    • Aggregates and where necessary further filters results from providers, and returns a canonical result set to client apps.

  • The name service keeps a register of all the providers available on the Luminesce platform and the data they expose, and can restart providers if necessary.

  • The file service stores user configuration changes made to certain Lusid.* entity providers to retrieve properties, custom views created using Sys.Admin.SetupView and custom providers based on the Sql.Db ‘provider factory’ (see below).

  • Communication is handled by the RabbitMQ message broker using the AMQPS protocol.

Providers

  • We supply read, analyse and write providers for the investment data stored in LUSID, one provider per operation and per entity type (that is, portfolios, instruments, transactions and so on). For example, you can extract instrument data from LUSID using the Lusid.Instrument provider, calculate holdings using the Lusid.Portfolio.Holding provider, and upsert performance returns using the Lusid.Portfolio.Return.Writer provider. See a list.

  • We supply read and write providers for files stored in Drive, so you can read data from, and write data to, LUSID's secure file repository. See a list.

  • We supply read, and in some cases write, providers for other applications in the FINBOURNE platform such as Scheduler, Notification Service, Configuration Store and Workflow Service.

  • We supply read and write providers for SFTP sites. See a list.

  • We supply read and write providers for local file access. See a list.

  • We supply read and write providers for Luminesce itself, so you can analyse the state of the system, and make changes to it. See a list.

  • We supply a set of utility providers that can be used in conjunction with any other providers to perform operations such as pivoting data, parsing JSON, splitting strings and using Mustache templates. See a list.

  • We supply read, and in some cases write, providers for Gitlab, Slack and many other data sources, whether your data in these platforms is stored in the cloud or on-premise. We host these providers but you can also host them yourself if, for example, you have an on-premise Gitlab instance.

  • We supply a generic SQL database provider (Sql.Db) that you can host yourself and use to publish your own custom providers. The goal is to enable end users to quickly and simply read from or write to your own SQL database, which they might not otherwise have the knowledge or permissions to do. More information.

  • We supply a generic view provider (Sys.Admin.SetupView) that you can use to publish your own custom views. The goal is to enable end users to quickly and simply use other published providers, which they might not otherwise have the knowledge or permissions to do. More information.

  • You can use Lumipy to create and run your own data providers in Python, so you can connect Python data science applications to Luminesce. More information.

  • You can create and host your own custom provider for almost any data source (more information coming soon).

Hosting and networking

  • Port 443 for HTTPS must be open (presumably this is always the case).

  • You must open port 5671 for AMQPS if you:

    • Use the Luminesce CLI client app and choose the option to communicate directly with the query engine.

    • Write your own custom app that communicates directly with the query engine via its .NET API.

    • Use a 3rd party BI tool with the Windows ODBC driver and choose the option to communicate directly with the query engine.

    • Use a 3rd party BI tool with JDBC or ODBC client drivers and the Windows DBC server driver. Note you only need open the port on the Windows server computer, not on individual client computers hosting BI apps.

    • Host any provider yourself.

    More information.

  • If you wish to host one of our 3rd party data providers for Gitlab, Slack and so on then contact us for a ZIP file or Docker image that you can then configure for your .NET environment using a JSON file.

Licensing, authentication and access control

  • You require a Luminesce license for each FINBOURNE provider you wish to query. You require a different license if you want to host one of our providers, or if you want to create your own provider. Contact Technical Support for a trial license.

  • Each user running a query from any client app must authenticate using standard OAuth 2 flows. In addition, you must apply a client certificate (supplied by FINBOURNE) to any computer:

    • Running the Luminesce CLI client app and using the option to communicate directly with the query engine.

    • Hosting the Windows ODBC driver and using the option to communicate directly with the query engine.

    • Hosting the Windows DBC server driver and using the option to communicate directly with the query engine.

    • Hosting a provider.

  • The LUSID domain owner is automatically granted permission to query any provider licensed to that domain, courtesy of the default lusid-administrator role. Every other user running a query from any client app:

    • Must have a LUSID user account with a suitable role assigned (you can manage users and roles using either the LUSID web app or the Identity API).

    • Must have a dedicated policy authorising access to the provider(s) they wish to query. See how to create policies for providers.

    • Must have this policy (or policies) assigned to their role (you can assign policies to roles using either the LUSID web app or the Access API).