In this tutorial we'll see how to use Scheduler to create, test and automate a job to securely upsert transactions into LUSID from a CSV file.
Note: To complete this tutorial, you must have suitable access control permissions. This can most easily be achieved by assigning your LUSID user the built-in lusid-administrator role. This should already be the case if you are the domain owner.
A job must be based on a Docker image; this tutorial assumes you are familiar with Docker. The code in the Docker image can call any endpoint in the core LUSID or in the Identity or Access APIs. We'll need to authenticate in the standard way by obtaining an API access token.
In this tutorial, we’ll call the LUSID UpsertTransactions API endpoint using the LUSID Python SDK. We’ll pull our credentials to use the SDK into the job from the Configuration Store, and also add a pair of command line arguments to allow the user to nominate a particular transaction portfolio to upsert to at runtime.
- Step 1: Creating a Docker image containing a suitable script
- Step 2: Uploading LUSID credentials to the Configuration Store
- Step 3: Creating a job
- Step 4: Running the job and examining results
- Step 5: Creating a schedule for automation
- Step 6: Troubleshooting a failed job
A transactions.py Python script to upsert transactions into LUSID from a CSV file using the LUSID Python SDK might look like the one below.
We'll need to authenticate to the SDK but we don't want to store credentials in the Docker image. So instead we'll pass them securely into the job from the Configuration Store as environment variables at runtime; see Step 2.
The Dockerfile to create a suitable image for this Python script might look like this (the image must contain no critical or high vulnerabilities in order to pass AWS gate checks):
To build this Docker image, you might run a command like this:
You could test the image works locally before uploading it by upserting transactions to one of the example portfolios provided with LUSID:
Every call made to a LUSID API must be authorised by an API access token.
Note: If your job does not exercise the core LUSID or the Identity or Access APIs (in other words, it performs a non-LUSID operation), you can skip this step.
The LUSID SDKs have helper classes that automate the process of obtaining an API access token and refreshing it upon expiry. To enable an SDK to do this, we'll need to assemble the following details and pass them into the job as environment variables at runtime:
- Our LUSID account username and password
- A client ID and client secret pair
- The dedicated Okta token URL for our LUSID domain
To see how to assemble this information, read this article.
We can pass the less-sensitive account username, client ID and dedicated token URL directly into the job. However, the only way to securely pass in the password and client secret is to:
- Upload these credentials in advance to the Configuration Store, in order to extract them at runtime. See how to do this.
- Authorise the Configuration Store to act on your behalf (you only need perform this operation once, the first time you set up a job; subsequently, the Configuration Store is authorised for all future jobs). See how to do this.
For the purposes of this tutorial, we'll assume we have authorised the Configuration Store and created a configuration set in our Personal area containing two items, one for the account password and one for the client secret:
To start the process of creating a new job:
- Sign into the LUSID web app and select Workflows & Scheduling > Jobs from the left-hand menu:
- On the Jobs dashboard, click the Create job button (top right).
- Specify a Scope and Code that together uniquely identify the job in your LUSID domain:
On the Docker information screen, you'll soon be able to choose to Upload new image.
For now, however, you must follow these instructions to upload your Docker image to the FINBOURNE AWS store.
Upon returning to this screen, you should be able to select your image from the Choose existing image dropdown:
Nominating a transaction portfolio using command line arguments
On the Arguments screen, you can (optionally) define command line arguments to pass in to your job at runtime.
In our example, we’ll add two arguments, one to pass in a scope and one to pass in a code (choose the equals operator when doing this). Both should be mandatory, since transactions must upsert to a transaction portfolio. We'll nominate a default portfolio (Finbourne-Examples/UK-Equities) and allow this to be overridden by a user at runtime:
On the Arguments screen, you can (optionally) define environment variables to pass in to your job at runtime.
In our example, since we're exercising the LUSID API via the LUSID Python SDK, we'll use this facility to pass in the credentials assembled in step 2, so the SDK can obtain an API access token.
We need to define the following environment variables:
|Variable name||Source of the value||Data type||Example value|
|FBN_USERNAME||LUSID account username||String||johndoe|
|FBN_CLIENT_ID||Client ID of the application||String||example-sdk-app|
|FBN_LUSID_API_URL||LUSID domain URL||String||https://acmecorp.lusid.com/api|
|FBN_TOKEN_URL||Dedicated Okta token URL for LUSID domain||String||https://lusid-acmecorp.okta.com/oauth2/aus91lnun55CZjjav2p7/v1/token|
To pull in the account password and client secret from the Configuration Store, select Configuration from the Data type dropdown and click in the Default value key area to be guided to select the appropriate configuration set and item (the config://... syntax is automatically generated):
Nominating the LUSID ecosystem API to call
On the Resources screen, select the LUSID ecosystem API to call. In our example, this is Data API to call the core LUSID REST API via the LUSID Python SDK:
You can run the job manually at any time by selecting on the Jobs dashboard. The Run a job screen prompts to confirm or override values for mandatory arguments, if these are defined.
To examine the status of the job, open the History dashboard:
If the Job Status column records Success, click the link in the Run ID column to examine the results on the Details dashboard (you may need to re-arrange columns to see them as they are here).
On the Details dashboard, if the job writes information to the console you can examine it on the Console Output tab:
If the Job Status column records Failure, see Troubleshooting.
To create a cron-like schedule for your job so that it runs automatically:
- Navigate to the Schedules dashboard and click the Create schedule button (top right).
- On the Specify job screen, choose the Job to automate from the dropdown list, and a Scope and Code that together uniquely identify the schedule (note, this is not the same as the scope and code for the job itself):
- On the Arguments screen, confirm the default values for any command line arguments and environment variables passed in to the job, or override the defaults.
- On the Triggers screen, choose either:
- Time-based to specify a daily or hourly schedule.
- File-based to trigger the schedule when a file is added to a particular Amazon S3 bucket.
- File-based with window to combine the two.
If a job fails, contact Technical Support.