--- title: "Performing daily post-trade compliance runs" slug: "performing-daily-post-trade-compliance-runs" updated: 2024-12-03T14:04:08Z published: 2024-12-03T14:04:08Z canonical: "support.lusid.com/performing-daily-post-trade-compliance-runs" --- > ## Documentation Index > Fetch the complete documentation index at: https://support.lusid.com/llms.txt > Use this file to discover all available pages before exploring further. # Performing daily post-trade compliance runs In this tutorial, we’ll see how to use Scheduler to create and automate a daily post-trade [compliance run](/v1/docs/what-is-compliance-in-lusid). You can create a [schedule](/v1/docs/scheduler) to trigger a post-trade compliance run at a chosen time daily. You can add parameters to the schedule that allow you to change which [compliance rules](/v1/docs/what-compliance-rules-does-lusid-support) or portfolios to include in the run. A post-trade compliance run must be created as a job, and is based on a Docker image. This tutorial assumes you are familiar with [Docker](https://docs.docker.com/get-started/). ## Step 1: Creating a Docker image containing a suitable script The first step is to create a Python script using the LUSID Python SDK to perform a compliance run. A `Main.py` Python script to perform a compliance run might look like the one below. We’ll need to authenticate to the SDK but we don’t want to store credentials in the Docker image. So instead we’ll pass them securely into the job from the Configuration Store as environment variables, along with a [recipe](/v1/docs/what-is-a-recipe), [compliance rule scope](/v1/docs/how-do-i-create-a-compliance-rule) and run scope ([step 2](/v1/docs/performing-daily-post-trade-compliance-runs#step-2-uploading-lusid-credentials-to-the-configuration-store)). **Click to expand **Python script for post-trade compliance run**** ```python # Import Packages import os import lusid from lusid import (    SyncApiClientFactory,    ArgsConfigurationLoader,    EnvironmentVariablesConfigurationLoader, ) def run_compliance(api_factory: SyncApiClientFactory, is_pre_trade: bool, recipe_address: str, rule_scope: str, run_scope: str) -> lusid.ComplianceRunInfoV2:    # Initialisation of the compliance run    pre_or_post = "Pre" if is_pre_trade else "Post"    print(f"Running {pre_or_post}-Trade Compliance")    compliance_api = api_factory.build(lusid.ComplianceApi)    try:        # Running Compliance        compliance_response = compliance_api.run_compliance(            run_scope = run_scope,            rule_scope = rule_scope,            is_pre_trade = is_pre_trade,            recipe_id_scope = recipe_address.split("/")[0],            recipe_id_code = recipe_address.split("/")[1]        )            except Exception as e:        print(f"Error running {pre_or_post}-Trade Compliance: {e}")    print(f"Completed {pre_or_post}-Trade Compliance at: {compliance_response.completed_at.strftime('%Y-%m-%d %H:%M:%S')}")        return compliance_response def main():    config_loaders=(        ArgsConfigurationLoader(app_name="PostTradeComplianceRun"),        # This expects FBN_ACCESS_TOKEN and FBN_LUSID_URL as 'https://domain.lusid.com/api' environment variables        EnvironmentVariablesConfigurationLoader())    api_factory = SyncApiClientFactory(config_loaders=config_loaders)    recipe_address = os.getenv("RECIPE_ADDRESS")    rule_scope = os.getenv("RULE_SCOPE")    run_scope = os.getenv("RUN_SCOPE")    assert recipe_address is not None    assert rule_scope is not None    assert run_scope is not None        # Run post trade compliance    run_compliance(        api_factory=api_factory,        is_pre_trade=False,        recipe_address=recipe_address,        rule_scope=rule_scope,        run_scope=run_scope    )     if __name__== "__main__":    main() ``` The Docker file to create a suitable image for this Python script might look like the one below. Note the base image (for example `debian:12-slim`) must contain no critical or high vulnerabilities to pass AWS gate checks. [Read more on troubleshooting vulnerabilities.](/v1/docs/troubleshooting-docker-image-vulnerabilities) ```dockerfile FROM debian:12-slim as base WORKDIR /app RUN apt-get update RUN apt-get install python3 python3-pip python3-venv -y RUN python3 -m venv /opt/venv RUN /opt/venv/bin/pip install --no-cache-dir lusid-sdk==2.1.608 COPY main.py . FROM gcr.io/distroless/cc-debian12 as temp COPY --from=base /usr/bin/c_rehash /usr/bin/openssl /usr/bin/pdb3 /usr/bin/pdb3.11 /usr/bin/py3clean /usr/bin/py3compile /usr/bin/py3versions /usr/bin/pydoc3 /usr/bin/pydoc3.11 /usr/bin/pygettext3 /usr/bin/pygettext3.11 /usr/bin/python3 /usr/bin/python3.11 /usr/bin/ COPY --from=base /usr/lib/binfmt.d /usr/lib/ssl /usr/lib/valgrind /usr/lib/ /usr/lib/ COPY --from=base /usr/share/applications/ /usr/share/zoneinfo/ /usr/share/lintian/ /usr/share/binfmts/ /usr/share/python3 /usr/share/readline/ /usr/share/ COPY --from=base /opt/venv/ /opt/venv/ FROM temp as final COPY --from=base /app /app WORKDIR /app ENTRYPOINT ["/opt/venv/bin/python3", "main.py"] ``` To build this Docker image, you might run a command like this: ```shell docker build -t lusid-compliance-image:latest . ``` You could test the image works locally before uploading it by setting `RECIPE_ADDRESS`, `RULE_SCOPE` and `RUN_SCOPE` as environment variables and performing a compliance run on a test portfolio. ```shell docker run lusid-compliance-image ``` ## Step 2: Uploading LUSID credentials to the Configuration Store Every call made to a LUSID API must be authorised by an [API access token](/v1/docs/what-are-an-api-access-token-and-a-client-secret). For this tutorial, we’ll use a [Personal Access Token (PAT)](/v1/docs/what-is-a-personal-access-token) to set up our schedule. To enable an SDK to access our token, we'll need to assemble the following details and pass them into the job as environment variables at runtime: - The LUSID API URL for our domain - A Personal Access Token (PAT) > [!NOTE] > Refreshing tokens > > Instead of a PAT, you might prefer to set up your schedule to use a refreshing token. The LUSID SDKs have helper classes that automate the process of obtaining an API access token and refreshing it upon expiry. To enable an SDK to do this, you'll need to assemble the following details and pass them into the job as environment variables at runtime: > > - Your LUSID account username and password > - A client ID and client secret pair > - The dedicated Okta token URL for our LUSID domain > > To see how to assemble this information, read [this article](/v1/docs/how-do-i-use-an-api-access-token-with-the-lusid-sdk). We can pass the less-sensitive LUSID API URL directly into the job. However, the only way to securely pass in the PAT is to upload this credential in advance to the Configuration Store, in order to extract it at runtime. [See how to do this](/v1/docs/how-do-i-upload-information-to-the-configuration-store). > [!TIP] > Tip > > You can set **Block Reveal** to `true` to allow only LUSID itself to access the value; it cannot be exposed by any user once saved. [Read more.](/v1/docs/how-do-i-upload-information-to-the-configuration-store) For the purposes of this tutorial, we'll assume we have created a configuration set in our **Personal keys** area of the Configuration Store containing our PAT: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(378).png) ## Step 3: Creating a job To create a job that can run the content of the Docker image: 1. Sign into the [LUSID web app](https://www.lusid.com/app/home) and select **Jobs & Scheduling > Jobs** from the left-hand menu: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(379).png) 2. On the **Jobs** dashboard, click the **Create job** button (top right). 3. Specify a **Scope** and **Code** that together uniquely identify the job in your LUSID domain: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(362).png) ### Uploading your Docker image First, [follow these instructions](/v1/docs/how-do-i-upload-a-docker-image-to-the-finbourne-aws-store) to upload your Docker image to the FINBOURNE AWS store. Upon returning to this screen, you should be able to select your image from the `Image name` and `Version` dropdowns: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(376).png) Note it is not possible to select an image with any critical or high vulnerabilities. [Read more on troubleshooting Docker image vulnerabilities.](/v1/docs/troubleshooting-docker-image-vulnerabilities) ### Authenticating to the LUSID Python SDK using environment variables On the **Arguments** screen, you can (optionally) define environment variables to pass in to your job at runtime. In our example, since we're exercising the LUSID API via the LUSID Python SDK, we'll use this facility to pass in the credentials assembled in [step 2](/v1/docs/performing-daily-post-trade-compliance-runs#step-2-uploading-lusid-credentials-to-the-configuration-store), so the SDK can obtain an API access token. We’ll also add environment variables to pass in a recipe, rule scope, and run scope. We'll nominate a default value for each. We need to define the following environment variables: | Variable name | Value source | Data type | Example value | | --- | --- | --- | --- | | `FBN_ACCESS_TOKEN` | Configuration Store | `Configuration` | `config://personal/00u91lo2eeX42sdse2p7/scheduler/my-image/PAT` | | `FBN_LUSID_URL` | LUSID domain URL | `String` | `https://acmecorp.lusid.com/api` | | `RECIPE_ADDRESS` | Your recipe scope/code | `String` | `Finbourne-Examples/Recipe` | | `RULE_SCOPE` | Your compliance rule scope | `String` | `Finbourne-Examples` | | `RUN_SCOPE` | Your compliance run scope | `String` | `Finbourne-Examples` | ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(380).png) ## Step 4: Running the job and examining the results You can run the job manually at any time by selecting the **Run** icon on the **Jobs** dashboard. The **Run a job** screen prompts you to confirm or override values for mandatory arguments, if these are defined. ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(381).png) To examine the status of the job, open the **History** dashboard: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(389).png) If the **Job Status** column records `Success`, click the link in the **Run ID** column to examine the results on the **Details** dashboard. ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(383).png) On the **Details** dashboard, if the job writes information to the console you can examine it on the **Console Output** tab: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(384).png) If the **Job Status** column records `Failure`, see [Troubleshooting](/v1/docs/performing-daily-post-trade-compliance-runs#step-6-troubleshooting-a-failed-job). ## Step 5: Creating a schedule to run the job daily To create a cron-like schedule for your job so that it runs automatically: 1. Navigate to the **Schedules** dashboard and click the **Create schedule** button (top right). 2. On the **Specify job** screen, choose the **Job** to automate from the dropdown list, and a **Scope** and **Code** that together uniquely identify the schedule (note, this is not the same as the scope and code for the job itself): ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(386).png) 3. On the **Arguments** screen, confirm the default values for the environment variables passed in to the job, or override the defaults. 4. On the **Triggers** screen, select `Time based` as **Trigger type** and configure the trigger as `Daily` to trigger the schedule at a particular hour on particular days. For example, we want to set up our post-trade schedule to run at 11pm each day: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(387).png) Once the schedule is created, we simply need to wait until the next run is scheduled to take place. You can also run the schedule on demand via the **Schedules** dashboard by selecting **Menu > Run now**: ![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image(388).png) ## Step 6: Troubleshooting a failed job [Read more on jobs monitoring and troubleshooting a failed job.](/v1/docs/monitoring-job-run-history-and-troubleshooting-failures)