--- title: "How do I create and run a data quality check?" slug: "how-do-i-create-and-run-a-data-quality-check" updated: 2026-03-06T13:54:24Z published: 2026-03-06T13:54:24Z canonical: "support.lusid.com/how-do-i-create-and-run-a-data-quality-check" --- > ## Documentation Index > Fetch the complete documentation index at: https://support.lusid.com/llms.txt > Use this file to discover all available pages before exploring further. # How do I create and run a data quality check? In this tutorial, we’ll see how to use the [CheckDefinitions API](https://www.lusid.com/docs/api/lusid/endpoints/check-definitions/CreateCheckDefinition) to model a set of rules and run a simple data quality (DQ) check. Let’s imagine we want to check instruments that were ingested or updated in the most recent run of an integration for some essential properties. To do this, we first create a check definition that defines which data to check for. We can then run the check and inspect the results for rule breaches. Note we’ll only run checks manually in this tutorial. To learn how to automate checks and handle the results, [follow this tutorial on setting up a data quality check worker.](/v1/docs/how-do-i-set-up-a-data-quality-check-workflow) > [!NOTE] > Note > > - To complete this tutorial, you must have suitable access control permissions. This can most easily be achieved by assigning your LUSID user the `lusid-administrator` role. > - You can follow this tutorial in Python using the [data quality checks tutorial](https://github.com/finbourne/finbourne-notebooks/blob/main/V2/product/data-quality/data-quality-check-tutorial.ipynb) Jupyter notebook. We recommend running the notebook in your [LUSID JupyterHub instance](/v1/docs/running-jupyter-notebooks-in-your-lusid-domain), which handles authentication to LUSID automatically. ## Step 1: Creating a check definition to model rulesets We must first create a check definition using the [CreateCheckDefinition](https://www.lusid.com/docs/api/lusid/endpoints/check-definitions/CreateCheckDefinition) API, passing in the following: - A `scope` and `code` that together uniquely identify the check definition - A friendly `displayName` and `description` - The type of `datasetSchema` for the check: - `type`: Currently `LusidEntity` is the only valid value - `entityType`: Currently `Instrument` is the only valid value - A list of `ruleSets`, with each item containing the following: - `ruleSetKey`: A unique identifier - `displayName`: A friendly name - `description`: A detailed description of the rule set - `ruleSetFilter`: Defines which data to run the rule checks using [LUSID filtering syntax](/v1/docs/filtering-information-retrieved-from-lusid) - Optionally, a list of [properties](/v1/docs/properties) to include on the check definition **Shell session** ```json curl -X POST 'https://.lusid.com/api/api/dataquality/checkdefinitions'  -H 'Authorization: Bearer '  -H 'Content-Type: application/json-patch+json'  -d '{  "id": {    "scope": "Finbourne-Examples",    "code": "DQ-Check-instrument-properties"  },  "displayName": "Instruments check",  "description": "A check definition to validate instruments are populated with the correct properties",  "datasetSchema": {    "type": "LusidEntity",    "entityType": "Instrument"  },  "ruleSets": [    {      "ruleSetKey": "instrument-properties-checks",      "displayName": "Instrument properties checks ruleset",      "description": "A set of rules to apply to instruments assigned for data refreshes to check for appropriate properties.",      "ruleSetFilter": "Properties[Instrument/Finbourne-Examples/RefreshData] exists"    }  ], }' ``` **Python** ```python # Set up CheckDefinitions API import lusid import lusid.api import lusid.models import lusid.extensions from lusidjam import RefreshingToken lusid_api_factory = lusid.extensions.SyncApiClientFactory(config_loaders=[    lusid.extensions.ArgsConfigurationLoader(        api_url="https://.lusid.com/api",        access_token=RefreshingToken(),        app_name="LusidJupyterNotebook"    ) ]) checkDefinitions_api = lusid_api_factory.build(lusid.api.CheckDefinitionsApi) # Create check definition rule_set_key = "instrument-properties-checks" try:    checkDefinitions_api.create_check_definition(        create_check_definition_request=lusid.models.CreateCheckDefinitionRequest(            id=lusid.models.ResourceId(                scope="Finbourne-Examples",                code="DQ-Check-instrument-properties"            ),            display_name="Instruments check",            description="A check definition to validate instruments are populated with the correct properties",            dataset_schema=lusid.models.CheckDefinitionDatasetSchema(                type="LusidEntity",                entity_type="Instrument"            ),            rule_sets=[lusid.models.UpdateCheckDefinitionRuleSet(                rule_set_key=rule_set_key,                display_name="Instrument properties checks ruleset",                description="A set of rules to apply to instruments assigned for data refreshes to check for appropriate properties.",                rule_set_filter="Properties[Instrument/Finbourne-Examples/RefreshData] exists"            )]        )    ) except lusid.ApiException as e:    if e.status == 400 and "EntityWithIdAlreadyExists" in str(e.body):        print("Check definition already exists, skipping creation.")    else:        raise ``` Part of a response is shown below. Note the `asAtVersionNumber` of the check definition, which increments if you update the check definition: ```json {  "id": {    "scope": "Finbourne-Examples",    "code": "DQ-Check-instrument-properties"  },  "version": {    "asAtCreated": "2026-01-22T11:29:44.2505000+00:00",    "asAtModified": "2026-01-22T11:29:44.2505000+00:00",    "userIdModified": "00ujk6twb4jDcHGjN2p8",    "asAtVersionNumber": 1,    ...  }, ... ``` ## Step 2: Adding rules to the check definition Now that we have our check definition, we can upsert rules to the rulesets within the definition. To do so, we call the [UpsertRules](https://www.lusid.com/docs/api/lusid/endpoints/check-definitions/UpsertRules) API, passing in the following: - Within the request URL, the `scope` and `code` of the check definition we want to add the rules to - Within the request body: - `ruleSetKey`: The unique for a ruleset within the check definition - `rule`: - `ruleKey`: A unique identifier for the rule - `displayName`: A friendly name - `description`: A detailed description of the rule - `ruleFormula`: A formula that uses [derived property syntax](/v1/docs/derivation-formulas-part-1-operations-and-allowed-values) to determine a value of `true` or `false` for the data check - `severity`: A number to indicate the importance of the rule **Shell session** ```json curl -X POST 'https://.lusid.com/api/api/dataquality/checkdefinitions/Finbourne-Examples/DQ-Check-instrument-properties/$upsertRules'  -H 'Authorization: Bearer '  -H 'Content-Type: application/json-patch+json'  -d '[  {    "ruleSetKey": "instrument-properties-checks",    "rule": {      "ruleKey": "issuer-exists",      "displayName": "Issuer exists check",      "description": "Checks whether an instrument is decorated with the Instrument/Finbourne-Examples/Issuer property.",      "ruleFormula": "properties[Instrument/Finbourne-Examples/Issuer] exists",      "severity": 1    }  }, ] ``` **Python** ```python try:    checkDefinitions_api.upsert_rules(        scope="Finbourne-Examples",        code="DQ-Check-instrument-properties",        upsert_data_quality_rule=[lusid.models.UpsertDataQualityRule(            rule_set_key=rule_set_key,            rule=lusid.models.CheckDefinitionRule(                rule_key="issuer-exists",                display_name="Issuer exists check",                description="Checks whether an instrument is decorated with the Instrument/Finbourne-Examples/Issuer property",                rule_formula="properties[Instrument/Finbourne-Examples/Issuer] exists",                severity=1            )        )]    ) except lusid.ApiException as e:    print(f"Error upserting rules: {e}") ``` Part of a response is shown below, containing the check definition which now shows the ruleset populated with our rule. ```json {  "id": {    "scope": "Finbourne-Examples",    "code": "DQ-Check-instrument-properties"  },  "ruleSets": [    {      "ruleSetKey": "instrument-properties-checks",      "displayName": "Instrument properties checks ruleset",      "description": "A set of rules to apply to instruments assigned for data refreshes to check for appropriate properties.",      "ruleSetFilter": "Properties[Instrument/Finbourne-Examples/RefreshData] exists",      "rules": [        {          "ruleKey": "issuer-exists",          "displayName": "Issuer exists check",          "description": "Checks whether an instrument is decorated with the Instrument/Finbourne-Examples/Issuer property.",          "ruleFormula": "properties[Instrument/Finbourne-Examples/Issuer] exists",          "severity": 1        }      ]    }  ], ... } ``` ## Step 3: Running the check on some instruments Our check definition now contains rules in rule sets. To run the check on some data, we can call the [RunCheckDefinition](https://www.lusid.com/docs/api/lusid/endpoints/check-definitions/RunCheckDefinition) API, passing in the following: - Within the request URL, the `scope` and `code` of the check definition we want to run - Within the request body: - A `lusidEntityDataset` containing: - `asAt`: An asAt date for the data to be checked; for our example we’ll let it default to the latest - `effectiveAt`: An effectiveAt date for the data to be checked; for our example we’ll let it default to the latest - `scope`: The scope for the data we want to check; for our example we want to check instruments in the `Finbourne-Examples` scope - `asAtModifiedSince`: Optionally, an asAtModified date for the data to be checked; LUSID will only run the check against entities that have been modified since the date specified - `selectorAttribute`: A field name, property key, or identifier to define the data to run the check against; for our example we’ll run the check on instruments that have the property `Instrument/Finbourne-Examples/RefreshData` - `selectorValue`: A value for the `selectorAttribute`; for our example we’ll run the check on instruments with selector property value of `True` - `returnIdentifierKey`: A preferred identifier type to return for each of the checked entities - `limitIndividualBreachesPerRule`: A limit for the number of breaches per rule (above the limit, further breaches are grouped into a single result) **curl** ```json curl -X POST 'https://.lusid.com/api/api/dataquality/checkdefinitions/Finbourne-Examples/DQ-Check-instrument-properties/$run'  -H 'Authorization: Bearer '  -H 'Content-Type: application/json-patch+json'  -d '{  "lusidEntityDataset": {    "scope": "Finbourne-Examples",    "selectorAttribute": "Properties[Instrument/Finbourne-Examples/RefreshData]",    "selectorValue": "True",    "returnIdentifierKey": "Instrument/default/ClientInternal"  },  "limitIndividualBreachesPerRule": 20 }' ``` **Python** ```python try:    run_check_response = checkDefinitions_api.run_check_definition(        scope="Finbourne-Examples",        code="DQ-Check-instrument-properties",        run_check_request=lusid.models.RunCheckRequest(            lusid_entity_dataset=lusid.models.LusidEntityDataset(                scope="Finbourne-ExamplesDQ-Check",                selector_attribute="Properties[Instrument/Finbourne-Examples/RefreshData]",                selector_value="True",                return_identifier_key="Instrument/default/ClientInternal"            ),            limit_individual_breaches_per_rule=100        )    )    results = run_check_response.data_quality_check_results    total = len(results)    print(f"Check complete: {total} results (breaches)") except lu.ApiException as e:    print(f"Error running check: {e}") ``` ## Next steps [Follow this tutorial on setting up a DQ check workflow to](/v1/docs/how-do-i-set-up-a-data-quality-check-workflow): - Use the check definition in a workflow to automate the DQ check - Inspect the results from a run of the DQ check workflow via the LUSID web app For bitemporal data, the **asAt** timestamp reflects the point in time at which a data item is saved into LUSID. This allows data to be extracted from LUSID as it was at a point in the past, regardless of any subsequent updates or corrections. It is determined internally by LUSID and cannot be updated by the user. For bitemporal data, the **effectiveAt** timestamp reflects the point in time at which a data item becomes valid in LUSID from a business perspective. You can specify the field when you make a read or update request to LUSID, indicating that you want to retrieve or update data that is effective at the specified date.