What is structured result data?

You can upsert your own structured data (JSON or CSV) to LUSID's bitemporal, immutable structured result store (SRS).

You can upsert any valid content, but the SRS is designed to store external result data that can feed into LUSID valuation operations. When you ask LUSID to value a portfolio, you can augment the report with data from the SRS. For certain types of result data, you can override LUSID to change the way LUSID computes PV and other metrics.

Note: You can upsert multiple incomplete documents and ask the SRS to merge them into a single 'virtual' document for you.

For example, you could store:

• Risk characteristics for a portfolio or YTD performance figures for an ETF and augment a valuation report with this information.

• MV or GainLoss for the holdings in a portfolio and augment a valuation report with this information.

• Accruals for a bond and override LUSID's internal accrual calculation to change the way LUSID computes PV and PnL.

• Cashflows for a swap and override LUSID's internal cashflow calculation to generate upsertable transactions based on the external data.

You may want do this because:

• You have data that LUSID's data model is not able to natively store (in this sense, the SRS behaves in a similar way to custom properties).

• You prefer the precision of your own result data over that calculated by LUSID (or retrieved by LUSID from a 3rd party library).

• It may be quicker for LUSID to find your result data in the SRS than to perform or retrieve a calculation itself.

• You want to reconcile your result data with that calculated or retrieved by LUSID.

In effect, when you perform a valuation operation you can tell LUSID to “find or calculate”; that is, to calculate or retrieve valuation results itself, to prefer valuation results from the SRS, or some combination of the two.

Note you shouldn't use the SRS to:

• Extend built-in entities; this should be done using custom properties.

• Store large (2Mb+) or arbitrary (unstructured) documents; these should be stored in Drive and, if bitemporal capabilities are required, elements can be modelled using custom entities.

• Store documents that do not contain data intended to augment, override or reconcile against valuation results generated by LUSID; these can be stored in Drive and modelled using custom entities.

Providing you have appropriate permissions, you can interact with the SRS using:

• The API endpoints in the Structured Result Data set.

• Luminesce read and write providers.

Use cases

The SRS stores a document with a particular result type. Each result type has a different use case and implications:

Mapping external result data using identifiers

Before you upsert a document to the SRS, you must provide a data map that enables LUSID to identify and understand the data in that document.

A data map consists of keys, one for each ‘column’ in the document. At least one of those keys must act as a unique identifier. The precise number and nature of identifier keys depends on the result type of the document.

For example, consider the following single-row document containing accrual data for a bond upserted using the UnitResult/Analytic result type:

This result type requires LusidInstrumentId as an identifier, so you can designate the LUID column as the identifier key. Note if there are multiple rows in the document, every row must have a value in this column. The identifier key in the data map might look like this:

{
  "address": "UnitResult/LusidInstrumentId",  # see the table below for identifiers for other result types
  "name": "LUID",                             # maps the address to the column name in the document
  "dataType": "string",                       # other options are "decimal" and "resultOd"
  "keyType": "Unique"                         # only one constituent is required as an identifier for this result type
}

If an identifier key for a result type only has one constituent, the key type is Unique. If it has multiple constituents, each key type is PartOfUnique:

Note that columns in a SRS document mapped to identifier keys cannot be retrieved by LUSID from that document.

Locating external result data using recipes

LUSID uses the market data and pricing models marshalled by a recipe to perform a valuation. You must create a recipe, or modify an existing one, to incorporate external result data into a valuation report.

For each category of external result data you must specify a result data rule in the resultDataRules array of the pricing section of your recipe. Consider the following example, of one rule that enables LUSID to locate and retrieve result data of type UnitResult/Analytic encapsulated in a my-bond-accrual-result-srs-scope from the SRS:

"pricing": {
  "resultDataRules": [
    {
      "resourceKey": "UnitResult/*",
      "supplier": "Client",
      "dataScope": "my-bond-accrual-result-srs-scope",
      "documentCode": "instrumentaccrued-results",
      "quoteInterval": "2D",
      "documentResultType": "UnitResult/Analytic",
      "resultKeyRuleType": "ResultDataKeyRule"
    }
  ]
  ...
},
...

Example of overriding LUSID's calculation of accrual

You have a portfolio containing 100 units of a bond paying 5%.

Normally, when you ask LUSID to value this portfolio, you can use the built-in Valuation/InstrumentAccrued metric to calculate the interest accrued since the last coupon payment. Note this metric is unitised; that is, it represents accrual for one unit of the instrument rather than for the total number of units held. LUSID then automatically scales this figure by your holding and uses it to compute Valuation/PV in conjunction with the market value. So for example a typical valuation report might look like this:

Valuation/InstrumentAccrued is one of the metrics it's possible for the SRS to override and propagate for use in downstream LUSID calculations. You can store your own, externally-generated accrual data in a document in the SRS, and ask LUSID to use that instead of it's own accrual figure when calculating PV. You might want to do this if you trust the provenance of your data over that computed by LUSID.

Upserting your own accrual result data to the SRS

For example, imagine you have a CSV file ready for upload to the SRS like this:

In order to override LUSID data and propagate external results, the first step is call the UpsertStructuredResultData API to upsert this document using the UnitResult/Analytic result type.

Creating a data map in the SRS

Then, you call the CreateDataMap API to create a data map that defines a key for each column in the document as per the table below:

Note the following:

• The first part of a data map key Address must be UnitResult.

• Subsequent parts of the Address must match built-in LUSID metrics in order to override them.

• The Name must match the column header in the SRS document.

• The Name of the CompositeLeaf Key type is deliberately omitted. Instead, the two ‘child’ keys are mapped to the appropriate columns in the document.

• Data maps are version-controlled and immutable; once upserted into LUSID, they cannot be changed. You must upsert a new version.

As an alternative to the API, creating this data map using the Python SDK might look like this:

srs_api = api_factory.build(la.StructuredResultDataApi)
data_map_key = lm.DataMapKey(version = "1.0.0", code = "datamapkey")
data_map = lm.DataMapping(
    data_definitions = [
        lm.DataDefinition(address="UnitResult/Analytic/default/ValuationDate", name = "Date", data_type = "string", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/LusidInstrumentId", name = "LUID", data_type = "string", key_type = "Unique"),
        lm.DataDefinition(address="UnitResult/Instrument/default/Name", name = "Security", data_type = "string", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/Notes", name = "Source", data_type = "string", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/Valuation/InstrumentAccrued", data_type = "Result0D", key_type = "CompositeLeaf"),
        lm.DataDefinition(address="UnitResult/Valuation/InstrumentAccrued/Amount", name = "Accrual", data_type = "decimal", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/Valuation/InstrumentAccrued/Ccy", name = "Currency", data_type = "string", key_type = "Leaf"),
    ]
)
try:    
    response = srs_api.create_data_map(
        scope = "srs-accrual", 
        request_body = {
            "data-map-request-1": lm.CreateDataMapRequest(
                id = data_map_key,
                data = data_map
            ),
        }
    )
except lu.ApiException as e:
    display(json.loads(e.body))

Upserting a recipe that can locate data in the SRS

The next step is to call the UpsertConfigurationRecipe API to create a recipe (or amend an existing one) to have a result data rule, for example:

"resultDataRules": [
  {
    "resourceKey": "UnitResult/*",
    "supplier": "Client",
    "dataScope": "srs-accrual",
    "documentCode": "instrumentaccrued-results",
    "quoteInterval": "1D",
    "documentResultType": "UnitResult/Analytic",
    "resultKeyRuleType": "ResultDataKeyRule"
  }
],
...

Performing a valuation

Now when you call the GetValuation API to perform a valuation you can request metrics from the SRS as well as, or instead of, those built-in to LUSID. For example, your valuation request might contain the following metrics:

Your valuation report would look like this:

Note the following:

• The PV reflects the external calculation of accrual, scaled by the holding.

• The valuation report is augmented by the extra note from the SRS identifying the data source of the accrual.