What is structured result data? · Customer Self-Service

Views:

You can upsert your own structured data (JSON or CSV) to LUSID's bitemporal, immutable structured result store (SRS).

You can upsert any valid content, but the SRS is designed to store external result data that can feed into LUSID valuation operations. When you ask LUSID to value a portfolio, you can augment the report with data from the SRS. For certain types of result data, you can override LUSID to change the way LUSID computes PV and other metrics.

Note: You can upsert multiple incomplete documents and ask the SRS to merge them into a single 'virtual' document for you.

For example, you could store:

Risk characteristics for a portfolio or YTD performance figures for an ETF and augment a valuation report with this information.
MV or GainLoss for the holdings in a portfolio and augment a valuation report with this information.
Accruals for a bond and override LUSID's internal accrual calculation to change the way LUSID computes PV and PnL.
Cashflows for a swap and override LUSID's internal cashflow calculation to generate upsertable transactions based on the external data.

You may want do this because:

You have data that LUSID's data model is not able to natively store (in this sense, the SRS behaves in a similar way to custom properties).
You prefer the precision of your own result data over that calculated by LUSID (or retrieved by LUSID from a 3rd party library).
It may be quicker for LUSID to find your result data in the SRS than to perform or retrieve a calculation itself.
You want to reconcile your result data with that calculated or retrieved by LUSID.

In effect, when you perform a valuation operation you can tell LUSID to “find or calculate”; that is, to calculate or retrieve valuation results itself, to prefer valuation results from the SRS, or some combination of the two.

Note you shouldn't use the SRS to:

Extend built-in entities; this should be done using custom properties.
Store large (2Mb+) or arbitrary (unstructured) documents; these should be stored in Drive and, if bitemporal capabilities are required, elements can be modelled using custom entities.
Store documents that do not contain data intended to augment, override or reconcile against valuation results generated by LUSID; these can be stored in Drive and modelled using custom entities.

Providing you have appropriate permissions, you can interact with the SRS using:

The API endpoints in the Structured Result Data set.
Luminesce read and write providers.

Use cases

The SRS stores a document with a particular result type. Each result type has a different use case and implications:

Result type	Use case
`UnitResult/Portfolio`	Use this result type to store a representation of a portfolio from an external system and reconcile it against LUSID's representation. Jupyter Notebook example. This result type cannot interact with LUSID's valuation system. Note you cannot upsert incomplete documents of this type and ask the SRS to merge them into a single ‘virtual’ document for you.
`UnitResult/Grouped`	Use this result type to store portfolio-level data from an external system, such as risk characteristics or YTD figures. This result type can interact with LUSID's valuation system in order to augment a valuation report with external result data.
`UnitResult/Holding`	Use this result type to store data from an external system about the holdings in a portfolio, for example MV or GainLoss. Jupyter Notebook example. This result type can interact with LUSID's valuation system in order to augment a valuation report with external result data.
`UnitResult/Analytic`	Use this result type to override LUSID result data for the instruments underlying the holdings in a portfolio. This result type can interact with LUSID's valuation system in two ways: You can override any LUSID `Valuation/` metric ('queryable key') in order to replace LUSID's result data with that from the external system. So for example you could store PV data and override the built-in `Valuation/PV` metric to use external PV results rather than LUSID's calculation of PV. You can override the following built-in metrics and* propagate external result data through the valuation engine in order to not only replace LUSID's result data but also to use it in downstream calculations: `Valuation/InstrumentAccrued` - You can override LUSID's calculation of accrual and change the way LUSID calculates PV for instruments such as bonds. See below and also this Jupyter Notebook example. `Valuation/Cashflows` - You can override LUSID's calculation of cashflows and change the upsertable transactions emitted by certain APIs as part of a lifecycle management strategy for cashflow-generating instruments such as swaps and bonds. Jupyter Notebook example.
`UnitResult/Custom`	Use this result type to store any other structured data. Jupyter Notebook example.

Mapping external result data using identifiers

Before you upsert a document to the SRS, you must provide a data map that enables LUSID to identify and understand the data in that document.

A data map consists of keys, one for each ‘column’ in the document. At least one of those keys must act as a unique identifier. The precise number and nature of identifier keys depends on the result type of the document.

For example, consider the following single-row document containing accrual data for a bond upserted using the UnitResult/Analytic result type:

Date	LUID	Security	Accrual	Currency
2022-10-21	LUID_00003DAH	UKT 0 ⅜ 10/22/26	5.25	GBP

This result type requires LusidInstrumentId as an identifier, so you can designate the LUID column as the identifier key. Note if there are multiple rows in the document, every row must have a value in this column. The identifier key in the data map might look like this:

{
  "address": "UnitResult/LusidInstrumentId",  # see the table below for identifiers for other result types
  "name": "LUID",                             # maps the address to the column name in the document
  "dataType": "string",                       # other options are "decimal" and "resultOd"
  "keyType": "Unique"                         # only one constituent is required as an identifier for this result type
}

If an identifier key for a result type only has one constituent, the key type is Unique. If it has multiple constituents, each key type is PartOfUnique:

Result type	Identifier key(s)
`UnitResult/Portfolio`	`UnitResult/Instrument/default/LusidInstrumentId` `UnitResult/Holding/default/Currency`
`UnitResult/Grouped`	`UnitResult/PortfolioScope` `UnitResult/PortfolioCode`
`UnitResult/Holding`	`UnitResult/Instrument/default/LusidInstrumentId` `UnitResult/Portfolio/Id` `UnitResult/Portfolio/Scope` `UnitResult/Portfolio/Code` In addition, if the containing portfolio has sub-holding keys (SHKs) then the properties comprising those SHKs must be included in the data map as identifier keys, for example: `UnitResult/Transaction/Bloomberg/Strategy` `UnitResult/Transaction/Refinitiv/Country`
`UnitResult/Analytic`	`UnitResult/LusidInstrumentId`
`UnitResult/Custom`	`UnitResult/<any-string>`

Note that columns in a SRS document mapped to identifier keys cannot be retrieved by LUSID from that document.

Locating external result data using recipes

LUSID uses the market data and pricing models marshalled by a recipe to perform a valuation. You must create a recipe, or modify an existing one, to incorporate external result data into a valuation report.

For each category of external result data you must specify a result data rule in the resultDataRules array of the pricing section of your recipe. Consider the following example, of one rule that enables LUSID to locate and retrieve result data of type UnitResult/Analytic encapsulated in a my-bond-accrual-result-srs-scope from the SRS:

"pricing": {
  "resultDataRules": [
    {
      "resourceKey": "UnitResult/*",
      "supplier": "Client",
      "dataScope": "my-bond-accrual-result-srs-scope",
      "documentCode": "instrumentaccrued-results",
      "quoteInterval": "2D",
      "documentResultType": "UnitResult/Analytic",
      "resultKeyRuleType": "ResultDataKeyRule"
    }
  ]
  ...
},
...

Example of overriding LUSID's calculation of accrual

You have a portfolio containing 100 units of a bond paying 5%.

Normally, when you ask LUSID to value this portfolio, you can use the built-in Valuation/InstrumentAccrued metric to calculate the interest accrued since the last coupon payment. Note this metric is unitised; that is, it represents accrual for one unit of the instrument rather than for the total number of units held. LUSID then automatically scales this figure by your holding and uses it to compute Valuation/PV in conjunction with the market value. So for example a typical valuation report might look like this:

Valuation/InstrumentAccrued is one of the metrics it's possible for the SRS to override and propagate for use in downstream LUSID calculations. You can store your own, externally-generated accrual data in a document in the SRS, and ask LUSID to use that instead of it's own accrual figure when calculating PV. You might want to do this if you trust the provenance of your data over that computed by LUSID.

Upserting your own accrual result data to the SRS

For example, imagine you have a CSV file ready for upload to the SRS like this:

Date	LUID	Security	Source	Accrual	Currency
2022-10-21	LUID_00003DAH	UKT 0 ⅜ 10/22/26	Refinitiv	5.25	GBP

In order to override LUSID data and propagate external results, the first step is call the UpsertStructuredResultData API to upsert this document using the UnitResult/Analytic result type.

Creating a data map in the SRS

Then, you call the CreateDataMap API to create a data map that defines a key for each column in the document as per the table below:

Name (of column)	Address	Data type	Key type	Notes
Date	`UnitResult/Analytic/default/ValuationDate`	`String`	`Leaf`
LUID	`UnitResult/LusidInstrumentId`	`String`	`Unique`	This is the identifier key.
Security	`UnitResult/Instrument/default/Name`	`String`	`Leaf`
Source	`UnitResult/Notes`	`String`	`Leaf`
	`UnitResult/Valuation/InstrumentAccrued`	`Result0D`	`CompositeLeaf`	The `ResultOD` data type consists of an amount and a currency, so these are defined and mapped as separate ‘child’ keys below.
Accrual	`UnitResult/Valuation/InstrumentAccrued/Amount`	`String`	`Leaf`
Currency	`UnitResult/Valuation/InstrumentAccrued/Ccy`	`String`	`Leaf`

Note the following:

The first part of a data map key Address must be UnitResult.
Subsequent parts of the Address must match built-in LUSID metrics in order to override them.
The Name must match the column header in the SRS document.
The Name of the CompositeLeaf Key type is deliberately omitted. Instead, the two ‘child’ keys are mapped to the appropriate columns in the document.
Data maps are version-controlled and immutable; once upserted into LUSID, they cannot be changed. You must upsert a new version.

As an alternative to the API, creating this data map using the Python SDK might look like this:

srs_api = api_factory.build(la.StructuredResultDataApi)
data_map_key = lm.DataMapKey(version = "1.0.0", code = "datamapkey")
data_map = lm.DataMapping(
    data_definitions = [
        lm.DataDefinition(address="UnitResult/Analytic/default/ValuationDate", name = "Date", data_type = "string", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/LusidInstrumentId", name = "LUID", data_type = "string", key_type = "Unique"),
        lm.DataDefinition(address="UnitResult/Instrument/default/Name", name = "Security", data_type = "string", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/Notes", name = "Source", data_type = "string", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/Valuation/InstrumentAccrued", data_type = "Result0D", key_type = "CompositeLeaf"),
        lm.DataDefinition(address="UnitResult/Valuation/InstrumentAccrued/Amount", name = "Accrual", data_type = "decimal", key_type = "Leaf"),
        lm.DataDefinition(address="UnitResult/Valuation/InstrumentAccrued/Ccy", name = "Currency", data_type = "string", key_type = "Leaf"),
    ]
)
try:    
    response = srs_api.create_data_map(
        scope = "srs-accrual", 
        request_body = {
            "data-map-request-1": lm.CreateDataMapRequest(
                id = data_map_key,
                data = data_map
            ),
        }
    )
except lu.ApiException as e:
    display(json.loads(e.body))

Upserting a recipe that can locate data in the SRS

The next step is to call the UpsertConfigurationRecipe API to create a recipe (or amend an existing one) to have a result data rule, for example:

"resultDataRules": [
  {
    "resourceKey": "UnitResult/*",
    "supplier": "Client",
    "dataScope": "srs-accrual",
    "documentCode": "instrumentaccrued-results",
    "quoteInterval": "1D",
    "documentResultType": "UnitResult/Analytic",
    "resultKeyRuleType": "ResultDataKeyRule"
  }
],
...

Performing a valuation

Now when you call the GetValuation API to perform a valuation you can request metrics from the SRS as well as, or instead of, those built-in to LUSID. For example, your valuation request might contain the following metrics:

Metric	Data comes from...	Notes
`Analytic/default/ValuationDate`	LUSID	You could pull this data from the SRS instead using the `UnitResult/Analytic/default/ValuationDate` metric.
`Instrument/default/LusidInstrumentId`	LUSID	You cannot pull this data from the SRS because it is an identifier.
`Instrument/default/Name`	LUSID	You could pull this data from the SRS instead using the `UnitResult/Instrument/default/Name` metric.
`Holding/default/Units`	LUSID	You cannot pull this data from the SRS because it does not exist in the document.
`Holding/default/cost`	LUSID	You cannot pull this data from the SRS because it does not exist in the document.
`UnitResult/Notes`	SRS	You cannot pull this data from LUSID because it does not exist.
`UnitResult/Valuation/InstrumentAccrued`	SRS	This metric overrides `Valuation/InstrumentAccrued` to prefer your SRS accrual result for a single unit of the instrument.
`Valuation/InstrumentAccrued`	SRS	This metric has to be included in the report in order for `Valuation/PV` to be computed properly; note, however, that LUSID's value is overridden by the SRS value.
`Valuation/PV`	LUSID	LUSID computes PV from any change in market value plus the accrual amount from the SRS scaled to your holding, ignoring it's own calculation of accrual.

Your valuation report would look like this:

Note the following:

The PV reflects the external calculation of accrual, scaled by the holding.
The valuation report is augmented by the extra note from the SRS identifying the data source of the accrual.

Keywords: structured results store, SRS, results, external data