Recipes: Locating market data

A recipe has a market section that you must populate so LUSID can locate and retrieve suitable market data for each type of instrument you wish to value:

  • For simple exchange-traded instruments such as equities, you are likely to require market prices held in the LUSID Quote Store. Note you can override market prices in some circumstances using portfolio-level data such as unit cost or last traded price.

  • For complex exchange-traded or OTC instruments, you are likely to require interest rate, discounting or credit curves and/or volatility surfaces held in the LUSID Complex Market Data Store.

  • For any instrument (including a cash instrument) held in a different currency to the portfolio currency, you are likely to require FX rates held in the LUSID Quote Store.

Note: You can call APIs to check the quality/quantity of market data required for the valuation operation you are trying to perform. See how to do this.

By default, a recipe must be able to locate market data that is valid ~ one day prior to each date in the valuation schedule. The precise validity period is determined by the valuation time. For example, if you are valuing a portfolio containing a single equity on:

  • 7 March 2022, the time defaults to 00:00:00 and the recipe must be able to locate at least one market price for that equity (and an FX rate, if applicable) effective 6 March 00:00:00 -> 7 March 00:00:00. Prices effective 5 March or earlier, or 7 March later than midnight are not valid.

  • 7 March 2022 17:00:00, the recipe must be able to locate at least one market price effective 6 March 00:00:00 -> 7 March 17:00:00.

  • 7 March 2022 23:59:59, the recipe must be able to locate at least one market price effective 6 March 00:00:00 -> 7 March 23:59:59.

In other words, there is an automatic look back, or ‘grace’, period of between 24 and 48 hours. You can specify an explicit quoteInterval to look back further than this if you wish.

Understanding market data rules

For each category of market data (that is, market price, FX rate, interest rate curve and so on) you are required to locate you should specify a market data rule in the marketRules array and/or groupedMarketRules array of the market section of your recipe. Consider the following example, of three market data rules in the marketRules array intended for a portfolio valuation on 7 March 2022:

Note the following:

  • Each market data rule is self-contained and must contain all the information required to locate a particular category of market data in an appropriate store.

  • Market data rules are processed in order. The first matching rule found for a category of market data is used (as determined by its key).

  • You can provide multiple rules for the same key in order to provide fallback logic. For example, you could have two rules for Quote.Figi.* that are identical except the first has a supplier of DataScope and the second a supplier of Bloomberg. Prices from Refinitiv would be preferred, but the recipe would fall back to Bloomberg if no Refinitiv prices could be found.

  • You can provide more sophisticated fallback behavior by specifying market data rules in the groupedMarketRules array in addition to the marketRules array. More information.

  • You can perform operations such as taking the highest or latest price from a group of providers, or even synthesising an average price, by specifying market data rules in the groupedMarketRules array. More information.

  • You can specify ultimate fallback behaviour for market prices (not FX rates or complex market data) by using pricing data stored at the portfolio level, such as unit cost or last traded price. More information.

Summary of fields in a market data rule

For information on all the fields and allowed values in a market data rule, examine the API reference (expand the configurationRecipe/market/marketRules section):

The following table provides a summary. Note in particular that:

  • The key field defines the category of market data to locate.

  • Given the category, all the other fields define the rule's criteria for retrieving appropriate data.

Market data rule field

Status

Applicable LUSID market data store(s)

Explanation

Matching market data field in store

key

Mandatory

Quote, Complex Market Data

A dot-separated string specifying the category of market data to locate, and implicitly the store in which to locate it. See the table below for more information.

n/a

supplier

Mandatory

Quote, Complex Market Data

The market data vendor. This value must match the value of the market data provider field in the appropriate store. See a list of valid vendors.

provider

priceSource

Optional

Quote, Complex Market Data

The sub-supplier to the market data vendor (above). This value must match the value of the market data priceSource field if it is specified, for example  Tradeweb or RRPS for Refinitiv DataScope.

priceSource

dataScope

Mandatory

Quote, Complex Market Data

This value must match the scope encapsulating the market data in the appropriate store.

scope

quoteType

Mandatory

Quote

If the rule locates market data in the Quote Store, this value must match the value of the quoteType field, for example Price or Rate.

If the rule locates market data in the Complex Market Data Store, omit this field.

quoteType

field

Mandatory

Quote

If the rule locates market data in the Quote Store, this value must match the value of the field field, for example bid, mid or ask. Note this value is case-sensitive, so Bid does not match bid. See valid values for different vendors.

If the rule locates market data in the Complex Market Data Store, omit this field.

field

quoteInterval

Optional

Quote, Complex Market Data

Specifies a 'look back' period for market data longer than the implicit default of 1D.0D, which looks back to 00:00:00 on the day before the valuation date. If no market data is found within this range then a market data resolver error occurs.

You can use the <startdate>.<enddate> tenor syntax to look back further, for example 1W.0D to look back seven days from the valuation date, or 0D.0D to look back to the beginning of the valuation date itself. If you want to consider the latest data invalid instead you can change the end date, for example 2D.1D to look back two days before the valuation date, but not to include the valuation date itself. For more information, see the API documentation.

Note if you choose BD and your valuation schedule specifies a holiday calendar, LUSID ignores non-business days and looks back further.

n/a

sourceSystem

Optional

Quote, Complex Market Data

Defaults to Lusid to locate market data in either the Quote Store or Complex Market Data Store. Do not change this value unless you want to locate data in a 3rd party store.

n/a

More information on the key field

The following table explains the key field in a market data rule in more detail:

Market data category

LUSID market data store

key field syntax

Examples

Market price

Quote Store

Quote.<IdentifierType>.<IdentifierValue>

  • Quote.RIC.* to locate prices loaded with an instrumentIdType of RIC.

  • Quote.LusidInstrumentId.* to locate prices loaded with an instrumentIdType of LusidInstrumentId.

  • Quote.Isin.GB0031348658 to locate prices loaded with an instrumentIdType of Isin and instrumentId of GB0031348658.

Note: Quote.*.* does NOT locate any price loaded using any unique or non-unique instrument identifier.

FX spot rate

Fx.<FgnCcy>.<DomCcy>

  • Fx.USD.GBP to locate rates loaded with an instrumentIdType of CurrencyPair and instrumentId of USD/GBP (note forward slash).

  • Fx.*.* to locate any FX rate.

Inflation fixing

Data loaded with quoteType of Index

Inflation.InflationIndex.<IdentifierType>.<IdentifierValue>

  • Inflation.InflationIndex.UKRPI to locate index fixings loaded with an instrumentIdType of ClientInternal and instrumentId of UKRPI.

  • Inflation.InflationIndex.* to locate any index fixing.

Data loaded with quoteType of Ratio

Inflation.InflationRatio.<IdentifierType>.<IdentifierValue>

  • Inflation.InflationRatio.ClientInternal.MX.UDI to locate ratio fixings loaded with an instrumentIdType of ClientInternal and instrumentId of MX.UDI.

  • Inflation.InflationRatio.* to locate any ratio fixing.

Data loaded with quoteType of InflationAssumption

Inflation.InflationAssumption.<IdentifierType>.<IdentifierValue>

  • Inflation.InflationAssumption.ClientInternal.IPCA to locate assumption fixings loaded with an instrumentIdType of ClientInternal and instrumentId of IPCA.

  • Inflation.InflationAssumption.* to locate any assumption fixing.

Discount factor curve

Complex Market Data Store

Rates.<Ccy>.<Ccy>OIS

  • Rates.USD.USDOIS to locate USD discount factor curves with a marketAsset of USD/USDOIS.

  • Rates.*.* to locate any discounting curve.

FX forward curve

FxForwards.<DomCcy>.<FgnCcy>.FxFwdCurve

  • FxForwards.GBP.USD.FxFwdCurve to locate a curve with a marketAsset of GBP/USD/FxFwdCurve.

  • FxForwards.*.*.* to locate any curve.

Interest rate projection curve

Rates.<Ccy>.<Tenor>.<Index>

  • Rates.GBP.1Y.LIBOR to locate one year GBP - LIBOR IR projection curves loaded with a marketAsset of GBP/1Y/LIBOR.

  • Rates.*.*.* to locate any IR projection curve.

Credit curve

Credit.<REDCode>.<Ccy>.<Seniority>.<RestructuringClause>

  • Credit.FF667M.USD.SNR.MR to locate senior CDS with modified restructuring loaded with a marketAsset of FF667M/USD/SNR/MR.

  • Credit.*.*.*.* to locate any credit curve.

Equity volatility surface

EquityVol.<RIC>.<Ccy>.<N-or-LN>

  • EquityVol.AAPL.USD.LN to locate Apple log normal volatility surfaces loaded with a marketAsset of AAPL/USD/LN.

  • EquityVol.*.*.* to locate any equity volatility surface.

FX volatility surface

FxVol.<DomCcy>.<FgnCcy>.<N-or-LN>

  • FxVol.GBP.USD.N to locate GBPUSD normal volatility surfaces loaded with a marketAsset of GBP/USD/N.

  • FxVol.*.*.* to locate any FX volatility surface.

Interest rate volatility surface

IrVol.<Ccy>.<N-or-LN>

  • IrVol.USD.LN to locate USD log-normal volatility surfaces loaded with a marketAsset of USD/LN.

  • IrVol.*.* to locate any IR volatility surface.

Resolving market data errors

When you call the GetValuation API, you may see a MarketResolverFailure similar to the one below if one or more market data rules fails to locate suitable market data in an appropriate store. The first step is to make sure field values in the market data rule match those in market data objects according to the table above. If they do, you can try setting the allowPartiallySuccessfulEvaluation pricing model option to True to relax validation and call the API again, though of course missing market data will not be used in calculations:

{"name":"MarketResolverFailure","errorDetails":[{"id":"Failed to resolve market data item [Provider: Edi, PriceSource: , InstrumentId: BBG000C05BD1, InstrumentIdType: Figi, QuoteType: Price, Field: mid].","detail":"Attempted to resolve with supplier [Edi] and sourceSystem [Lusid] in scope [recipe1] for effective date [07/03/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store."}…]",…}

Grouping market data rules and performing preference operations

You can specify market data rules in the groupedMarketRules array as well as, or instead of, the marketRules array:

  • You might specify market data rules in both arrays in order to provide sophisticated fallback behavior from a preferred supplier of quotes to a group of secondary suppliers.

  • You might specify market data rules in just the groupedMarketRules array to perform an operation such as taking the most recent or highest price from a group of suppliers, or even synthesising an average price.

You can of course combine these use cases to perform operations on a group of secondary suppliers.

Consider the following example, of a market section with one market data rule for prices in the marketRules array and three inside a single group in the groupedMarketRules array, intended for a portfolio valuation on 7 March 2022. Note the operation on the group of rules is FirstLatest:

Note the following:

  • A recipe should have at least one market data rule in order to be functional. It can be in the marketRules array, or on its own in a group in the groupedMarketRules array.

  • You can have any number of market data rules in the marketRules array, and any number of rules in any number of groups in the groupedMarketRules array.

  • LUSID reads the marketRules array first, then the groupedMarketRules array.

  • Within the marketRules array, the order of market data rules is significant.

  • Within the groupedMarketRules array, the order of groups is significant. The order of market data rules within a group is significant in some circumstances, depending on the operation being performed.

  • Within the groupedMarketRules array, all market data rules must have the same key and quoteType. One exception is for quoteType with the FirstLatest operation, where you can mix the Price and DirtyPrice quote types.

  • LUSID uses the first matching rule found for a particular category of market data.

The following marketDataKeyRuleGroupOperations are available:

Operation

Explanation

FirstLatest

The rule retrieving the most recent market data item (market price, FX rate and so on) is used. If all items were loaded with exactly the same date and time, the rule specified first in the group is used.

FirstMinimum

The rule retrieving the lowest value market data item is used. If all items have exactly the same value, the rule specified first in the group is used.

FirstMaximum

The rule retrieving the highest value market data item is used. If all items have exactly the same value, the rule specified first in the group is used.

AverageOfAllQuotes

These operations are different in that LUSID synthesises a new market data item rather than using a rule to retrieve an existing item. See the section below.

AverageOfQuotesFound

Example: Synthesising an average market data item

You can request that LUSID retrieves market data items from all the rules in a group and synthesises a new market data item representing an average:

  • Choose the AverageOfAllQuotes operation if you want the request to fail if a single rule in the group fails to retrieve a market data item.

  • Choose the AverageOfQuotesFound operation to ignore failures unless all rules fail to retrieve items.

For example, imagine you have upserted to the LUSID Quote Store the following three market prices for a particular stock:

Data field

Price #1

Price #2

Price #3

Notes

provider

DataScope

Lusid

Bloomberg

priceSource

Tradeweb

BBG

This field is optional when upserting.

instrumentIdType

RIC

RIC

RIC

In order to synthesise an average, all instrument identifier types must be the same.

instrumentId

ABC.N

ABC.N

ABC.N

In order to synthesise an average, all values must identify the same instrument.

quoteType

Price

Price

Price

In order to synthesise an average, all quote types must be the same.

field

open

bid

ask

effectiveAt

2024-01-02T09:00:00Z

2024-01-02T11:00:00Z

2024-01-02T12:00:00Z

metricValue.value

105

1.1

9.7

metricValue.unit

USD

USD

USD

In order to synthesise an average, all currency units must be the same.

scaleFactor

100

10

This field is optional when upserting; it defaults to 1.

lineage

A Refinitiv quote

A BBG quote

This field is optional when upserting.

uploadedBy

00ubs2temhwlYz2lI2p7

00ubk5b5apJVhOlAg2p8

00uhj7Gh8kklio92pl97

This field is automatically populated by LUSID.

asAt

2024-01-02T13:00:00Z

2024-01-02T14:00:00Z

2024-01-02T15:00:00Z

This field is automatically populated by LUSID.

...and created a recipe with the following three market data rules in a group with the AverageOfAllQuotes operation:

"market": {
  "groupedMarketRules": [
    {
      "marketDataKeyRuleGroupOperation": "AverageOfAllQuotes",
      "marketRules": [
        {
          "key": "Quote.RIC.*",
          "supplier": "DataScope",
          "dataScope": "RefinitivPrices",
          "quoteType": "Price",
          "field": "open",
        },
        {
          "key": "Quote.RIC.*",
          "supplier": "Lusid",
          "dataScope": "LusidPrices",
          "quoteType": "Price",
          "field": "bid",
        },
        {
          "key": "Quote.RIC.*",
          "supplier": "Bloomberg",
          "dataScope": "BBGPrices",
          "quoteType": "Price",
          "field": "ask",
        },
      ]
    }
  ]
},
...

Assuming all three original market prices can be located, LUSID synthesises the following average price:

Data field

Synthesised value

Notes

provider

Lusid

The provider is always set to this value, irrespective of the original providers.

priceSource

AverageOfAllQuotes('Tradeweb', '', 'BBG')

The prefix is AverageOfQuotesFound if the other average operation is chosen.

instrumentIdType

RIC

instrumentId

ABC.N

quoteType

Price

field

AverageOfAllQuotes('open', 'bid', 'ask')

The prefix is AverageOfQuotesFound if the other average operation is chosen.

effectiveAt

2024-01-02T12:00:00Z

The latest original datetime is used.

metricValue.value

1.04

In this example, the original scale factors are different, so the synthesised value is unitised. The calculation is as follows:

Original price #1:  105 / 100 = 1.05
Original price #2:  1.1 / 1 = 1.1
Original price #3:  9.8 / 10 = 0.97
Average price:  (1.05 + 1.1 + 0.97) / 3 = 1.04

metricValue.unit

USD

scaleFactor

1

In this example, since the synthesised value is unitised, the scale factor is set to 1. If the original scale factors were the same and not 1, then this field is set to that scale factor.

lineage

AverageOfAllQuotes('A Refinitiv quote', '', 'A BBG quote')

The prefix is AverageOfQuotesFound if the other average operation is chosen.

uploadedBy

SystemComputed-Average

The user is always set to this value, irrespective of the original users.

asAt

2024-01-03T17:00:00Z

The as at datetime of the valuation operation is used.

Note a synthesised market data item is not upserted to the Quote Store, so it cannot be retrieved by an API such as ListQuotesForScope. However, it is available in the manifest for a valuation operation.

Overriding market prices or falling back to portfolio-level pricing data

If the valuation operation you want to perform requires a market price (not an FX rate or complex market data), you can optionally specify override and/or fallback rules to use portfolio-level pricing data such as unit cost or last traded price:

  • You might specify an override rule to use portfolio-level data in preference to a market price stored in the Quote Store.

  • You might specify a fallback rule to use portfolio-level data if a market price cannot be located in the Quote Store.

Rules can be global or specific; that is, apply to every type of instrument or just to a particular type or asset class. Note if you specify a global override rule and no specific overrides then LUSID never attempts to locate market prices in the Quote Store.

Note: If you are valuing multiple portfolios, this is likely to mean they will be valued differently, even with positions in the same underlying instruments.

To do this, specify a holdingPricingInfo object in the pricing object of a recipe (not the market object); see the API reference for more information:

The following pricing data is available for each holding in any portfolio being valued:

Available pricing data

Explanation

UnitCost

The total number of units held divided by the total cost.amount.

UnitAmortisedCost

The total number of units held divided by the total amortisedCost.amount.

LastTradedPrice

The transactionPrice.price of the most recent transaction contributing to the holding.

Example: Specifying override rules

In the following example, if the valuation operation requires market prices:

  • Instruments of type ComplexBond are valued using UnitAmortisedCost.

  • Instruments of type Equity trading in euros are valued using LastTradedPrice.

  • Instruments of every other type are valued using prices located in the Quote Store. Since no global overrideField is provided,  a market data error occurs if no such prices can be found:

"pricing": {
  "holdingPricingInfo": {
    "overrideField": "None",
    "specificOverrides": [
      {
        "dependencySourceFilter": {
          "instrumentType": "ComplexBond"
        },
        "field": "UnitAmortisedCost"
      },
      {
        "dependencySourceFilter": {
          "instrumentType": "Equity",
          "domCcy": "EUR"
        },
        "field": "LastTradedPrice"
      }
    ]
  },
  ...
}

Example: Specifying fallback rules

In the following example, if the valuation operation requires market prices that cannot be located in the Quote Store:

  • Instruments of type Bond are valued using LastTradedPrice.

  • Instruments of every other type are valued using UnitCost:

"pricing": {
  "holdingPricingInfo": {
    "fallbackField": "UnitCost",
    "specificFallbacks": [
      {
        "dependencySourceFilter": {
          "instrumentType": "Bond"
        },
        "field": "LastTradedPrice"
      }
    ]
  },
  ...
}