How do I read and write quote metadata fields using Luminesce?

You can give every quote in your domain extra information, such as the vendor that supplied it, the asset class it belongs to, or an internal quality score. Once these quote metadata fields are defined in LUSID, you can add them to the Lusid.Instrument.Quote and Lusid.Instrument.Quote.Writer providers as ordinary columns. You can then select them, filter on them and write to them like any other column on the provider.

Note: Quote metadata fields and the LUSID APIs that define them are currently Early Access.

Understanding quote metadata fields

A quote's standard fields, such as its scope, instrument, quote type, field, effective date, value and unit, say what the price is. Metadata fields record extra information about the quote.

You define metadata fields once for your domain, in LUSID's quote market data field configuration. This is a schema shared by every quote in your domain. Each quote can then hold a value for any field in the schema. Each field definition has:

Note: LUSID currently only supports metadata fields whose data type has a string value type, such as system/string or system/iso4217Currency. These appear in Luminesce as Text columns.

Prerequisites

Luminesce doesn't create metadata fields. Before you can add them to the quote providers, you must define them in LUSID by calling the UpdateMarketDataFieldConfiguration API with the Quote market data category. For example, to add three fields:

curl -X POST "https://<your-domain>.lusid.com/api/api/marketdata/fieldconfiguration/Quote/$update"
  -H "Authorization: Bearer <your-API-access-token>"
  -H "Content-Type: application/json-patch+json"
  -d '{
  "add": {
    "metadataFields": [
      { "fieldName": "vendor", "displayName": "Vendor", "description": "The vendor that supplied the quote",
        "dataTypeId": { "scope": "system", "code": "string" } },
      { "fieldName": "assetClass", "displayName": "Asset Class", "description": "The asset class of the quoted instrument",
        "dataTypeId": { "scope": "system", "code": "string" } },
      { "fieldName": "Vendor.Score/Rank", "displayName": "Vendor score rank", "description": "The vendor's score rank for the quote",
        "dataTypeId": { "scope": "system", "code": "string" } }
    ]
  }
}'

The request can also have update (field definitions to replace) and remove (field names to delete) sections. To see the current fields, call GET /api/api/marketdata/fieldconfiguration/Quote.

The rest of this article uses these three fields.

Methods

You add metadata fields to the providers using Sys.Admin.Lusid.Provider.Configure, the same provider you use to configure other LUSID entity providers to read and write properties. Configuring Lusid.Instrument.Quote configures Lusid.Instrument.Quote.Writer at the same time, and the change affects your entire domain.

For each field you configure, Sys.Admin.Lusid.Provider.Configure reads the field's definition and fills in anything you don't supply. See Appendix A for the columns you can supply.

Method 1: Add every metadata field

If you don't pass a Configuration table, the providers get a column for every metadata field currently in the schema:

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote';

With the fields defined above, the providers get three new columns: Vendor, AssetClass and VendorScoreRank.

Method 2: Add a subset of fields

To add only some fields, list their names as keys:

@fields = values
('vendor'),
('Vendor.Score/Rank');

@config = select column1 as [Key] from @fields;

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote'
and Configuration = @config
and WriteAction = 'Set';

Method 3: Choose column names, main columns and descriptions

@fields = values
('vendor', 'QuoteVendor', true, 'The vendor that supplied the quote'),
('assetClass', null, false, null),            -- column name defaults to AssetClass
('Vendor.Score/Rank', 'ScoreRank', false, null);

@config = select column1 as [Key], column2 as Name, column3 as IsMain, column4 as Description from @fields;

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote'
and Configuration = @config
and WriteAction = 'Set';

Method 4: Create a separately named provider

To build a separate provider pair with its own set of columns, pass ProviderNameExtension. This leaves the default Lusid.Instrument.Quote provider unchanged. The following example creates Lusid.Instrument.Quote.Vendor and Lusid.Instrument.Quote.Vendor.Writer:

@config = select 'vendor' as [Key];

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote'
and ProviderNameExtension = 'Vendor'
and Configuration = @config
and WriteAction = 'Set';

Updating and removing fields

To pick up fields added to the schema since you last configured the providers, while keeping your existing column names, use Modify with no Configuration table:

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote'
and WriteAction = 'Modify';

To remove a field:

@config = select 'assetClass' as [Key];

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote'
and Configuration = @config
and WriteAction = 'Remove';

To remove every metadata field column and return the providers to their standard columns:

select * from Sys.Admin.Lusid.Provider.Configure
where Provider = 'Lusid.Instrument.Quote'
and WriteAction = 'Reset';

See Appendix B for exactly what each write action does.

Checking the result

The Error column of Sys.Admin.Lusid.Provider.Configure is empty on success. On failure nothing is saved, and Error explains why (see Appendix D). If the new configuration drops a column that was configured before, the query also returns a warning listing the columns the providers will stop returning.

Shortly after a successful configuration, the providers re-register with the new columns. You may need to select the refresh button in the Luminesce catalog to see them.

How the providers change

Once you configure metadata fields:

• Lusid.Instrument.Quote returns one extra column per configured field, after the standard quote columns.

• Lusid.Instrument.Quote.Writer accepts the same columns in its ToWrite table and returns them in its results, after the standard quote columns and before WriteAction and the write result columns such as WriteErrorCode and WriteErrorDetail.

• If a quote has no value for a field, its column is null.

• Fields marked IsMain are returned by select ^.

• If you delete a field from the schema after configuring it, the column stays on the providers but is always null. Filtering on it doesn't fail, but writing a value to it does (see Writing metadata fields). Run Sys.Admin.Lusid.Provider.Configure with Remove, or with Set and no Configuration table, to drop the column.

The number of configured metadata fields has no meaningful effect on query performance. LUSID always returns every quote's metadata field values when it lists quotes. Configuring fields only exposes those values as columns, and doesn't trigger extra calls to fetch them.

Reading and filtering quotes

The following examples use the fields defined above, configured with their default column names.

To list quotes with their metadata fields:

select InstrumentId, QuoteEffectiveAt, Value, Unit, Vendor, AssetClass, VendorScoreRank
from Lusid.Instrument.Quote
where QuoteScope = 'my-quotes';

You filter on metadata field columns like any other column:

select InstrumentId, Value, Vendor, AssetClass
from Lusid.Instrument.Quote
where QuoteScope = 'my-quotes'
and Vendor in ('bbg', 'refinitiv')
and AssetClass = 'Equity';

Field names that contain . or / need no special handling. You filter on the column name:

select InstrumentId, VendorScoreRank
from Lusid.Instrument.Quote
where QuoteScope = 'my-quotes'
and VendorScoreRank like 'A%';

To find quotes with no value for a field:

select InstrumentId, Value
from Lusid.Instrument.Quote
where QuoteScope = 'my-quotes'
and Vendor is null;

Writing metadata fields

To write metadata field values, include the metadata field columns in the table of data you pass to Lusid.Instrument.Quote.Writer:

@quotes =
select 'my-quotes' as QuoteScope, 'Client' as Provider, '' as PriceSource, 'my-instrument-1' as InstrumentId,
'ClientInternal' as InstrumentIdType, 'Price' as QuoteType, 'mid' as Field, #2024-01-01# as QuoteEffectiveAt,
101.0 as Value, 'GBP' as Unit, 'bbg' as Vendor, 'Equity' as AssetClass, 'A+' as VendorScoreRank, 'Upsert' as WriteAction
union all
select 'my-quotes', 'Client', '', 'my-instrument-2', 'ClientInternal', 'Price', 'mid', #2024-01-01#,
102.0, 'GBP', 'refinitiv', null, null, 'Upsert';

select * from Lusid.Instrument.Quote.Writer where ToWrite = @quotes;

The results return the metadata field values that were sent for each quote.

Note the following:

• An upsert replaces the whole quote. LUSID identifies a quote by its quote series (Provider, PriceSource, InstrumentId, InstrumentIdType, QuoteType and Field) and its QuoteEffectiveAt. Upserting a quote that already exists writes a complete new version of it. Every field, including Value, Unit, ScaleFactor, Lineage and the metadata fields, comes from the row you write. Nothing is carried over from the previous version, so a column you leave null doesn't keep its earlier value. For example, re-upserting my-instrument-1 with Vendor set to null and AssetClass set to 'Fixed Income' leaves the quote with only AssetClass.

• null means no value. Luminesce doesn't send a null metadata field to LUSID, so the new version of the quote has no value for that field.

• Earlier versions are kept. Quotes are bitemporal, so you can still see the version from before an upsert by querying Lusid.Instrument.Quote with an earlier AsAt.

• Each quote succeeds or fails on its own. If a quote has a value for a configured column whose field has been removed from the schema, that quote fails and WriteErrorDetail names the field. Other quotes in the same write still succeed.

• Deleting a quote removes its metadata. When WriteAction is Delete, Luminesce ignores the metadata field columns.

To see which quotes failed:

@response = select * from Lusid.Instrument.Quote.Writer where ToWrite = @quotes;

select InstrumentId, WriteErrorCode, WriteErrorDetail
from @response
where WriteErrorCode <> 0;

Appendix A: Configuration columns

The AsAt column has no effect on metadata fields. Sys.Admin.Lusid.Provider.Configure always reads the latest field definitions.

Appendix B: Write actions

Appendix C: Column names

If you don't supply a Name, the column is named after the field in PascalCase, with any characters that aren't letters or digits removed:

Note the following:

• A column name must start with a letter or underscore, and contain only letters, digits and underscores.

• A column name can't duplicate another configured column, or one of the providers' standard columns such as Value, Unit, Provider, QuoteScope or WriteAction. The comparison ignores case. If a field's default name clashes, supply a Name for it.

Appendix D: Configuration errors