Extending LUSID's data model using custom entities

Views:

LUSID has a set of built-in entities that represent real-world objects central to the task of managing investment data, such as portfolios, instruments, quotes and so on.

You can create a custom entity to model a real-world object or concept that LUSID does not natively represent, for example corporate office location. Note the following:

A custom entity must have a custom entity type that specifies values for two built-in data fields, displayName and description, and defines user-specified data fields. Once created, you can modify a custom entity type, but only with caveats.
A custom entity must have at least one identifier. Under the hood, each identifier is defined as a property with a constraintStyle of Identifier and a 3-stage property key in the CustomEntity domain.
You can create relationships between a custom entity and certain other types of entity, for example between a corporate office location and the person entities allowed to work there.
You can apply access metadata to a custom entity. This may be a more effective way of restricting access to custom entity data stored in LUSID.
LUSID stores custom entity data bitemporally and you can retrieve historical information by rolling back the as at timeline in the normal way. However, LUSID does not know how to interpret your custom entity data. It cannot be used to inform holding calculations or valuation operations.
Custom entities are not feature-complete yet.

Providing you have appropriate access control permissions, you can interact with a custom entity (and any relationships it might have):

Using the LUSID API endpoints in the Custom Entity and Custom Entity Definitions collections.
If you have a Luminesce license, using read and write providers you create yourself.

Defining a custom entity type

The first task is to create a custom entity type defining core characteristics. You can then create as many custom entities of this type as you need.

To do this, obtain an API token and call the CreateCustomEntityDefinition API for your LUSID domain, specifying a unique name for the type, values for the built-in data fields displayName and description, and defining user-specified data fields. Note LUSID automatically prefixes the entity type name with ~ to distinguish it from current or future built-in entity types.

User-specified data fields are analogous to, though not the same as, properties. Each can:

Be required or not.
Have a lifeTime of Perpetual or TimeVariant (that is, expected to vary during different time periods). More information.
Have a data type of either String, Boolean, DateTime or Decimal.
Have a collectionType of Array to store multiple values (note that array values are ordered and may contain duplicates).

Note: Unlike properties, user-specified data fields cannot be scoped (that is, entitled) independently of parent custom entities.

For example, to create a custom entity type to represent the concept of corporate office location:

curl -X POST "https://<your-domain>.lusid.com/api/api/customentities/entitytypes"
 -H "Authorization: Bearer <your-API-access-token>"
 -H "Content-Type: application/json-patch+json"
 -d '{
  "entityTypeName": "Office",
  "displayName": "Office location",
  "description": "An office or branch location",
  "fieldSchema": [
    {
      "name": "address",
      "lifetime": "Perpetual",
      "type": "String",
      "required": false,
      "description": "The address of the location"
    },
    {
      "name": "seatingCapacity",
      "lifetime": "TimeVariant",
      "type": "Decimal",
      "required": false,
      "description": "The seating capacity of the location"
    },
    {
      "name": "isHeadOffice",
      "lifetime": "TimeVariant",
      "type": "Boolean",
      "required": true,
      "description": "Whether or not the location is a head office"
    },
    {
      "name": "Amenities",
      "lifetime": "TimeVariant",
      "type": "String",
      "collectionType": "Array",
      "required": false,
      "description": "A list of facilities for staff"
    }
  ]
}'

Providing the request is successful, the response identifies the entity type as ~Office:

{
    "entityTypeName": "Office",
    "displayName": "Office location",
    "description": "An office or branch location",
    "entityType": "~Office",
    "fieldSchema": [
        {
            "name": "isHeadOffice",
            "lifetime": "TimeVariant",
            "type": "Boolean",
            "required": true,
            "description": "Whether or not the location is a head office"
        },
        {
            "name": "seatingCapacity",
            "lifetime": "TimeVariant",
            "type": "Decimal",
            "required": false,
            "description": "The seating capacity of the location"
        },
        {
            "name": "address",
            "lifetime": "Perpetual",
            "type": "String",
            "required": false,
            "description": "The address of the location"
        },
        {
           "name": "Amenities",
           "lifetime": "TimeVariant",
           "type": "String",
           "collectionType": "Array",
           "required": false,
           "description": "A list of facilities for staff"
        }
    ]
}

Understanding identifiers

A custom entity must have at least one identifier. Consider the following JSON fragment defining One Carter Lane, a corporate location that is both FINBOURNE Technology's London office and European headquarters, and so has an identifier for each context:

"displayName": "One Carter Lane",
"description": "FINBOURNE office and headquarters",
"identifiers": [
  {
    "identifierScope": "Location",
    "identifierType": "OfficeId",
    "identifierValue": "London"
  },
  {
    "identifierScope": "Location",
    "identifierType": "HeadquartersId",
    "identifierValue": "Europe"
  }
],
...

An identifier consists of three components: an identifierScope, identifierType and identifierValue. The values you assign to these components combine to uniquely identify One Carter Lane in each of the contexts in which it operates. In this example:

To uniquely identify One Carter Lane as a ...	identifierScope	identifierType	IdentifierValue
Office location	`Location`	`OfficeId`	`London`
Headquarters	`Location`	`HeadquartersId`	`Europe`

For each identifier you intend to give a custom entity, you must first call the CreatePropertyDefinition API to create a property type with the following mandatory characteristics:

A domain of CustomEntity.
A scope with the identifierScope value, for example Location.
A code with the identifierType value, for example OfficeId or HeadquartersId.
A constraintStyle of Identifier.
A lifeTime of Perpetual.

Note you can call the SearchProperties API with a suitable filter to find all the property types that have been created as identifiers for custom entities, in case a suitable definition already exists:

curl -X GET "https://<your-domain>.lusid.com/api/api/search/propertydefinitions?filter=constraintStyle%20eq%20%27Identifier%27%20and%20domain%20eq%20%27CustomEntity%27"
  -H "Authorization: Bearer <your-API-access-token>"

Creating a property type to constitute an 'Office location' identifier

The following call creates a property type with a 3-stage property key of CustomEntity/Location/OfficeId:

curl -X POST "https://<your-domain>.lusid.com/api/api/propertydefinitions"
 -H "Authorization: Bearer <your-API-access-token>"
 -H "Content-Type: application/json-patch+json"
 -d '{
  "domain": "CustomEntity",
  "scope": "Location",
  "code": "OfficeId",
  "displayName": "Office ID",
  "dataTypeId": {"scope": "system", "code": "string"},
  "lifeTime": "Perpetual",
  "constraintStyle": "Identifier",
  "description": "Identifier property used to identify custom entities that are office locations"
 }'

Creating a property type to constitute a 'Headquarters' identifier

The following call creates a property type with a 3-stage property key of CustomEntity/Location/HeadquartersId:

curl -X POST "https://<your-domain>.lusid.com/api/api/propertydefinitions"
 -H "Authorization: Bearer <your-API-access-token>"
 -H "Content-Type: application/json-patch+json"
 -d '{
  "domain": "CustomEntity",
  "scope": "Location",
  "code": "HeadquartersId",
  "displayName": "Headquarters ID",
  "dataTypeId": {"scope": "system", "code": "string"},
  "lifeTime": "Perpetual",
  "constraintStyle": "Identifier",
  "description": "Identifier property used to identify custom entities that are headquarters"
 }'

Creating a custom entity with unique identifier values

You can now call the UpsertCustomEntity API to create a custom entity belonging to a custom entity type. Note the following:

The underlying custom entity type is identified by its type name in the URL of the UpsertCustomEntity API, in this case /api/customentities/~Office (note the ~ prefix).
The identifierScope of each identifier must be the scope of its underlying property type, in this case Location.
The identifierType of each identifier must be the code of its underlying property type, in this case OfficeId or HeadquartersId.
The identifierValue of each identifier must be unique among all identifiers with the same identifierScope and identifierCode, in this case London or Europe.

Note: You can call the UpsertCustomEntities API to batch upsert multiple custom entities, and decide whether to fail the entire operation if one fails validation.

For example, to create a custom entity representing One Carter Lane:

curl -X POST "https://<your-domain>.lusid.com/api/api/customentities/~Office"
 -H "Authorization: Bearer <your-API-access-token>"
 -H "Content-Type: application/json-patch+json"
 -d '{
  "displayName": "One Carter Lane",
  "description": "FINBOURNE office and regional headquarters",
  "identifiers": [
    {
      "identifierScope": "Location",
      "identifierType": "OfficeId",
      "identifierValue": "London"
    },
    {
      "identifierScope": "Location",
      "identifierType": "HeadquartersId",
      "identifierValue": "Europe"
    }
  ],
  "fields": [
    {
      "name": "address",
      "value": "One Carter Lane, London, EC4V 5ER"
    },
    {
      "name": "seatingCapacity",
      "value": 150,
      "effectiveFrom": "2021-08-01T00:00:00.00Z"
    },
    {
      "name": "isHeadOffice",
      "value": true,
      "effectiveFrom": "2021-08-01T00:00:00.00Z"
    },
    {
      "name": "Amenities",
      "value": ['Bike parking', 'Kitchenette', 'Table tennis'],
      "effectiveFrom": "2021-08-01T00:00:00.00Z"
    }
  ]
}'

Note the use of the effectiveFrom field to give time-variant user-specified data fields a 'valid from' date.

Providing the request is successful, the response confirms the date ranges that custom entity fields are valid between:

{
    "href": "https://<your-domain>.lusid.com/api/api/customentities/~Office",
    "entityType": "~Office",
    "version": {
        "effectiveFrom": "0001-01-01T00:00:00.0000000+00:00",
        "asAtDate": "2022-09-20T14:47:45.0500820+00:00"
    },
    "displayName": "One Carter Lane",
    "description": "FINBOURNE office and regional headquarters",
    "identifiers": [
        {
            "identifierScope": "Location",
            "identifierType": "OfficeId",
            "identifierValue": "London",
            "effectiveFrom": "0001-01-01T00:00:00.0000000+00:00",
            "effectiveUntil": "9999-12-31T23:59:59.9999999+00:00"
        },
        {
            "identifierScope": "Location",
            "identifierType": "HeadquartersId",
            "identifierValue": "Europe",
            "effectiveFrom": "0001-01-01T00:00:00.0000000+00:00",
            "effectiveUntil": "9999-12-31T23:59:59.9999999+00:00"
        }
    ],
    "fields": [
        {
            "name": "address",
            "value": "One Carter Lane, London, EC4V 5ER",
            "effectiveFrom": "0001-01-01T00:00:00.0000000+00:00",
            "effectiveUntil": "9999-12-31T23:59:59.9999999+00:00"
        },
        {
            "name": "Amenities",
            "value": [
              "Bike parking",
              "Kitchenette",
              "Table tennis"
            ],
            "effectiveFrom": "2021-08-01T00:00:00.0000000+00:00",
            "effectiveUntil": "9999-12-31T23:59:59.9999999+00:00"
        },
        {
            "name": "isHeadOffice",
            "value": true,
            "effectiveFrom": "2021-08-01T00:00:00.0000000+00:00",
            "effectiveUntil": "9999-12-31T23:59:59.9999999+00:00"
        },
        {
            "name": "seatingCapacity",
            "value": 150.0,
            "effectiveFrom": "2021-08-01T00:00:00.0000000+00:00",
            "effectiveUntil": "9999-12-31T23:59:59.9999999+00:00"
        }
    ],
   ...
}

Retrieving custom entities and their relationships

You can retrieve a particular custom entity using the GetCustomEntity API.

You can retrieve all the custom entities of a particular type using the ListCustomEntities API, or perform a filter operation to only retrieve a matching set. Custom entities support filtering on user-specified data field values using the fields keyword, for example:

fields[streetAddress] startswith 'One'

Note if you create relationships between a custom entity and other entities you can retrieve them using the GetCustomEntityRelationships API.

Modifying an existing custom entity type

You can optionally modify a custom entity type using the UpdateCustomEntityDefinition API. Please note, however, this is a risky operation if custom entity instances of this type already exist:

If you modify the data type of a field (for example, from String to DateTime), values of custom entities that do not match the new type cannot be retrieved. If you revert the change, original values are returned even if they have been updated in the meantime (so your updates are lost).
If you modify the lifeTime of a data field (for example, from TimeVariant to Perpetual), values cannot be retrieved. If you revert the change, original values are returned even if they have been updated in the meantime.
If you modify the required status (for example, from True to False), values can be retrieved, but note that the new validation applies to all future updates to values.

A note on unsupported features

Custom entities are in the early phase of their development and do not yet support:

Properties.
Deleting a custom entity type.
Identifier properties that are specific to the custom entity type (identifier properties in the CustomEntity domain can currently be applied to custom entity instances of any type).
Filtering on identifier properties when retrieving custom entities.

Keywords: custom entity, identifier type, identifier scope, extensible data model