Troubleshooting a failed request using the Insights service

In this tutorial we'll see how to use LUSID to perform the following task:

“As an administrator, I want to use Insights, LUSID's built-in troubleshooting, monitoring and audit service, to investigate failed requests on behalf of LUSID end users.”

We'll see how to:

• Get the information you need from an end user to start troubleshooting.

• Use the Insights GUI built-in to the LUSID web app.

• Examine request logs to see API requests and responses.

• Examine access logs to diagnose access denied errors.

• Write SQL queries to explore logs using Luminesce, LUSID's data virtualisation service.

• Grant access to FINBOURNE support engineers to your domain.

To perform the operations in this tutorial you require access control permissions, which can most easily be achieved by assigning your LUSID user the built-in lusid-administrator role. This should already be the case if you are the domain owner. If you are informed you do not have a license to perform a particular operation, please contact support.

Getting an error request ID from an end user

If an API request fails in the LUSID web app, an end user is presented with an error message like the one below:

In the LUSID Python SDK, the same failed API request generates an HTTP response like this (response headers redacted for brevity):

{
  "name": "MarketResolverFailure",
  "errorDetails": [
    {
      "id": "Failed to resolve market data item [Provider: Edi, PriceSource: , InstrumentId: BBG000C05BD1, InstrumentIdType: Figi, QuoteType: Price, Field: mid].",
      "detail": "Attempted to resolve with supplier [Edi] in scope [FBNUniversity] for effective date [28/04/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store."
    },
    {
      "id": "Failed to resolve market data item [Provider: DataScope, PriceSource: , InstrumentId: GB0007980591, InstrumentIdType: Isin, QuoteType: Price, Field: mid].",
      "detail": "Attempted to resolve with supplier [DataScope] in scope [FBNUniversity] for effective date [28/04/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store."
    },
    {
      "id": "Failed to resolve market data item [Provider: Lusid, PriceSource: , InstrumentId: LUID_00003D7X, InstrumentIdType: LusidInstrumentId, QuoteType: Price, Field: mid].",
      "detail": "Attempted to resolve with supplier [Lusid] in scope [default] for effective date [28/04/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store."
    }
  ],
  "code": 321,
  "type": "https://docs.lusid.com/#section/Error-Codes/321",
  "title": "One or more failures occurred.",
  "status": 404,
  "detail": "One or more failures occurred. Failed to resolve market data for economic dependency 'Quote:{ {LusidInstrumentId: LUID_00003D7X}, {Figi: BBG000C05BD1}, {Isin: GB0007980591} }:2022-04-28T00:00:00.0000000+00:00'. Further details: \n[(0). Failed to resolve market data item [Provider: Edi, PriceSource: , InstrumentId: BBG000C05BD1, InstrumentIdType: Figi, QuoteType: Price, Field: mid]. Attempted to resolve with supplier [Edi] in scope [FBNUniversity] for effective date [28/04/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store.],\n[(1). Failed to resolve market data item [Provider: DataScope, PriceSource: , InstrumentId: GB0007980591, InstrumentIdType: Isin, QuoteType: Price, Field: mid]. Attempted to resolve with supplier [DataScope] in scope [FBNUniversity] for effective date [28/04/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store.],\n[(2). Failed to resolve market data item [Provider: Lusid, PriceSource: , InstrumentId: LUID_00003D7X, InstrumentIdType: LusidInstrumentId, QuoteType: Price, Field: mid]. Attempted to resolve with supplier [Lusid] in scope [default] for effective date [28/04/2022 00:00:00], quoteInterval [None] and with predicate [AsAt=Latest], but failed because failed to find quote in store.]",
  "instance": "https://acmecorp.lusid.com/app/insights/logs/0HMH8I0A24HIB:00000006",
  "extensions": {}
}

To troubleshoot the problem you require the error request ID, which is the final resource in the Insights URL (in this case 0HMH8I0A24HIB:00000006).

Using Insights built-in to the LUSID web app

Sign in to the LUSID web app as an administrator and open the Insights > Request Logs dashboard from the left-hand menu:

You can:

• Change the order or columns by dragging them.

• Configure which columns are displayed in the dashboard (and save changes) by clicking the Configuration icon (highlighted in red above).

• See more detail for a particular request by clicking the View icon at the end of the record (highlighted in green above).

• Sort most columns by clicking top right in the column header to show an up arrow, again to show a down arrow, and once more to cancel sorting:
   

• Filter columns that have a text box under the header using partial, exact or inverse matches:
   

• Right-click a column header to get a shortcut menu of further operations:
   

Using request logs to examine API requests and responses

You can paste a request ID into the Request ID text box (highlighted in red below) to retrieve the request. Note that if the operation occurred more than one hour ago you will need to extend the From date and time (highlighted in green) to return historical results:

In this example, we can see:

• The Date and Time the end user made the request.

• The API Operation: GetValuation. Note you can correleate API calls together in order to group and track related calls.

• The User ID (you can look this up using the FindUsersById API)

• The calling Application: lusid. Note other possible values in this column refer to other apps in the LUSID ecosystem and can either be intuitive names (for example drive) or codenames (such as shrine for the Access service or lydia for the Identity service). It might be useful to filter out website values in this column since these records just represent the LUSID web app calling APIs under-the-hood.

• The LUSID Error Code: 321. This may be useful if you need to contact FINBOURNE support. For more help on access denied errors (code 186), see this section.

Click the View icon at the end of the record to see the HTTP Request details and Response details on separate tabs (highlighted). Note authentication tokens and other PII is redacted, and no requests or responses are logged for the Identity service or for the Configuration Store due to the quantity of PII:

Using access logs to diagnose access denied errors

End users may commonly encounter access denied errors (LUSID error code 186). This may be because:

• Your LUSID domain does not have a license for a particular feature.

• A particular end user does not have a policy granting permission to use a feature.

Navigate to the Insights > Access Logs dashboard from the left-hand menu to diagnose these errors. As with the request logs, if the error occurred more than one hour ago you will need to extend the From date and time (highlighted in green below) to return historical results. Results from the last 7 days should be relatively quick to retrieve; older results may take longer.

For a missing licence, the Result column displays DoesNotHaveRequiredLicence. The Required Licence Policy ID column contains the license code you need to request from FINBOURNE support, in this case feature-product-pms (underlined in red below):

For a policy, the Result column displays either 

• NoGrantingPolicy
  The end user is missing a policy granting permission to perform the operation.

• DeniedAccessByPolicy
  The end user has a policy that explicitly denies them permission to perform the operation.

A missing policy may either be a feature policy (granting access to API endpoints) or a data policy (granting access to data). More information.

• If the Action column contains the value Feature/LUSID/Execute (red bar below) then you need to create a feature policy for the end user. The API endpoint to grant permission is recorded in the Resource column; for the first record below, this is the LUSID ListPortfolios API.

• If the Action column contains a value other than Feature/<app>/Execute (green bar below) then you need to create a data policy for the end user.

Writing Luminesce SQL queries to examine logs

If you have a license to use Luminesce, LUSID's data virtualization service, and you are comfortable writing SQL then it may be more convenient to query one or more of the logging providers supplied by FINBOURNE:

You can write a Luminesce SQL query using any of these tools, including a GUI built in to the LUSID web app under the Data Virtualisation menu. For more information on the precise syntax of a Luminesce SQL query, see this article.

For example, the following query uses the Lusid.Logs.Metrics.Entitlement provider to retrieve all the access requests that failed in the last week because a suitable license could not be found. Note we recommend using the StartAt (and potentially EndAt) parameters to specify a date range with most queries, since data volumes can be very large:

@@oneWeekAgo = select date('now', '-7 days');
select * from Lusid.Logs.Metrics.Entitlement where StartAt = @@oneWeekAgo and Result = 'DoesNotHaveRequiredLicence';

Contacting FINBOURNE support and granting access to your domain

If you still need help you can contact our technical support team from the Help > Contact Support menu. If you choose to communicate via email you can access replies from the Help > Support Inbox menu:

To grant FINBOURNE support engineers access to your domain (with lusid-administrator privileges) to help solve problems, navigate to the Help > Data Access dashboard and click the Grant access button. Each grant can last between 1 and 14 days, and you can revoke access at any time: