Using stacks to identify and manage recurring issues

You can define a stacking key within a task definition to populate a stacking key field for any exception tasks it creates. For example, if you have a data quality check workflow containing a worker that creates exception child tasks, you can ensure recurring exceptions join a stack. You can then monitor and resolve multiple instances of the same issue together. 

Let's imagine you want to model the following data quality check workflow (created in this tutorial) in a task definition:

1. A parent task worker imports quotes from a new file in Drive.

2. A child task worker checks the quotes for outliers.

3. The worker creates an exception child task for each quote outlier.

When creating (or updating) your task definition, you can populate the mapStackingKeyFrom field in an action to automatically set a stacking key when a new exception task is created. 

For example, you could pass in the following as part of your task definition to set the stacking key value to a Client Internal Instrument ID the exception relates to. LUSID can then group all exceptions for a certain instrument together:

...
"actions": [
  {
    "name": "quote-outliers-check-worker",
    "actionDetails" : {
      "type": "RunWorker",
      "workerId": {
        "scope": "Finbourne-Examples",
        "code": "Check-Quotes-For-Iqr-Outliers"
      },
      ...
      "childTaskConfigurations": [
        {
          "taskDefinitionId": {
            "scope": "Finbourne-Examples",
            "code": "Quote-Outliers-Exception"  
          },
          "initialTrigger": "start",
          "childTaskFields": {
            "clientInternal": {  "mapFrom": "ClientInternal" },
            "lowerLimit": {  "mapFrom": "LowerLimit" },
            "upperLimit": {  "mapFrom": "UpperLimit" },
            "price": {  "mapFrom": "Price" },
            "priceDate": {  "mapFrom": "PriceDate" },
          },
          "mapStackingKeyFrom": "ClientInternal"
        }
      ]
    }
  },
  ...

You can tailor the stacking key mapping to best suit exceptions you might anticipate in your data quality check. You might map the stacking key value from a particular field to group exceptions by data vendor, or by the filename the quotes were imported from. You could then track and automatically close a stack when the lead exception from that data vendor or file is resolved.

Note you can currently use the mapStackingKeyFrom field in the following action types:

• RunWorker

• CreateChildTasks

When does a stack open?

When a new task is created to check for quote outliers and this generates an exception task for an outlier quote, the Workflow Service assesses the stacking key value (in this case, a Client Internal ID) to determine what to do next:

• If the stacking key value does not match an existing open stack, nothing happens for now.

• If the stacking key value matches the value of another task's stacking key (of the same task definition), a stack opens:

  • The existing task becomes the stack leader.

  • The newly-created task becomes a stack member.

How do I view my stacks?

You can view a task's stacking key in the LUSID web app by navigating from the top left menu to Workflow Service > Dashboard.

A stack icon indicates a task is part of a stack and whether the task is the stack leader or a member:

You can select the stacking key value for a task to view more stack information, including all other tasks within the stack. 

When does a stack close?

A stack remains open until the leader task reaches a terminal state. 

Currently, you can reference stacks in a guard condition to customise how your workflow handles tasks within a stack. For example:

• To prevent people from performing duplicate work on member tasks, you might specify a guard condition of stack.membershipType neq 'Member' or stack.membershipType not exists to prevent a task transition from occurring if the child task is a member within a stack.

• You might specify a guard condition of "childTasks all (state eq 'Resolved' OR stack.membershipType eq 'Member')" to prevent a task transition from occurring unless all child tasks are either resolved or accounted for in a stack as a member.

In the future, it will be possible for member tasks to be automatically resolved when the stack leader is resolved.

What happens if a stacking key value changes or is removed?

The following table provides information on stack behaviour in various scenarios: