LUSID’s Workflow Service equips you with configuration options that can drive endless combinations of unique workflows. This page lists our best practice recommendations to help you create and monitor efficient, robust, and manageable workflows.
Constructing task definitions and workers
Before you can get started with workflows, you must be familiar with the building blocks. Read more.
The simplest way to create manageable workflows is to construct a comprehensive plan. To construct a task definition, you could consider:
What kicks off an instance of my workflow?
What data does my workflow take in? Where will the data come from?
What is the end state of the workflow? Will there be more than one possible end state?
What functionality does my workflow need?
Will the workflow need to break off into child tasks?
How do I want to handle exceptions?
For example, this tutorial for creating a data quality control begins by mapping out everything that must be encompassed in the task definition in a flow diagram.
Lowering task quantities
There are currently four main ways to kick off a task:
Using the Workflow REST API
Using system event handlers
Using the Workflow.Task.Writer Luminesce provider
Manually via the LUSID web app
If you want to automate task creation with one of these methods, try to reduce the number of tasks the workflow creates (within reason).
For example when using system event handlers, consider the number of events that occur for your chosen event type. Let’s imagine you choose to kick off a task every time the OrderCreated event occurs. If you place 10,000 trades a day you could accumulate 10,000 separate tasks a day:
To manage the same process more efficiently, you might instead kick off a workflow task that takes in a file containing many orders. Instead of creating thousands of tasks a day, one task ingests all of the orders and kicks off one worker to monitor. The task might then create child tasks for any exceptions which you can manage using:
Guards to prevent the closure of the day’s parent task until all child tasks are resolved.
Stacks to monitor and resolve all child tasks with the same issue simultaneously.
Running actions as a service user
We recommend setting up workflows to run on behalf of a service user to prevent any need to migrate the task in the event a personal user account is deleted.
You can specify a runAsUserId value in your task definition to use a service user to perform an action. Read more.
Worker timeouts
Workers are powered by Luminesce custom views. Luminesce custom views can use PRAGMAs to override default Luminesce settings.
By default, worker Luminesce queries timeout after running for 30 minutes. You can define a timeout in your worker and the Workflow Service will automatically pass it into the timeout PRAGMA in the Luminesce custom view.
If a timeout occurs, the Workflow Service notes the failure and performs up to three retries. You can view worker failures and retries using action logs.
Monitoring current tasks
To monitor your current tasks in a simple view, use the Workflow Service > Dashboard in the LUSID web app.
You can also use the dedicated dashboards for Tasks, Task Definitions, and Workers.
You can call the GetTaskHistory API to retrieve versioning information for every task update, such as:
The asAt timestamp for the particular task update.
The modifier’s user ID.
Details of the update, for example state changes.
Read more about retrieving version information and tracking changes in LUSID.
Troubleshooting tasks
When workflows fail, it’s often when performing actions after a successful state transition. Use the GetActionLogs API to retrieve diagnostic information for the last submitted action by the task. See how.
Viewing historic tasks
To view up to 15,000 historic tasks, you can use the Workflow Service > Tasks dashboard in the LUSID web app.
To retrieve more than 15,000 tasks, use the Workflow API or Workflow.* Luminesce providers.
Deleting tasks
Workflow Service > Dashboard in the LUSID web app filters tasks it displays by day. This means deleting tasks does not affect performance or monitoring; you can store as many tasks as you wish.
If you do wish to delete many tasks for any reason, you can do so asynchronously via the DeleteTasks API.