Setting up and monitoring Horizon file transfers

When you subscribe to a Horizon integration which requires files to be transferred via SFTP (such as Horizon: JPMorgan IMOS), we provision your LUSID domain with a schedule which you simply need to enable on Day One to allow data files to be transferred to the relevant Horizon partner folder in LUSID Drive. LUSID creates a record of each transferred file so you can easily track the status of your data files as they're processed.

This article explains how to enable the schedule on Day One, view the file transfer records it creates, and monitor runs of the schedule.

• Setting up the Horizon file transfer schedule
• Viewing file transfer records
• Monitoring schedule runs and troubleshooting errors

Setting up the Horizon file transfer schedule

Note: This is a one-time setup. Once the schedule has been enabled for the first time, you do not need to repeat the steps for new integrations.

To set up the Horizon file transfer schedule:

1. Ensure you have already contacted your FINBOURNE sales representative and have the following:
   • Your LUSID domain provisioned with the relevant Horizon integration, such as Horizon: JPMorgan IMOS.
   • Opened an SFTP connection with your LUSID domain.
2. Enable the schedule. To do this:
   1. Sign in to the LUSID web app using the credentials of a LUSID administrator.
   2. From the top left menu, select Jobs & Scheduling > Schedules.
      
   3. In the Schedules page, select the Unpause icon on the Drive-SFTP-FileTransfer schedule.
      

Once the schedule is enabled, any new files received in LUSID are transferred from the sftp folder in Drive to a Deliveries folder for the relevant integration. For example, a new JPMorgan IMOS file with a filename matching the wildcard defined in its configuration set is copied to LUSID-Horizon/JPM-IMOS-V0608/Deliveries. 

Note the following:

• By default, the schedule runs hourly to copy the latest data files for your chosen integrations into the right location, ready for processing.
• You can manually run the schedule at any time to immediately transfer your files into the relevant Deliveries folder. To do this, locate the schedule in the LUSID web app under Jobs & Scheduling > Schedules and select Run now.

Each Horizon integration you subscribe to has one or more corresponding configuration sets, one per file type, containing the following information for your reference:

• integration: The Horizon data vendor the file transfer configuration relates to.
• source: The folder location in Drive the file transfer job looks within for new data files.
• destination: The folder location in Drive the file transfer job copies new data files to.
• wildcard: The matching pattern a new file name must meet for the file transfer job to move it to the relevant Deliveries folder.

You can view configuration sets in the LUSID web app by navigating to System Settings > Configuration Store and selecting the Shared Keys tab. All file transfer configuration sets are under the scope LUSID-Horizon-FTC.

Viewing file transfer records

When the schedule runs and finds a new file, a record is created within LUSID as the custom entity type LUSIDHorizonFileTransferCoordinatorRecord. You can view the transfer record for each file via the LUSID web app by navigating from the top left menu to Data management > Custom Entities:

Each record is populated with the following fields:

Field	Information
FileName	The name of the file from the vendor.
Integration	The name of the integration the data relates to.
ReceivedFromVendorAt	The datetime FINBOURNE received the file from the vendor.
DeliveredAt	The datetime the file arrived in the Deliveries folder in Drive.
DeliveryStatus	Indicates whether the file has been successfully copied to the relevant Deliveries folder using one of the following three values:  new is the default state; this indicates the file has arrived from the vendor but not yet been transferred. errored indicates the FTC failed to transfer the file to the Deliveries folder. delivered indicates the FTC has successfully transferred the file to the Deliveries folder.
DeliveryUrl	The location of the delivered file in Drive.
LastUpdated	The last time the record was updated; this typically indicates when a schedule last finished a step in its process.
IntegrationStatus	Indicates the file's processing progress once it reaches the relevant Deliveries folder, out of the following values:  pending is the default state; this indicates the file is in the Deliveries folder but the orchestrator has not begun processing it. processing indicates the file has begun to be processed. complete indicates the file has completed processing without error; its data is now ready for use within LUSID. errored indicates the file has completed processing with one or more errors.
ProcessingAttempts	The number of times the relevant integration orchestrator has attempted to process the file once it reaches the relevant Deliveries folder. The default maximum is 5 attempts.
ProcessedUrl	The location of the file in Drive once it has been processed by the relevant integration orchestrator.
Comment	A field you can use to specify any other details, for example if you have manually retried the file.

Note a LUSIDHorizonFileTransferCoordinatorRecord entity must exist for every file. If you manually upload a file to a Deliveries folder, a LUSIDHorizonFileTransferCoordinatorRecord is automatically created for the file.

Monitoring schedule runs and troubleshooting errors

You can monitor the status of file transfer schedule runs by signing into the LUSID web app and navigating from the top left menu to Jobs & Scheduling > Data Feed Monitoring.

The Rows processed column tells you the number of files the schedule attempted to transfer and how many completed successfully or failed.

You can click the Process name to view previous runs of the schedule, or click the View icon to see more details on the latest run in the Run Information screen:

If a run encounters an error, the schedule creates an output file in Drive that provides more details on errored rows. To view an output file, go to the Files tab in the Run Information screen and click on the hyperlink in the File Name column; this takes you to the file location in Drive:

File failing to be transferred

If a file fails to be transferred, check the filename matches the wildcard pattern defined in its related Horizon integration FTC configuration set.

Note the sftp folder in Drive is read-only - you cannot manually upload a file to it. You can manually upload a file to the relevant Deliveries folder in Drive; if doing so, you must ensure the filename matches the defined wildcard pattern and a LUSIDHorizonFileTransferCoordinatorRecord exists for the file.