Using Luminesce to read/write local files (NAS)

You can request LocalFs.* providers from FINBOURNE and host them locally to interact with files on a network share (via a NAS or similar device).

Providing you have a suitable license and access control permissions, you can:

• Read from CSV, Excel, SQLite, XML, Parquet and plain text files stored locally, perhaps in preparation for loading investment data into LUSID.
• Write to files of the same types, perhaps having retrieved investment data from LUSID.
• Search folders for files.

Hosting the LocalFs.* providers

1. Make sure your computer has .NET 6 installed.
2. As a LUSID user with the lusid-administrator role, navigate to https://<your-domain>.lusid.com/honeycomb/swagger/index.html, where <your-domain> is the root of your LUSID domain, for example acmecorp. If you do not have the lusid-adminstrator role, see the appendix.
3. Use Swagger's Try it out button for the DownloadBinary API to request the LocalFileSystem_Providers binary:
   
4. Click the Download file link in the response to download the binary to your browser's download location: 
   
5. Navigate to the download location and run the following install command (omitting the .nupkg extension):

   C:\Users\JohnDoe\Download> dotnet tool install Finbourne.Luminesce.LocalFs -g --add-source "."

   The package is installed in the following location:
   • Windows: %USERPROFILE%\.dotnet\tools\.store\
   • Linux/MacOS: $HOME/.dotnet/tools/.store/
6. Run the following command to start the service (omitting the .exe extension on Linux or MacOS):

   C:\Users\JohnDoe\.dotnet\tools> luminesce-local-file-access.exe --quiet --authClientDomain=<your-domain> --certificateFallback=DownloadMayMint

   ...where <your-domain> is the root of your LUSID domain, for example --authClientDomain=acmecorp. Note the DownloadMayMint option mints a new certificate if you do not have one already; more information on managing certificates is coming soon.
    

   Note: For information on all the command line options, run the command luminesce-local-file-access.exe --help.

7. Open your LUSID web app, navigate to the Data Virtualisation > Query Editor window, and click the Refresh button (highlighted in red). You should be able to search for LocalFs providers:
   

To stop the service, enter Ctrl-C in the terminal window. The providers should disappear from your web app catalog.

To uninstall the package, perhaps in order to upgrade to a newer version, run the command dotnet tool uninstall Finbourne.Luminesce.LocalFs -g.

Renaming the LocalFs.* providers

You can optionally rename or remove the Mine suffix for all providers (you cannot change the LocalFs.* prefix). To do this, open the <install-location>\.dotnet\tools\.store\finbourne.luminesce.localfs\<version>\finbourne.luminesce.localfs\<version>\tools\net6.0\any\api-settings.json file and search for the following line:

"ProviderNameFormat": "{0}.Mine",

Change the text after {0} to something more intuitive or memorable, or remove it altogether, for example:

"ProviderNameFormat": "{0}",

The new names are reflected in the Luminesce catalog the next time you sign in. Note you must restart the service if you change this file while it is running.

Understanding access control permissions for the LocalFs.* providers

You and any other LUSID user wishing to run queries against a LocalFs.* provider must have suitable access control permissions. The general principle of creating policies for Luminesce providers, assigning them to roles and roles to users, is explained in this article. Note you should automatically have sufficient privileges if you are the LUSID domain owner with the built-in lusid-administrator role.

Getting help for the LocalFs.* providers

You can access the latest online help for LocalFs.* providers by running a query like this:

@x = use LocalFs.Csv.Mine
--help
enduse;

Using the LocalFs.* providers

The LocalFs.* providers are very similar to the Drive.* providers for interacting with files in Drive, except you must provide either an absolute location, or a location relative to that of the luminesce-local-file-access.exe file.

For reading data using the LocalFs.Csv.Mine, LocalFs.Excel.Mine, LocalFs.Sqlite.Mine, LocalFs.Xml.Mine, LocalFs.RawText.Mine and LocalFs.Parquet.Mine providers, specify a full path using the --file option, for example:

@x = use LocalFs.Csv.Mine
--file=C:\Users\JohnDoe\instruments.csv   # Either an absolute file path...
--file=..\..\instruments.csv              # ...or one relative to luminesce-local-file-access.exe
enduse;
select * from @x

For writing data using the LocalFs.SaveAs.Mine provider, specify a folder path using the --path option and the names of one file per input table using the --fileNames option, for example:

@data1 = select distinct TableName from Sys.Field order by 1;
@data2 = select distinct TableName, FieldName from Sys.Field order by 1,2;
@x = use LocalFs.SaveAs.Mine with @data1, @data2
--path=C:\Users\JohnDoe
--fileNames
file1
file2
enduse;
select * from @x

For finding files using the LocalFs.File.Mine provider, specify the RootPath parameter to nominate a base folder in the where clause of your query, for example:

select * from LocalFs.File.Mine where RootPath = 'C:\Users\JohnDoe' and Name like '%.csv' and IncludeSubdirectories = True

For examples of other operations, see the documentation for the similar Drive.* providers below:

To interact with...	Read from using the ... provider (links are to docs for similar Drive.* providers)	Write to using the ... provider (links are to docs for similar Drive.* providers)
CSV and similarly-structured files	LocalFs.Csv.Mine --> Drive.Csv	LocalFs.SaveAs.Mine --> Drive.SaveAs
Excel files	LocalFs.Excel.Mine --> Drive.Excel
SQlite files	LocalFs.Sqlite.Mine
XML files	LocalFs.Xml.Mine --> Drive.Xml
Text files	LocalFs.RawText.Mine --> Drive.RawText
Apache Parquet files	LocalFs.Parquet.Mine --> Drive.Parquet
Find files	LocalFs.File.Mine --> Drive.File	N/A

Appendix A: Allowing non-administrative users to downloading binaries

Once a license is granted, users with the lusid-administrator role have permission to download the LocalFileSystem_Providers binary out-of-the-box.

To enable less privileged users to download, create a feature policy consisting of the following JSON, assign it to a role, and the role to their LUSID user:

{
    "description": "User can download LocalFS binary",
    "applications": [
        "Honeycomb"
    ],
    "grant": "Allow",
    "selectors": [
        {
            "idSelectorDefinition": {
                "identifier": {
                    "code": "LocalFileSystem_Providers",
                    "scope": "Honeycomb"
                },
                "actions": [
                    {
                        "scope": "Honeycomb",
                        "activity": "DownloadBinary",
                        "entity": "Feature"
                    }
                ],
                "name": "Local file system binary",
                "description": "User can download LocalFS binary"
            }
        }
    ],
    "when": {
        "activate": "2023-01-01T00:00:00.0000000+00:00",
        "deactivate": "9999-12-31T23:59:59.9999999+00:00"
    }
}