---
title: "Understanding the Luminesce file system"
slug: "understanding-the-luminesce-file-system"
updated: 2026-10-27T10:04:14Z
published: 2026-10-27T10:04:14Z
canonical: "support.lusid.com/understanding-the-luminesce-file-system"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://support.lusid.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Understanding the Luminesce file system

The Luminesce file system is where Luminesce keeps the configuration you create: your custom views, your `Sql.Db` provider definitions, your LUSID entity provider configurations and more. Every change is recorded as a version, so you can always see what a definition looked like, who changed it and when. This article explains how the file system stores that configuration, how it stays consistent across the servers that run it, and what that means for you if you look after Luminesce in your organisation.

For an overview of where the file system sits alongside the query engine, the name service and providers, see [Understanding the Luminesce platform architecture](/v1/docs/understanding-the-luminesce-platform-architecture).

## What the file system stores

The file system holds **user-driven configuration**: definitions that tell Luminesce how to behave. It is not a place to store business data such as trade files or price CSVs. Use [Drive](/v1/docs/drive) for those.

The following features save their definitions to the file system:

| **Feature** | **What is stored** |
| --- | --- |
| `Sys.Admin.SetupView` | The SQL, parameters and metadata for each custom view you create. |
| `Sys.Admin.Lusid.Provider.Configure` | The configuration that makes your LUSID entity providers read and write properties. |
| `Sql.Db` [provider factories](/v1/docs/readwrite-to-sql-databases-sqldbmine) | The definitions of the custom providers you publish over your own SQL databases. |
| `Sys.Admin.File.SaveAs` | Any file you save to the file system directly. |
| Other provider factories | Definitions for some other provider families that create providers from configuration. |

Each file belongs to exactly one LUSID domain. Luminesce checks the domain in your access token on every read and write, so you only ever see and change files that belong to your own domain.

## Seeing what is stored

You can query the file system like any other data source.

- `Sys.File` returns the current version of every file in your domain.
- `Sys.File.History` returns every version of every file, including files that have since been deleted.

Both providers return the same columns, including the file's `Path`, `Content`, the `Delta` (the change from the previous version), the `PreviousAuthor` and `PreviousCommitMessage` of the change that produced it, and a `CommitOrder`, which acts as a version number for that file.

For example, to see the current definition of a view:

```sql
select Path, CommitOrder, PreviousAuthor, PreviousCommitMessage, Content
from Sys.File
where Name like '%MyView%'
```

To see how that view has changed over time:

```sql
select CommitOrder, PreviousAuthor, PreviousCommitMessage, Delta
from Sys.File.History
where Name like '%MyView%'
order by CommitOrder desc
```

Luminesce requires a comment for every change it saves. That comment appears as the `PreviousCommitMessage`, so `Sys.File.History` gives you an audit trail of every change.

![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/image-LQHJHSHS.png?sv=2026-02-06&spr=https&st=2026-10-29T10%3A28%3A37Z&se=2026-10-29T11%3A28%3A37Z&sr=c&sp=r&sig=KLJNuY3ETHsG5WdlLe6uZAjd0XgTMZ36nhXyXa4Jn1I%3D)

## How the file system works

### A git repository is the source of truth

Behind the scenes, the file system is a [git](https://git-scm.com/) repository with a single branch. Every save is a git commit, and every version you see in `Sys.File.History` is a commit in that repository. Git gives the file system a complete, tamper-evident history, and makes it straightforward to compare any two versions of a file.

The repository lives on a shared network disk. Luminesce runs several instances of the file system service, typically three, and they all read from and write to that one repository. Because it is a network disk rather than a local SSD, each disk operation takes noticeably longer. The file system is designed around that: it reads from the disk as little as possible and serves almost every request from memory.

![](https://cdn.document360.io/d575ad81-c0ed-4980-bbd1-d59ac5c3de82/Images/Documentation/luminesce-file-system-diagram.png?sv=2026-02-06&spr=https&st=2026-10-29T10%3A28%3A37Z&se=2026-10-29T11%3A28%3A37Z&sr=c&sp=r&sig=KLJNuY3ETHsG5WdlLe6uZAjd0XgTMZ36nhXyXa4Jn1I%3D)

Every instance serves reads from memory. Git on the shared disk is the single source of truth.

### Each instance keeps an in-memory cache

Each file system instance holds a cache of the whole repository in memory: every file, every version and every change. Reads come straight from this cache, so `Sys.File` queries, view definition lookups and all such system config reads never wait on the network disk.

The cache always records the commit it is up to date with. Git is the only source of truth, so the question an instance asks is never "is my cache right?" but "has git moved on since my cache's last commit?"

### Every request checks it is up to date

Before serving any request, read or write, an instance compares the latest commit in the git repository with the commit its cache is at. This is a quick check. If git has moved on, for example because another instance has just saved a change, the instance reads the new commits and applies them to its cache in order before it answers. Instances never serve a request from a cache that is behind git.

### Writes are locked across every machine

Only one instance can write to the repository at a time, across every machine. Luminesce uses a lock coordinated through its message broker (RabbitMQ), so it holds regardless of which server an instance runs on.

When you save a view, the instance that receives the request:

1. Acquires the cross-machine write lock.
2. Brings its cache up to date with the latest commit in git.
3. Writes your file to the repository and commits it, with your comment and your user ID.
4. Applies the new commit to its in-memory cache.
5. Releases the lock.
6. Notifies the rest of Luminesce, such as the query engine and any provider hosts, that the file has changed.

So the save is recorded in storage first, and the cache is then updated while the lock is still held. If anything fails part way through a save, the instance resets the repository to its last good commit and the save is rejected, so a partial change is never left behind.

### Consumers get notified and also re-check regularly

Components that act on these files, such as the query engine (which turns view definitions into providers you can query) and `Sql.Db` provider hosts, learn about changes in two ways:

- **Immediately**, from the change notification sent at the end of each save. A new or updated view is usually available within moments.
- **Regularly**, by re-reading the full set of files they care about from the file system at a fixed interval. In FINBOURNE's hosted environments this is every 30 to 45 seconds. This catches up any change whose notification was missed, for example during a network interruption or a restart.

### Starting an instance re-checks the cache against git

Building the whole cache from git history is slow on a network disk, so instances also keep a copy of the cache in a file next to the repository. Whenever an instance starts, for example during a deployment or when the hosting platform moves it to another server, it:

1. Commits any files found on disk that git is not yet tracking, so nothing stored outside a normal save is lost.
2. Loads the cache file into memory.
3. Brings that cache up to date with every commit made since the cache file was written.
4. Checks the cache against the current state of git, and records as deleted any file that the cache thinks exists but git does not contain.
5. If anything changed, writes the updated cache file back, while holding the cross-machine lock. The new file replaces the old one in a single step, so other instances never read a half-written cache file.

Steps 3 and 4 mean every instance re-checks its cache against git each time it starts, even when nothing has gone wrong.

## FAQs

### How is the cache updated when a view is created, updated or deleted?

Storage first, then the cache. The instance handling the save commits the change to git and then updates its own in-memory cache, all while holding the cross-machine write lock. The other instances do not update immediately. The next time any of them receives a request, it sees that git has moved on and applies the new commit before answering. See [Writes are locked across every machine](/docs/understanding-the-luminesce-file-system#writes-are-locked-across-every-machine).

### Is there a periodic check of the cache against storage to catch drift?

There are three checks:

- **On every request.** Each instance compares its cache with the latest commit in git before serving each request, and catches up if it is behind.
- **On every start-up.** Each instance loads its saved cache, catches it up from git and checks it against the current state of git, writing a corrected cache file back if anything changed. Instances restart routinely, for example during deployments.
- **In the components that use the files.** The query engine and provider hosts re-read the full set of definitions they use every 30 to 45 seconds, in addition to acting on change notifications.

### Luminesce runs on several servers. Does each have its own cache, and does that make drift more likely?

Each instance has its own in-memory cache, but the caches cannot drift apart from each other because none of them is the source of truth. Git is. Every instance starts from the same shared cache file, checks the git repository before every request, and catches up from git whenever it is behind. All writes go through the same repository under a single cross-machine lock, so there is only ever one history for the caches to follow.