A native Rust application
Archivist is written in Rust, including its desktop interface. Catalogue processing, verification and launch preparation live in Rust libraries that the application calls directly.
Rust gives us control over the costs that matter when a collection grows. We can borrow a record instead of cloning it, reuse a buffer instead of allocating another one, and share immutable data between workers. Ownership makes those choices explicit in the code.
That control is useful because collection work has several different bottlenecks. Reading a disk, calculating a digest and drawing a list each need a different approach. We keep those operations separate so a slow storage operation doesn’t require the interface to wait for it.
GPUI removes the UI IPC boundary
We moved away from Tauri to GPUI so the interface and application logic could run together in Rust. The old browser interface required interprocess communication, or IPC: data had to cross a boundary between the Rust backend and the webview.
The catalogue bytes already occupy memory buffers in the Rust process. With Tauri, those bytes have to be copied and sent across an IPC socket, then copied into buffers owned by the JavaScript process. When the catalogue changes, the changed data has to travel through that same path to update the browser’s copy.
With GPUI, the application and rendering code run in one Rust process. They can access catalogue memory through references in the same address space. Updating the interface no longer requires sending those catalogue bytes to another process and maintaining its separate copy.
Fewer catalogue copies with rkyv
A normal buffered file read uses two places in physical memory. The operating system brings file pages into its page cache, then copies the requested bytes into a buffer owned by the application. The application reads its copy through the virtual addresses of that buffer.
A read-only memory mapping gives the process virtual addresses backed by the file’s pages. When a page is resident, the CPU accesses the same physical RAM page held in the OS page cache. A page that isn’t resident must still be fetched from storage when needed. Mapping removes the extra copy into an application-owned input buffer.
rkyv handles a different part of the problem: the format of the bytes. Its archived structures can be accessed directly in their stored layout. That works in a suitable byte buffer as well as in a file mapping. Deserialising into new structs and strings would be an additional allocation step; direct archive access avoids that step.
Archivist’s mapped catalogue work combines these two properties. The operating system supplies file-backed pages, and the runtime accesses rkyv’s archived fields in those pages. It can use the catalogue without first copying the input bytes into a separate buffer or rebuilding the archive as owned objects.
Assembly and hardware-assisted hashing
Verification needs to establish what is actually in a file. Names and folder paths can help locate it, but matching against preservation catalogues depends on content digests.
Our hashing optimisation work reaches down to hand-tuned assembly and CPU intrinsics. We prioritise hardware-accelerated implementations where the processor supports them, with instruction selection matched to the digest being calculated. The required digest still comes from the catalogue: a faster, unrelated algorithm can’t replace the hash we need to compare.
There is also useful work to avoid before reaching the hashing routine. Archivist calculates the requested digests from the same stream of file data, feeding each buffer to the selected algorithms. It doesn’t need to read the file separately for every digest.
Faster hashing matters most when digest calculation is the bottleneck. When storage is supplying bytes more slowly than the processor can consume them, avoiding another read is the more useful improvement. Both cases shape the verification pipeline.
Hashing ahead of time on the NAS
Historian is an always-on daemon for storage appliances that we’re developing alongside Archivist. It moves collection scanning and hashing to the machine that holds the files.
The plan is to hash items as they arrive in storage and keep those observations up to date as the collection changes. When you open Archivist, it can use the results already collected by Historian to match files against the catalogue. The desktop won’t need to pull every file across the network and hash it before it can tell you what you have.
Listing a large directory tree over NAS can be expensive before any file contents have been read. Hashing then needs to read those contents. Historian maintains the file list and digests on the storage appliance, so the desktop can retrieve prepared results and perform the comparatively cheap digest lookup. Detecting changed files and refreshing their digests is part of the daemon’s job, so a changed file doesn’t keep an old identity.
A collection shouldn’t need to be read from scratch each time you want to play something. Historian is how we intend to make the work of checking a NAS collection happen as it grows, leaving the desktop to use the results.