How fast can Archivist load a large collection?

Most game-collection managers are conventional desktop applications.

Some are built with Electron, which combines a Node.js main process with Chromium renderers. Others are C# applications running as managed code on .NET. These runtimes provide mature interface libraries, broad platform support and a productive way to build a frontend.

A conventional game-collection manager usually reads its collection through ordinary filesystem interfaces and stores the result in a general-purpose database or in-memory object model. That works well for a collection on a local disk. As the library grows, every search, list and filter has more records to traverse, while box art, screenshots and videos add thousands of files to the media cache.

Degradation arrives gradually. Startup takes longer, scrolling becomes less consistent and searches lose their snap. A network filesystem adds latency to every cache miss and small-file access. Running an entire collection and its media cache from network-attached storage over a file share is aspirational, to put it lightly.

Archivist starts with native Rust.

Archivist's catalogue compiler builds the playable catalogue and verification index. Its Historian service inventories storage, hashes files and matches them to catalogue evidence. The snapshot encoder and Tauri desktop backend are also native Rust, giving Archivist direct control over allocation, memory layout, concurrency and filesystem access. The remaining gains come from deciding where the data lives and how each consumer reads it.

A player can move from a platform timeline to a genre family tree, inspect release evidence, browse cover art and launch a game. Delayed searches or stuttering lists make a large collection feel cumbersome, however quickly the underlying query completed.

Our target is 144 frames per second on every screen. Each rendered frame gets 6.94 milliseconds. Process creation may exceed that budget during startup, but every interaction after startup must fit within it.

Archivist keeps the selected catalogue native and resident, and it loads the storage snapshot before browsing begins. The renderer creates only the rows currently on screen while every catalogue record remains immediately addressable.

The Historian service runs beside the storage.

Archivist moves collection discovery from the desktop to the network-attached storage. The Historian service reads file metadata, hashes logical members inside archives and matches those digests to preservation manifests beside the storage pool. The desktop receives one compact, immutable snapshot instead of pulling every game image across the network share.

Reducing frontend loading from 4.9 seconds to 420 milliseconds.

The earliest measured Historian snapshot reproduced the JavaScript frontend's object graph in MessagePack, including a full path in every entry. That representation repeated strings throughout the graph and spent 3.0 seconds in JavaScript decoding. We replaced named maps with tuples, full paths with parent indices and row objects with aligned field columns. The Rust backend then compressed the complete snapshot with gzip. Together, those changes cut the interval from HTTP request to completed frontend application from 4,911 to 420 milliseconds.

The current request spent 51 milliseconds fetching the snapshot and 28 milliseconds crossing the Tauri boundary. Browser decoding and path reconstruction took 227 milliseconds, and the frontend finished applying the snapshot 420 milliseconds after the request began. Without restarting Archivist, that measured process reached the same applied state 1.167 seconds after process launch. A parallel Rust thread produced the backend's resident copy in 174 milliseconds without delaying the interface.

Compiling 2.15 million manifest rows.

Archivist’s catalogue compiler selects the strongest available file digest from each manifest row and builds the in-memory hash index used for verification. It extracts the required evidence, normalises each binary digest key and emits the resident structure used to match collection files.

Across the benchmark’s 1,221-manifest policy snapshot, the raw pass took 6.548 seconds. It parsed 909,779,869 bytes, produced 842,486 releases and 2,764,072 media rows, and rejected one malformed 303 MB TDC input. Bounded parallelism reduced the heavier static selection build from 383.289 to 196.547 seconds. Normal startup uses the versioned resident artefact, which memory-maps and hydrates in 0.557 seconds.

Choosing the snapshot format.

Archivist uses MessagePack for the snapshot's small, irregular header. The repetitive rows use a custom binary, padded structure-of-arrays (SoA) layout. Each field occupies a contiguous column, and the encoder pads each column boundary to its required alignment. A single zero-terminated UTF-8 blob holds the file basenames. JavaScript can create typed-array views over the numeric columns directly without allocating millions of row objects.

The format gives us control over where every bit lands. A pass reads only the columns it needs, and the next value is already beside the current one in memory. There are no per-row object headers or pointers to chase through the heap. That layout maximises cache locality and keeps the decoder in the same tight loop for every row.

The regular loop is also friendly to branch prediction. Tagged object fields, optional values and lexical token streams introduce data-dependent decisions. Fixed columns keep the hot path predictable. Our benchmark measures total decode time, so we don’t assign a separate number to branch prediction, but it is one reason we prefer the simpler layout when two formats compress to a similar size.

The live 4.70 MB payload is larger than the isolated 3.22 MiB codec fixture because it also carries the real roots, match projection, vault files, timestamps and status metadata.

Testing a filename dictionary

The path experiment included a dictionary for common components such as “Disk”. The top 128 tokens covered 69.4% of component occurrences. Parent indices already captured repeated directories, however, while leaf names were mostly unique. The second dictionary saved few bytes and added pointer chasing during reconstruction.

gzip already captures repeated byte strings through backreferences, and the embedded browser provides a native streaming DecompressionStream. Zstandard produced excellent sizes but would require a JavaScript or WebAssembly decoder in the current interface. gzip had the best measured request-to-applied result.

Hashing each file once.

Game-preservation catalogues commonly provide CRC32, MD5 and SHA-1 file digests: compact values used to identify exact file contents. The Historian service advances all three digest calculations during one sequential read, including for logical files inside ZIP containers.

This was a single run on our hardware. We excluded read rates because ZFS ARC and run order materially changed them. OxyROMon recorded the faster CRC32 result. Archivist led MD5, SHA-1 and writes on this machine.

Inventory took between 110 and 386 ms depending on cache state. Joining 12,297 computed digests against the resident manifest evidence took about 1.17 ms. Exact digest matching uses fixed-width binary keys in an FxHashMap. Ordered B-tree traversal offers no useful operation here.

Comparing different jobs.

Our friends behind OxyROMon and Igir have built mature, capable tools, and OxyROMon is the existing state of the art for this work. The shared-corpus timings measure different outputs. Archivist builds a deduplicated resident digest index, Igir writes a report, and OxyROMon imports normalised relational records.

On the shared 503-manifest corpus, Archivist’s purpose-built digest compiler is approximately 96× faster than Igir’s report generation and 152× faster than OxyROMon’s relational import. Their additional database and reporting work explains part of the difference.

What the current build uses.

• StorageNAS-side inventory, logical archive hashing, matching, filesystem notifications and guarded same-filesystem rename.
• DigestsCRC32, MD5 and SHA-1 advance during one read through reusable buffers. Assembly dispatch selects available CPU instructions, and fixed-width binary digests feed the FxHashMap joins.
• CatalogueStatic policy pruning reduces the benchmark’s 1,221 sources to 292 contributing sources, 313,914 retained releases and 1,239,634 unique strongest digests. Warm startup memory-maps the versioned 50 MiB artefact.
• StateA SQLite write-ahead log stores restartable file evidence. Hashing stays outside short write transactions, while an immutable snapshot cache and WebSocket generations serve the frontend.
• SnapshotMessagePack metadata, custom padded binary SoA columns, parent indices and one basename blob travel through native gzip. The browser reads typed-array views and performs one UTF-8 decode.
• MeasurementEnd-to-end timestamps cover process creation, HTTP transfer, Tauri interprocess communication, decoding and frontend application.

The TrueNAS virtual machine hides useful CPU instructions.

The physical Intel N100 processor exposes vector, carry-less multiplication and hash acceleration through instruction-set extensions including AVX2, PCLMULQDQ and SHA-NI. The TrueNAS virtual machine presents an x86-64-v2-AES virtual CPU that hides those extensions. Archivist's portable static binary already contains runtime-selected assembly implementations, but software inside a virtual machine can't select instructions that its virtual CPU doesn't advertise.

The one-run host-versus-guest comparison identifies a likely opportunity, but it doesn't support a production claim. The host and virtual machine are different execution environments, and each result comes from one run. The 3.8× SHA-1 difference is consistent with SHA-NI becoming available; the 52% combined-digest difference is more representative of Archivist's workload. The next test is to expose the host CPU to the TrueNAS virtual machine during a maintenance window and rerun the controlled benchmark.

The production NAS is currently bound by spinning disks and file scheduling. GPU hashing would also add a copy from host memory to the device. It becomes worth testing for large contiguous NVMe batches or when a GPU already consumes the same bytes.

Experiments worth running next.

The current measurements leave a short list of useful experiments. CPU passthrough has the clearest immediate case. Filesystem-native change detection and faster Linux I/O providers need separate HDD, SATA SSD and NVMe results before they earn a place in the production path.

• NextExpose the N100 host instruction set to the TrueNAS virtual machine and rerun hashing, Historian startup and power measurements.
• CandidateZFS snapshot-delta inventory, so a new scan can identify changed paths in one filesystem-native sweep.
• CandidateLinux statx identity reuse for hard links, fs-verity digests where the filesystem can prove them, and optional digest xattrs for portable rehash avoidance.
• Candidategetdents64/statx scanner and io_uring reader providers, with separate HDD, SATA SSD and NVMe benchmarks.
• CandidatePatch snapshots over WebSocket for small live changes. Startup should retain the single dense snapshot. A one-file mutation shouldn’t require another full projection.
• CandidateSame-pool reflinks and copy_file_range for moves that can’t use rename(2).
• RejectedWholesale JSON, CBOR, protobuf rows, FlatBuffer-style rows, lexical path tokens and a global filename dictionary. Each lost on total bytes, decode time or complexity.
• RejectedA B-tree digest index. Exact digest lookup favours the existing hash map, and this workload gains nothing from ordered traversal.
• DeferredPortable zero-copy shared memory into the WebView. Tauri doesn’t provide a cross-platform borrowed Rust buffer to JavaScript. Custom platform bridges would still leave JavaScript object materialisation.

What a native renderer would change.

The embedded-browser path now applies the live snapshot in 420 milliseconds. Its remaining costs include Tauri interprocess communication, JavaScript decoding and object materialisation, garbage collection, document styling and layout, and a second catalogue representation.

A native renderer would keep the compressed snapshot and presentation models in one Rust process. Storage and network costs would remain the same, but Archivist would gain direct control over allocation, layout, text shaping, GPU uploads, draw submission, input latency and frame pacing.

A native-renderer experiment should begin with one vertical slice of Archivist's largest timeline or verification tree, using the existing interface as its visual contract.

1. Freeze the visual contract.Extract colours, type scale, spacing, borders, shadows, animation curves and responsive rules into shared design tokens. Capture reference screenshots for every state and viewport.
2. Separate presentation models from DOM models.Move sorting, grouping, selection and row projection into Rust read models backed by the existing resident SoA catalogue. Keep full data resident and virtualise only visual nodes.
3. Build one native performance screen.Use the largest timeline or verification tree as the vertical slice. Run it beside the current screen behind a feature flag and compare frame time, memory, input latency and pixel output.
4. Render with winit + wgpu.Use persistent GPU buffers, instanced rectangles and images, a glyph atlas with cosmic-text/swash, explicit clipping and no per-frame heap allocation in steady state.
5. Retain and invalidate.Keep a scene graph and layout cache. Recompute only dirty subtrees, update only changed GPU ranges, and render only visible rows while every catalogue record remains immediately addressable.
6. Preserve desktop quality.Add AccessKit accessibility, IME and clipboard support, keyboard/gamepad focus, high-DPI scaling and platform window semantics before replacing the corresponding WebView screen.
7. Gate on measurable parity.Require perceptual screenshot parity, 144 Hz frame pacing on target hardware, bounded allocations and equal-or-better accessibility. Migrate screen by screen and keep the proven web implementation until each replacement wins.

egui is excellent, although its immediate-mode model differs from the retained renderer we want to test. Slint and Iced are credible candidates, with Slint preserving declarative layout. A custom wgpu layer offers the most direct control over Archivist’s data-heavy interface. The vertical slice must justify that additional work before we commit the whole product.

How the measurements were taken.

All results in this article are development measurements from the copied corpora and hardware described beside each table. Comparative runs used release binaries. The 2.15 million figure counts manifest media rows; the 222,242 figure counts files and directories in the Historian snapshot. Neither figure counts playable games.

The benchmark tools live with Archivist so future changes can use the same inventory, combined-hash, matching and catalogue workloads. We’ll update this page when a later implementation changes the result.