Building on Retool.

One Game, One ROM (1G1R) needed more than a region filter.

A preservation catalogue file, commonly called a DAT, can contain dozens of releases of the same game. Regions, languages, revisions, budget rereleases, compilations and prototypes all sit beside one another. Traditional 1G1R selection relies heavily on parent-and-clone links and region order. That can select a French release because Canada ranked first, or an older revision because it happened to appear earlier in the file.

Retool, created by unexpectedpanda, tackled the harder question. It ignored the parent-and-clone assignments inherited from the input catalogue, derived relationships from normalised title names, and supplemented them with human-maintained lists of equivalent releases. Its ordered rules covered region, language, version, release status and edition, right down to tiebreakers for obscure edge cases. Where those rules ran out, manual research took over—sometimes as far as booting suspected duplicates and comparing their title screens by hand.

It also understood that some decisions require research. A compilation may supersede several separate releases. A regional title may be the same game under another name. A cosmetic reskin may belong to the same family, while a release with new levels deserves its own identity. Retool recorded that knowledge instead of asking every user to rediscover it.

Retool is no longer maintained, but its idea and carefully assembled tables still form part of Archivist’s selection policy.

The main difference: Retool worked one catalogue at a time.

Retool reads one No-Intro or Redump catalogue, applies the user’s preferences, and writes a smaller catalogue for a collection manager. The input file defines the universe of possible titles. Equivalent-release detection, exclusions and selection all happen inside that boundary.

That boundary made Retool practical. It could understand the naming conventions of two preservation groups deeply, maintain equivalence lists for each catalogue, and leave file management to established tools such as RomVault, CLRMAMEPro and Igir.

Its documentation is candid about the catalogues it left alone. MAME and FinalBurn Neo would have required a large, frequently updated curated list. The Old School Emulation Center spans thousands of catalogue files and uses a naming system Retool didn’t implement. Retool focused on the catalogues it could curate well and left the others unsupported.

Archivist has a different unit of work: the historical system. A Sega Master System game may appear in No-Intro, a MAME software list, FinalBurn Neo, RetroAchievements and an older TOSEC catalogue. Those sources can disagree about names, categories, parentage and even which system owns the release. Archivist compares all accepted sources for the system before selecting its titles.

A preservation catalogue needs filtering before it becomes a game collection.

No-Intro, Redump, TOSEC and MAME document media and machines for preservation. Archivist uses those records as source material, but a raw import contains far more than its playable-game catalogue needs.

Import every available catalogue and you collect hundreds of thousands of rows that were never meant to represent distinct playable games: commercials, magazine cover media, manuals, system firmware, test cartridges, store kiosks, McDonald’s toys, educational software, multimedia discs and many varieties of demo and prototype.

Choosing only a folder or category called Games doesn’t solve it. Gambling machines, skill testers and vending machines can still slip through. A real game can be filed under Multimedia, Applications, Unofficial, a generic software list, or an unexpected system name. Categories help us find candidates, but we still need to check what each entry contains.

A commercial, a diagnostic cartridge and a soda dispenser controller may all be worth preserving. Archivist filters that material from its default collection so the catalogue presents playable games.

Our approach to a multi-source federated pipeline

Archivist begins with a reviewed system definition. That definition names the relevant catalogue sources, whether they use the common DAT format or another structured format such as XML. It also records aliases for catalogues whose filenames don’t line up neatly with the system’s canonical name. A source contributes to the playable collection only after it has been accepted into policy.

Each accepted manifest is parsed into the same release model. Archivist records its title and description, playable state, provider identifiers, parent relationships, media hashes and source path. The selection pass then runs across every accepted source for that system. When several catalogues describe the same game, their source paths accumulate under one selected title.

Some catalogues are useful as recognition evidence without being allowed to create titles. RetroAchievements, for example, can confirm that another source describes the same game while its unmatched rows wait for dedicated review. This lets Archivist retain useful evidence without silently expanding the playable collection.

Research catches the games that automatic rules would miss.

Once the sources have been joined, Archivist applies policy in layers. The broad rules are fast and repeatable. The narrow decisions record cases where filenames and provider metadata don’t tell the whole story.

A broad rule can reject entries marked (Unl), but some original-era commercial unlicensed games belong in the canon. A Games category can still contain a sewing-machine controller. A title with Collection in its name may be a redundant package or the only surviving release of unique material. Those cases become source-owned decisions with an explicit action and a stable canonical identity.

The 29,567-record minimum counts 26,085 reviewed system identities and 3,482 TDC title adjudications. It excludes other title-review ledgers where adding every row would count the same decision again after it was projected into policy.

The review process inspects every plausible source, every selected title, every discarded title and every parent/clone family. Archivist pins dated releases from each preservation catalogue so the review has a stable foundation. The catalogue for a system such as the Magnavox Odyssey is unlikely to change radically next week. Fixing the inputs lets us audit its sources and exceptions against a known body of evidence.

Freshness is the trade-off. TOSEC’s complete packs have recently arrived about once a year. No-Intro records can change several times in a day, Redump exposes catalogue files from its live database, and MAME publishes monthly. Many changes concern the furthest edges of the archive: a newly dumped prototype, a corrected category, or perhaps someone finally getting the Coca-Cola button working on a first-generation Burger King fountain controller.

Archivist won’t ingest each change as it lands. A new source can reopen decisions elsewhere in the joined system catalogue, so its snapshots will arrive less often. The preservation projects remain the live record. Archivist trades immediacy for a more thorough player-facing review.

MAME needs its own playable canon.

Retool never attempted MAME. Its author described the likely solution as a massive curated list that would demand frequent updates. That diagnosis was fair. Archivist is taking on the work because MAME preserves too much of video game history to sit outside the catalogue.

MAME already provides valuable canonical structure. Its parent and clone sets group closely related machine revisions and allow ROM data to be shared. Its software lists group releases, media parts and known variants for emulated systems. These relationships were designed around preservation, emulation and efficient storage. They don’t promise that the parent is the best player-facing release, and software-list parent links can’t cross from one list to another.

The remaining classification is leaky. Keeping runnable machines or rows labelled as games still admits gambling cabinets, redemption machines, skill testers, test fixtures, coffee and soda dispensers, and other fixed-function equipment. Filtering harder can remove unusual games, dedicated handhelds and playable electromechanical machines that belong in the history.

Our current MAME 0.288 audit starts with 50,097 machine records. The classifier proposes groups, but every uncertain boundary still requires review.

The research should save you work every time you add a game.

When No-Intro, TOSEC and MAME describe the same game, you should see one title in your collection, with the release that fits your preferences. You shouldn’t have to compare their filenames, investigate regional aliases or discover that the version you chose needs another disk before it will run. Archivist uses the reviewed catalogue to choose the release, verify its files and tell the launch system what it needs.

That is the part of Retool we want to carry forward: someone researches a difficult case once, and everyone using the software benefits. Archivist extends that work across preservation catalogues and into MAME. We’ll also publish reusable clone and equivalent-release relationships as a public dataset so other tools can use them.

Preservation communities have spent decades making these games survive. Our job is to make that work useful when you sit down to play: a collection you can browse, games you can verify, and the right release ready to launch.