To-do

Page to note down ideas. Not necessarily an official plan.

Website

Time Period	Category	Item	Status
	Interface	Page should be responsive, not fixed-width.
	Interface	Search bar on every page, that can search titles+hashes+serials for every system (like redump)
	Interface	Solicit feedback/help on the DoM sidebar?
	Interface	Keep users logged in for longer
	Interface	Login box at top of page	Done
	Interface	Remember last page/system/search-settings	Partially done - DoM remembers the last system for logged in users
	Interface	Line-up languages on archive edit page - they look wonky
	Interface	Add option to group items together in the News section like on mediawiki. E.g. if a single user does a bunch of edits on a single system, that would be displayed as a single entry like "100 edits by x in SNES" (you'd click on it to go to that system's news page to see the details)
	Interface	For Trusted Dump sources, highlight if the dumper is set to `!unknown`/marked as unconfirmed, or if dump tool is blank/set to `!unknown`
	Database	Put more stuff in parameters in URL (e.g. for search) so that more stuff can be linked to
	Database	Each archive and dat (at least) could have a GUID, and pages should be accessible via URL with that ID, so that database entries can be referred to by third party projects (like Wikidata), without there being a risk of the ID changing. Also, this would allow ROM managers and emulators to identify DATs easily.
	Database	Locked DATs should still be viewable.
	Database	It would be more efficient for the user if all forms (archive, source, file, attachment) were all on one edit page
	Database	"Extra" (user-specified) fields like the archive has would be useful for sources and files too
	Database	If an archive only has some files MIA, then the tag in Search and the top of the archive page should say "Partially Missing" or something like that, not just say "Missing".
	Database	DAT files could be cryptographically signed so people can tell if reuploads of the dat files are genuine. Probably best to include a text file with the filename, size in bytes and sha256 of each file in the download, and then sign that.
	Database	Dynamic cue generation/download for Non-Redump dats
	Database	Create new homepage design, including easy-to-use undumped list/database lookup tool	There is a new homepage design, but it could maybe do with some explanatory text based on the old one, although maybe on a secondary page. Hiccup has a draft of an improved version of the old text.
	Database	A new, modern, database should be developed, probably best doneas an open source project.
	Database	Exclude the header skipper tag in the dats by default	Done
	Database	Add form to search for a all edits done by a user
	Database	Allow updaters to edit (but not delete/create/rename) Extra fields	Done
	Database	Include more fields in the DATs, like additional tag, proto etc, license status, so people can filter stuff out client-side/using ROM managers
Long-term	Database	Allow customised daily packs (same options as with individual packs)
	Database	Maybe create ticket section for generic/whole-of-DoM tickets?
	Database	Add an option to see preview of datfile changes/result before saving archive/source/file changes
	Database	Add some way to find archives that don't have any sources with cart and/or pcb photos
	Database	Add option to mark dumper name as unconfirmed, like with dump date
	Database	Add option to mark attachments as missing/wanted (e.g. mark PCB photos as missing, if they haven't been provided). Probably should have deafult values for each dat. E.g. with SNES pcb would be flagged if missing, but not for DS.
	Data Import/Export	Create a YAML/YAML-like format for submissions in forums, as an alternative for the formats people end up "making up"
	Data Import/Export	Create a script that datters can use locally to convert the YAML format (or just variable human-readable text format) into the DoM import XML
	Data Import/Export	Make import XML support all DoM fields and use names consistent with those DoM fields
	Data Import/Export	Create schema for import XML	Done
	Data Import/Export	Use same XML format for export (instead of the "plain text" format that doesn't have all the fields)	Done?
	Data Import/Export	Create API to allow specific data to be retrieved/added to DoM: Allows emulators/dump tools to interact with DoM Check ROM status Submit data directly	Simple API to check ROM status via hash is likely
	Data Import/Export	Change-request system. I.e. Datters could accept/reject/edit/add comments to submissions by non-datters (maybe even people without logins)
	Data Import/Export	Allow CSV import/export
	New Database Fields	All fields where people use commas to separate values should probably be split into multiple fields?
	New Database Fields	Scene releases should have a Region field, like "normal" sources.	Done
	New Database Fields	Maybe add faster hash types. E.g. BLAKE3 (cryptographic hash, faster than SHA256 but less common) or xxhash (even faster but not a cryptographic hash).
	Input Validation	Some formats should have certain file extensions and vice versa. E.g.: N64 - BigEndian = z64, ByteSwapped = v64.
	Input Validation	Game serials generally follow a certain format.
	Input Validation	There is generally a fixed list of PCB serials for each system.
	Input Validation	Some sizes are invalid (or unlikely) for some systems
	DAT Generation	Add the DAT IDs to the daily pack, so ROMVault etc can identify dats between renames.	Done
	DAT Generation	For Nintendo CDN dats, include the digital serial fields' values (i.e. the Title IDs) from the source(s)	Done?
	DAT Generation	Add category tag like redump, but with support for multiple categories	Done
	DAT Generation	Add option to download daily pack with just numbered/just unnumbered dats.

Utilities

Item	Status
Create script to convert CSVs to the XML import format

Wiki

Item	Status
DAT Notes and the "Convention" pages overlap a bit - some stuff should be combined
Make video tutorials for DoM and put them on the wiki somewhere

DATs

General

Item	Status
Check for ROMs that are in DoM but not in the Nintendo lotcheck sheets/DSi whitelist to find the "blind spots" of the lotcheck sheet/DSi whitelist (e.g. non-Nintendo-manufactured games / games post-2013ish and DS games that uses the newer encryption, respectively)
Work out the best way to do "modifications"/"alt formats". Currently dats use a mixture of having "trusted modification" entries (which have the issue of modified files not being able to be matched to the original files, except manually by reading comments) and just having alt formats (which are ambiguous as to who did the modification/when and its unclear if all sources should have the alt format or not. Examples of relevant dats: ique/wii/dsi/3ds/wiiu/switch digital, nes, n64, fds, ds, dsi, 3ds, switch.
Move stuff from undumped lists to DoM
Standardise the dump tool names - currently there is lots of variation
Catch up with dump submissions on the forum
Catch up with corrections on the forum and in the DoM tickets
Transcribe cart serials from photos, for sources where this is not already one
Look into this, it maybe be useful https://tcrf.net/User:Revenant/Konami_catalog_numbers
For entries with PCB front photos, compare the sizes written on the ROM chip(s) with the ROM dump size
If the dumper says what dumping settings they used (e.g. what mapper the dumping tool is "dumping the cart as"), which they should do, then make sure that info is put into the comment2 of the DoM entry - go back and look at "Added to DoM" forum posts (e.g. ones using recent FlashGBX versions) and add it.
Might be worth putting torrentzip (RV7Zip?) hashes for every file - that way compressed data can be checked against dats without having to decompress or rely on crc32 in zip/7z headers (although maybe zip formats should just move to sha256 or something in their headers)
Create a tool that analyses overdumps to determine true ROM size and maybe require making overdumps and using the tool for all future dumps.
Make a list of dats where FilesOnly mode in RomVault should be used, i.e. dats that contains zips (and other archives) "as roms". At some point this could be integrated into DoM/the dats themselves.
Add bad dumps to the undumped lists so there's an easy overview of the high-priority items
Myria: "INL retro dumper gets different data "past the end" from beyblade versus gba_backup_tool. this is due to a phenomenon called "open bus" and electrical differences between the INL dumper and a nintendo DS." - this could potentially be simulated by specialised dumping hardware (or even software?) to determine when the real ROM ends.
Add categories to all archives using the new category field. Most can/should be automated somehow.
Revert changes where homebrew released worldwide on the web had their region changed from "World" to "Unknown" or "Unset" for no clear reason (note that latter tag has no documented purpose).
Add support for license and life span filters to the daily pack, and configure it so that homebrew and aftermarket ROMs are set to "off" on the individual and daily pack download pages by default
Make a script to compare GordonJ's ROMVault MIA dat (MammaMIA) with the DoM Daily Pack MIA values and run it every so often
Run MAME's romcmp tool to check ROM sets for common error patterns
Make a Neo Geo section	These checksums can be used as a base: https://github.com/mamedev/mame/blob/master/hash/neogeo.xml https://www.neo-geo.com/forums/index.php?threads/universe-bios-4-0-mvs-aes-released.258076/

Lotcheck sheet ROM size comparison

Do comparison, then update DoM entries with incorrect sizes as appropriate.

Item	Status
Nintendo - Nintendo Entertainment System
Nintendo - Game Boy
Nintendo - Super Nintendo Entertainment System
Nintendo - Super Nintendo Entertainment System - Royalty production (S-ROY)
Nintendo - Virtual Boy
Nintendo - Nintendo 64	Done (as of 2022-06)
Nintendo - Game Boy Color
Nintendo - Game Boy Advance	Done (as of 2022-06-06)
Nintendo - Pokemon Mini
Nintendo - Nintendo DS
Nintendo - Nintendo Wii
Nintendo - Nintendo DSi
Nintendo - Nintendo DSi (Digital) (CDN)
Nintendo - Nintendo 3DS

Batch-adding SHA256 to files

Item	Status
NES	Done for non-excluded files
N64	Done for most/all non-excluded BigEndian files

FDS

Item	Status
Add headered format so ROMs in that format can easily be matched with the dat.

DS/3DS

Item	Status
Create GodMode9 script to automate dumping a bit

DS

Item	Status
For revisions that are not confirmed to exist, it may be possible to de-confirm them by finding a cart that was manufactured after some of them by finding an unrevised cart that was manufactured after the revision's "ROM release date" on the lot check sheet. The manufacturing date could be determined by the date on the ROM chip (if there is such a date) or the date on the back of the PCB.
Find a way to dump the undumpable area at 0x1000. Its present in Wii U Virtual Console dumps and allows the header CRC16 to validate if present. Its difficult because DS carts use an ASIC (or something like that).
Compare the header/code hashes from the DS cart whitelists (from DSi and 3DS consoles) with the hashes generated from a fullset.
Fully document all the DSi whitelists and other built-in gamecart lists here: https://github.com/mariomadproductions/ds-built-in-cart-lists
DS firmwares: work out what determines the colour/behaviour when the gamecart is removed while PictoChat is open

iOS

What is known (or sort-of known):

IPAs downloaded by iDevices are "thin" IPAs, meaning they only work on their processor architecture. While IPAs downloaded from iTunes are "fat" IPAs, meaning they contain code for all supported architectures.
- https://help.apple.com/xcode/mac/current/#/devbbdc5ce4f
"On-Demand Resources" (i.e. run-time downloads) will be difficult to preserve properly.
IPA files can be downloaded from itunes on PC/mac, including old versions. Some unlisted apps can be downloaded, as long as they were in your library, some unlisted apps won't let you download them, even if they are in your library, some unlisted apps will be gone completely from your library.
IPA files are zips, and the zips themselves contain metadata in the form of last-modified dates. Not clear how timezones play into this.
Before download, the server encrypts the code in the IPA with a key tied to the Apple ID and unique values
The apps are installed by extracting the IPA to the iDevice filesystem. The dates from the ZIP are possibly preserved, although its not clear how timezones play into this.
There are tools for jailbroken iDevices that can decrypt the code of a running app.
- https://github.com/KJCracks/Clutch
The code is signed by the developer and apple. The code contains hashes for at least some of the other files in the zip

Todo:

Item	Status
See if timestamps are preserved when an IPA is installed.
Try out different jailbreak apps for decrypting IPAs
See if the code is still signed by the developer/apple after its decrypted
See what differs from a dump of the same app from two different Apple IDs/iTunes installations/iDevices.
See if a consistent file hash for the IPA can be achieved by dummying out unique values

iQue

Item	Status
Download contents from wayback machine backups of CDN and dat as "redumps"
cmd files should be un-excluded	Done
the missing cmds should be added, plus the z64 file to go alongside the cdn content as an alt format. dir2dat of said files: <machine name="cmds"> <description>cmds</description> <rom name="003e970e.cmd" size="10668" crc="2b20466f" md5="d1f22e52d8a81a2817741375b988fbbc" sha1="5b291b74edadd7c0b84ea0406949539198e9b869" sha256="e9376b57e273c2c6d53e63b7076597401cceff85951aa6010b5695b3b12e9c00" /> <rom name="0010d04e.cmd" size="10668" crc="4d4d73e2" md5="8d2c4cf5cdb44f1ef8e44aea1d854743" sha1="f9efa475c5fde298a91dc7a77af02f3aef8ae1db" sha256="8d41a1dc2b5250d40a59540bffb51707c3786d8931bf7d7b4dc8cc5915f22e15" /> <rom name="00989681.cmd" size="10668" crc="2514555c" md5="845205dae754b0dfa27b157b81dd4741" sha1="e4fce03c5c43ffcad8db93b9269a69677ceb6567" sha256="b21f440179f50a73868787158741fcd57d4a41bc575c2682811c9472686e6934" /> </machine> <machine name="z64s"> <description>z64s</description> <rom name="003e970e.z64" size="163840" crc="620b14c8" md5="817a1a6b013b052b52f1ca5b1d765db8" sha1="349df91ab1dbe53e94a8c5318543cdaa9835a7fa" sha256="e07fdc0a9116c396fd88c44f5163f145e703f3fa7b834834350fb6bcb4de6564" /> <rom name="0010d04e.z64" size="212992" crc="f882e342" md5="fac732031906241c7cc92e546629b62c" sha1="7d381b7587c64a933b1baf54700a7ff511f56b91" sha256="c3298ec21628b4a227a4793091d896bb702f9e70b42337920513162184f49fb2" /> <rom name="00989681.z64" size="557056" crc="f82b2aa7" md5="381bf4c81bf3106a1451867f6723d1df" sha1="3afa53100e62ba7e4bb0a1b99272811bc53ec266" sha256="d0926a6c050badd7c40c6e01b081441b3c0b70b27127fd5d0afdda9d8090fd49" /> </machine> not sure which titles these go with though, who dumped the cmds and from where and who converted the cdn contents to z64. Jhynjhiruu knows about this.	Hiccup has asked Jhynjhiruu about this.
Should the cmd files from sources like these be removed from the sources https://datomatic.no-intro.org/index.php?page=show_record&s=109&n=0005 ? If they are just taken from the scene release, then they should be. They should probably be set as "Trusted Modification" too.	Hiccup has asked Jhynjhiruu about this.

Wii/DSi/3DS/Wii U Digital

Item	Status
Make an easy too use tool for people who want to download contents from the Nintendo CDN in bulk, in a way that can be easily imported into datomatic and includes metadata.
Check that the "made-up" title version suffixes that cetks have been given match the title version field in the cetk itself.
Preserve the "met" file that each title has, on at least the Wii CDN, if not others.
Confirm which sources marked as dumped by Larsenv in the Wii Digital Deprecated and Wii Digital CDN dats were actually dumped by him - he's said that some/all weren't.

Wii/DSi Digital

Item	Status
Add Galaxy's Wii dumps	WIP (by xuom2)
Add Galaxy's DSi dumps	Done
Add ZedSeven's DSi dumps	Done
Add nold's Wii dumps
Wii: Import title version information and title key and title password hashes into database
DSi: Import title version information and title key and title password hashes into database	Done
Create software to determine what is undumped using CDN/eshop metadata and create wiki entries appropriately
Create software to backup non-title content endpoints (e.g. eshop metadata)
Once all URLs are gathered, send them to Archive Team as they want to make their own backup
Create "CDN" format equivalents for all titles in the deprecated dat, then merge the deprecated dat into the current one

3DS/Wii U Digital

Item	Status
Add Galaxy's 3DS dumps	Done
Add Galaxy's Wii U dumps	WIP (by xuom2)
Import the anonymous Wii U dumps	WIP (by xuom2)
Determine what is MIA and mark database entries appropriately
Create software to determine what is undumped using CDN/eshop metadata and create wiki entries appropriately	Look into xprism's tool for this
Import title version information and title key and title password hashes into database
Create software to backup non-title content endpoints (e.g. eshop metadata)	Look into this: https://github.com/mikage-emu/saveShop
Merge deprecated dat into current one
Once all URLs are gathered, send them to Archive Team as they want to make their own backup
Make tool to convert CIAs from P2P into CDN contents
Create "CDN" format equivalents for all titles in the deprecated dat, then merge the deprecated dat into the current one
Document the DLC metadata/shop endpoints and scrape them
Use the metadata that Hiccup got from a ROM collector's 3DS CDN set to improve the dat
Create a tool to backup SOAP endpoints
Create a tool to backup SpotPass/BOSS data

Switch Digital

Item	Status
Make the NSP and CDN formats equal - i.e. extract NCAs from all NSPs and dat them as CDN format, and pack all NCAs into NSPs and dat them as NSP format.
Investigate the fields in CNMTs that contain information on previous title versions.
Develop/find a method to retrieve title contents from the CDN directly, either through emulating the network process on PC or intercepting (and modifying if need be) switch network activity.
Create software to backup non-title content endpoints (e.g. eshop metadata)
Create software to extract NSPs for the CDN/NCA format dat.	Done by DarkMatterCore: https://github.com/DarkMatterCore/dom_xml_dataset_generators (xml_dataset_generator_nsp.py)
Create software to automatically process new scene/p2p releases, using the aforementioned NSP processing tool
Dat firmware/OS titles. A base to start off would be the dats (of p2p dumps?) by The1stOne and 8bitwonder.
Make sure these are datted correctly (e.g. extract the scene NSPs and make sure they match the trusted dump): https://datomatic.no-intro.org/index.php?page=show_record&s=103&n=00549 https://datomatic.no-intro.org/index.php?page=show_record&s=103&n=00550 https://datomatic.no-intro.org/index.php?page=show_record&s=103&n=01804 https://datomatic.no-intro.org/index.php?page=show_record&s=103&n=01804
Use the metadata that Hiccup got from a ROM collector's Switch NSP set to improve the dat
Create software to determine what is undumped using CDN/eshop metadata and create wiki entries appropriately

Switch

Item	Status
Transfer Card ID data from datted files, to hex data in the comment2 field https://github.com/mariomadproductions/nxdumptool_card_id_set_to_strings
Remove serials from file serial field - this data isn't actually present in the ROM in any document instance

Xbox 360 Digital

Post with useful info/tools: https://forum.no-intro.org/viewtopic.php?p=29383#p29383

V-Smile

Item	Status
There are some additions to MAME that are not in the dat yet.

Steam

Item	Status
Make a script to download the raw game data directly from the steam servers

NES

Item	Status
There is some more info on undumped Famicombox carts here https://forums.nesdev.org/viewtopic.php?t=23691&start=30 and here https://forums.nesdev.org/viewtopic.php?t=23691&start=45

GB/C/A

Item	Status
The lists of "ROMs that may be unique to the USA/PAL" release should be sorted based on the "ROM common" column in the master lists, since that column seems to say if the ROM is unique to that region.
Look at this https://www.reddit.com/r/emulation/comments/v2m2y6/edge_of_emulation_advance_movie_adapter/ https://shonumi.github.io/articles/art28.html

GOG

Item	Status
Import Connie's GOG dats
Import Gefflon's GOG dats
Find a way to make it easy for Gefflon to add new GOG stuff to DoM directly (e.g. make it so certain items can be marked as "current"/"latest", so a custom dat can be generated with only those items, like Gefflon's current dats)
Look into GOG API

Atari - 2600

Item	Status
Incorporate data from https://github.com/stella-emu/stella/blob/master/src/emucore/stella.pro (with credit), if there's anything new/useful there.

Nintendo - Misc

Item	Status
For the large disk images, add hashes for deterministically compressed versions (e.g. rv7z) and exclude the non-compressed one. It'd make the files easier to handle (less disk space needed, no need for people to compress/decompress themselves).

N64

Item	Status
Check for (mainly proto) ROMs that are corrupted due to being dumped using old dumping hardware like gamesharks and old copiers: https://pastebin.com/UpzXVu3m

Vita

Item	Status
For bad dumps with comment1 reason set to "[grw0 partition not in factory state]" (and not any other reason in addition), change the status from the default bad status to "HasSave" bad status

]

To-do

Contents

Meta

Website

Utilities

Wiki

DATs

General

Lotcheck sheet ROM size comparison

Batch-adding SHA256 to files

FDS

DS/3DS

DS

iOS

iQue

Wii/DSi/3DS/Wii U Digital

Wii/DSi Digital

3DS/Wii U Digital

Switch Digital

Switch

Xbox 360 Digital

V-Smile

Steam

NES

GB/C/A

GOG

Atari - 2600

Nintendo - Misc

N64

Vita

Navigation menu