To-do: Difference between revisions

From No-Intro ~ Wiki
Jump to navigation Jump to search
No edit summary
mNo edit summary
Line 107: Line 107:
| Database
| Database
| Allow updaters to edit (but not delete/create/rename) Extra fields
| Allow updaters to edit (but not delete/create/rename) Extra fields
|  
| Done
|-
|-
| Long-term
| Long-term

Revision as of 16:28, 18 May 2022

Website

Various suggestions from unexpectedpanda

Time Period Category Item Status
Interface Page should be responsive, not fixed-width.
Interface Search bar on every page, that can search titles+hashes+serials for every system (like redump)
Interface Solicit feedback/help on the DoM sidebar?
Interface Keep users logged in for longer
Interface Login box at top of page Done
Interface Remember last page/system/search-settings Partially done - DoM remembers the last system for logged in users
Interface Line-up languages on archive edit page - they look wonky atm
Database Put more stuff in parameters in URL (e.g. for search) so that more stuff can be linked to
Database Each archive (at least) could have a permanent ID (GUID or similar), and pages should be accessible via URL with that ID, so that database entries can be referred to by third party projects (like Wikidata), without there being a risk of the ID changing.
Database Locked DATs should still be viewable.
Database It would be more efficient for the user if all forms (archive, source, file, attachment) were all on one edit page
Database "Extra" (user-specified) fields like the archive has would be useful for sources and files too
Database If an archive only has some files MIA, then the tag in Search and the top of the archive page should say "Partially Missing" or something like that, not just say "Missing".
Database DAT files could be cryptographically signed so people can tell if reuploads of the dat files are genuine. Probably best to include a text file with the filename, size in bytes and sha256 of each file in the download, and then sign that.
Database Dynamic cue generation/download for Non-Redump dats
Database Create new homepage design, including easy-to-use undumped list/database lookup tool WIP by Hiccup
Database A new, modern, database should be developed. There is something in development by a third party in private, that is making good progress. May be worth waiting to see how that gets on.
Database Exclude the header skipper tag in the dats by default Done
Database Add form to search by datter
Database Allow updaters to edit (but not delete/create/rename) Extra fields Done
Long-term Database Allow customised daily packs (same options as with individual packs)
Data Import/Export Create a YAML/YAML-like format for submissions in forums, as an alternative for the formats people end up "making up"
Data Import/Export Create a script that datters can use locally to convert the YAML format (or just variable human-readable text format) into the DoM import XML
Data Import/Export Make import XML support all DoM fields and use names consistent with those DoM fields
Data Import/Export Create schema for import XML
Data Import/Export Use same XML format for export (instead of the "plain text" format that doesn't have all the fields)
Data Import/Export Create API to allow specific data to be retrieved/added to DoM:
  • Allows emulators/dump tools to interact with DoM
  • Check ROM status
  • Submit data directly
Data Import/Export Change-request system. I.e. Datters could accept/reject/edit/add comments to submissions by non-datters (maybe even people without logins)
Data Import/Export Allow CSV import/export
New Database Fields All fields where people use commas to separate values should probably be split into multiple fields?
New Database Fields Scene releases should have a Region field, like "normal" sources. Done
New Database Fields Maybe add faster hash types. E.g. BLAKE3 (cryptographic hash, faster than SHA256 but less common) or xxhash (even faster but not a cryptographic hash).
Input Validation Some formats should have certain file extensions and vice versa. E.g.:
  • N64 - BigEndian = z64, ByteSwapped = v64.
Input Validation Serials generally follow a certain format
  • N64 - BigEndian = z64, ByteSwapped = v64.
Input Validation There is generally a fixed list of PCB serials for each system.
Input Validation Some sizes are invalid (or unlikely) for some systems

Wiki

Item Status
DAT Notes and the "Convention" pages overlap a bit - some stuff should be combined
Make video tutorials for DoM and put them on the wiki somewhere

DATs

General

Item Status
Check for ROMs that are in DoM but not in the Nintendo lotcheck sheets/DSi whitelist to find the "blind spots" of the lotcheck sheet/DSi whitelist (e.g. non-Nintendo-manufactured games / games post-2013ish and DS games that uses the newer encryption, respectively)
Work out the best way to do "modifications"/"alt formats". Currently dats use a mixture of having "trusted modification" entries (which have the issue of modified files not being able to be matched to the original files, except manually by reading comments) and just having alt formats (which are ambiguous as to who did the modification/when and its unclear if all sources should have the alt format or not. Examples of relevant dats: ique/wii/dsi/3ds/wiiu/switch digital, nes, n64, fds, ds, dsi, 3ds, switch.
Move stuff from undumped lists to DoM
Standardise the dump tool names - currently there is lots of variation
Catch up with dump submissions on the forum
Catch up with corrections on the forum and in the DoM tickets
Transcribe cart serials from photos, for sources where this is not already one
Look into this, it maybe be useful https://tcrf.net/User:Revenant/Konami_catalog_numbers

Batch-adding SHA56 to files

Item Status
Add SHA256 to NES Done for non-excluded files
Add SHA256 to N64 Done for most/all non-excluded BigEndian files

FDS

Item Status
The verification count system doesn't work with the way dumps are entered into this dat. Either the counting system needs to be adjusted or the way the data is entered needs to be.

DS/3DS

Item Status
Create GodMode9 script to automate dumping a bit

iOS

What is known (or sort-of known):

  • IPAs downloaded by iDevices are "thin" IPAs, meaning they only work on their processor architecture. While IPAs downloaded from iTunes are "fat" IPAs, meaning they contain code for all supported architectures.
  • "On-Demand Resources" (i.e. run-time downloads) will be difficult to preserve properly.
  • IPA files can be downloaded from itunes on PC/mac, including old versions. Some unlisted apps can be downloaded, as long as they were in your library, some unlisted apps won't let you download them, even if they are in your library, some unlisted apps will be gone completely from your library.
  • IPA files are zips, and the zips themselves contain metadata in the form of last-modified dates. Not clear how timezones play into this.
  • Before download, the server encrypts the code in the IPA with a key tied to the Apple ID and unique values
  • The apps are installed by extracting the IPA to the iDevice filesystem. The dates from the ZIP are possibly preserved, although its not clear how timezones play into this.
  • There are tools for jailbroken iDevices that can decrypt the code of a running app.
  • The code is signed by the developer and apple. The code contains hashes for at least some of the other files in the zip

Todo:

Item Status
See if timestamps are preserved when an IPA is installed.
Try out different jailbreak apps for decrypting IPAs
See if the code is still signed by the developer/apple after its decrypted
See what differs from a dump of the same app from two different Apple IDs/iTunes installations/iDevices.
See if a consistent file hash for the IPA can be achieved by dummying out unique values

iQue

Item Status
Download contents from wayback machine backups of CDN and dat as "redumps"
cmd files should be un-excluded Done
the missing cmds should be added, plus the z64 file to go alongside the cdn content as an alt format. dir2dat of said files:
	<machine name="cmds">
		<description>cmds</description>
		<rom name="003e970e.cmd" size="10668" crc="2b20466f" md5="d1f22e52d8a81a2817741375b988fbbc" sha1="5b291b74edadd7c0b84ea0406949539198e9b869" sha256="e9376b57e273c2c6d53e63b7076597401cceff85951aa6010b5695b3b12e9c00" />
		<rom name="0010d04e.cmd" size="10668" crc="4d4d73e2" md5="8d2c4cf5cdb44f1ef8e44aea1d854743" sha1="f9efa475c5fde298a91dc7a77af02f3aef8ae1db" sha256="8d41a1dc2b5250d40a59540bffb51707c3786d8931bf7d7b4dc8cc5915f22e15" />
		<rom name="00989681.cmd" size="10668" crc="2514555c" md5="845205dae754b0dfa27b157b81dd4741" sha1="e4fce03c5c43ffcad8db93b9269a69677ceb6567" sha256="b21f440179f50a73868787158741fcd57d4a41bc575c2682811c9472686e6934" />
	</machine>
	<machine name="z64s">
		<description>z64s</description>
		<rom name="003e970e.z64" size="163840" crc="620b14c8" md5="817a1a6b013b052b52f1ca5b1d765db8" sha1="349df91ab1dbe53e94a8c5318543cdaa9835a7fa" sha256="e07fdc0a9116c396fd88c44f5163f145e703f3fa7b834834350fb6bcb4de6564" />
		<rom name="0010d04e.z64" size="212992" crc="f882e342" md5="fac732031906241c7cc92e546629b62c" sha1="7d381b7587c64a933b1baf54700a7ff511f56b91" sha256="c3298ec21628b4a227a4793091d896bb702f9e70b42337920513162184f49fb2" />
		<rom name="00989681.z64" size="557056" crc="f82b2aa7" md5="381bf4c81bf3106a1451867f6723d1df" sha1="3afa53100e62ba7e4bb0a1b99272811bc53ec266" sha256="d0926a6c050badd7c40c6e01b081441b3c0b70b27127fd5d0afdda9d8090fd49" />
	</machine>

not sure which titles these go with though, who dumped the cmds and from where and who converted the cdn contents to z64. Jhynjhiruu knows about this.

Hiccup has asked Jhynjhiruu about this.
Should the cmd files from sources like these be removed from the sources https://datomatic.no-intro.org/index.php?page=show_record&s=109&n=0005 ? If they are just taken from the scene release, then they should be. They should probably be set as "Trusted Modification" too. Hiccup has asked Jhynjhiruu about this.

Wii/DSi Digital

Item Status
Add Galaxy's Wii dumps WIP (by xuom2)
Add Galaxy's DSi dumps Done
Add ZedSeven's DSi dumps Done
Add nold's Wii dumps
Wii: Import title version information and title key and title password hashes into database
DSi: Import title version information and title key and title password hashes into database Done
Create software to determine what is undumped and create database entries appropriately
Create software to backup non-title content endpoints (e.g. eshop metadata)
Once all URLs are gathered, send them to Archive Team as they want to make their own backup

3DS/Wii U Digital

Item Status
Add Galaxy's 3DS dumps Done
Add Galaxy's Wii U dumps WIP (by xuom2)
Import the anonymous Wii U dumps WIP (by xuom2)
Determine what is MIA and mark database entries appropriately
Create software to determine what is undumped and create database entries appropriately
Import title version information and title key and title password hashes into database
Create software to backup non-title content endpoints (e.g. eshop metadata)
Merge deprecated dat into current one
Once all URLs are gathered, send them to Archive Team as they want to make their own backup

Switch Digital

Item Status
Make the NSP and CDN formats equal - i.e. extract NCAs from all NSPs and dat them as CDN format, and pack all NCAs into NSPs and dat them as NSP format.
Investigate the fields in CNMTs that contain information on previous title versions.
Develop/find a method to retrieve title contents from the CDN directly, either through emulating the network process on PC or intercepting (and modifying if need be) switch network activity.
Create software to backup non-title content endpoints (e.g. eshop metadata)
Create software to extract NSPs for the CDN/NCA format dat.
Create software to automatically process new scene/p2p releases, using the aforementioned NSP processing tool

Xbox 360 Digital

V-Smile

Item Status
There are some additions to MAME that are not in the dat yet.