The PokéCommunity Forums
The PokéCommunity Forums > Fan Games > ROM Hacking > Research & Development
Research PMD 2 music format (.smd) | Refined rips included below!

Notices
For all updates, view the main page.

Research & Development Got a well-founded knack with ROM hacking? Love reverse-engineering the Pokémon games? Or perhaps you love your assembly language. This is the spot for polling and gathering your ideas, and then implementing them! Share your hypothesis, get ideas from others, and collaborate to create!


Reply
 
Thread Tools
  #1    
Old November 29th, 2015 (6:54 AM). Edited November 30th, 2015 by ipatix.
ipatix's Avatar
ipatix ipatix is offline
Sound Expert
 
Join Date: May 2008
Location: Germany
Gender: Male
Nature: Relaxed
Posts: 144
SMD file format information
Introduction:
Hello dear PC members. So I've been busy the last few weeks reverse engineering the .smd format PMD 2 games use for their music sequences.
I just wanted to throw my research notes in here in case someone wants to do further research and or write tools to work with .smd files. I also have an IDB of my work. Interested in that? Send me a PM.
The notes include enough information to make proper rips from .smd to .mid files. I've included them for PMD EoS, see below for the download.

Research Notes:
Code:
/*
############################################
# Procyon Doc - written by ipatix          #
# my research information about .smd files #
############################################

Just some notes I took during research, the interesting stuff is below. 
You'll be most likely not looking for this up here.

00000000 ; ---------------------------------------------------------------------------
00000000
00000000 SONG_HEAD       struc ; (sizeof=0x40)
00000000 ident           DCD ?
00000004 field_4         DCD ?
00000008 field_8         DCW ?
0000000A field_A         DCW ?
0000000C field_C         DCW ?
0000000E field_E         DCW ?
00000010 ext             SONG_HEAD_EX ?
00000040 SONG_HEAD       ends
00000040
00000000 ; ---------------------------------------------------------------------------
00000000
00000000 SONG_HEAD_EX    struc ; (sizeof=0x30)
00000000 field_0         DCW ?
00000002 field_2         DCW ?
00000004 field_4         DCW ?
00000006 num_tracks      DCB ?                   ; needs confirmation
00000007 field_7         DCB ?
00000008 field_8         DCD ?
0000000C field_C         DCD ?
00000010 field_10        DCD ?
00000014 field_14        DCW ?
00000016 field_16        DCW ?
00000018 field_18        DCB ?
00000019 field_19        DCB ?
0000001A field_1A        DCB ?
0000001B field_1B        DCB ?
0000001C field_1C        DCD ?
00000020 field_20        DCB 16 dup(?)           ; string(C)
00000030 SONG_HEAD_EX    ends
00000030
00000000 ; ---------------------------------------------------------------------------
00000000
00000000 TRACK_HEAD      struc ; (sizeof=0x14)
00000000 ident           DCD ?
00000004 field_4         DCD ?
00000008 align           DCB ?                   ; aligned by the amount of bytes (usually 4)
00000009 field_9         DCB ?
0000000A field_A         DCW ?
0000000C length          DCD ?
00000010 field_10        DCB ?
00000011 field_11        DCB ?
00000012 field_12        DCB ?
00000013 field_13        DCB ?
00000014 TRACK_HEAD      ends
00000014

RAM:020B14C0 delay_lut       DCB   96,  72,  64,  48 ; 0
RAM:020B14C0                 DCB   36,  32,  24,  18 ; 4
RAM:020B14C0                 DCB   16,  12,   9,   8 ; 8
RAM:020B14C0                 DCB    6,   4,   3,   2 ; 0xC

###################################
# Interesting stuff starts here ! #
###################################

SMD Info:

In the beginning is a "FileHeader" which is followed by the actual SongHeader (size of each 0x40)
After both the individual Tracks follow. There is no index table. So one needs to go through each Track header to locate them all.
In the end there is an "eoc" block (don't know what it does).	
 */

struct FileHeader {             // offsets  | addítional information
    char ident[4] = "smdl";     // 0x0      | file type identifier
    char unknown_4[4];          // 0x4
    int fileSize;               // 0x8      | file size in bytes
    short magic = 0x415;        // 0xC      | must be that value, song shouldn't be recoginzed otherwise
    short unknown_E;            // 0xE
    char unknown_10[0x10];      // 0x10
    char songName[0x10];        // 0x20     | Null terminated (if not filled) string of the song name
    char unknown_30[0x10];      // 0x30
}; // size = 0x40

struct SongHeader {
    char ident[4] = "song";     // 0x0      | block identifier
    int unknown_4;              // 0x4
    short unkown_8;             // 0x8
    short unknown_A;            // 0xA
    short unknown_C;            // 0xC
    short unknown_E;            // 0xE
    short unknown_10;           // 0x10
    short unknown_12;           // 0x12
    short unknown_14;           // 0x14
    char numTracks;             // 0x16     | the amount of tracks the song has
    char unknown_17;            // 0x17
    int unknown_18;             // 0x18
    int unknown_1C;             // 0x1C
    int unknown_20;             // 0x20
    short unknown_24;           // 0x24
    short unknown_26;           // 0x26
    char unknown_28;            // 0x28
    char unknown_29;            // 0x29
    char unknown_2A;            // 0x2A
    char unknown_2B;            // 0x2B
    int unknown_2C;             // 0x2C
    char unknown[0x10];         // 0x30
}; // size = 0x40

struct Track {
    char ident[4] = "trk ";     // 0x0      | block identifier
    int unknown_4;              // 0x4
    char align;                 // 0x8      | Seems to leave room for at least n bytes (rounded up to 4 byte boundary) after the track header. This was 4 in all the cases I have seen.
    char unknown_9;             // 0x9
    short unknown_A;            // 0xA
    int trackLength;            // 0xC      | track length in bytes (including the mystic 'align' bytes)
    char spacing[4];            // 0x10     | As I've said, this is "usually" 4 bytes, see 'align' for details

    char trackData[];           // 0x14     | *!* This is where all the magic happens *!* See below for encoding details
}

/*
 * #####################
 * # Track Data Format #
 * #####################
 *
 * The Sequence Tracks work very similar to MIDI & co. Speed is given in BPM and "ticks" are 48 per quarter note.
 * There is controller-commands, delays, notes, etc. Each is initiated by one key byte. See table below for their meaning.
 * One command directly leads over to the next one if there is no delay spaced in between.
 *
 * When I refer to "cmd_byte" I refer to the byte that was initially used to decide which command to do. 
 * Also, some commands work on ranges of command-bytes. The *to* means including the last given byte.
 *
 * Terms:
 * byte             = 8 bits (0 to 255)
 * sbyte      = 8 bits (-128 to 127)
 * ushort   = 16 bits (0 to 65535)
 * short            = 16 bits (-32768 to 32767)
 * int              = 32 bits (-2147483648 to 2147483647)
 *
 *  - 0x0 to 0x7F:  Note:
 *                      The cmd_byte of the note command directly refers to it's velocity. The command byte has 1 + n argument bytes:
 *                      Arguments: (byte) "note_encoding", (n * bytes) note length in ticks (big endian)
 *                      "note_encoding" is a bit complicated and can be best displayed in it's individual bits:
 *                      8 bits: [XXYYZZZZ]
 *                        XX: range (unsigned 2 bits: 0 to 3):  This tells how many note length bytes (n) follow after note_encoding
 *                        YY: range (signed 2 bits: -2 to 1): This defines on which octave to play this note on. 
 *                                             It works relative to the last note read from the track data (therefore it also updates that "last octave" variable). There is also an "octave set" command. See below for details.
 *                        ZZZZ: range (unsigned 4 bits: 0 to 16, valid is 0 to 11):
 *                                             This contains the actual note to play on the resulting octave of bits YY.
 *                                             0 is equivalent to a 'C' and 11 to a 'B'.
 *                                             Values higher than 11 will probably work but obviously don't set the last octave to the higher one.
 *
 *                      As already said, after that the note 'n' note length bytes follow.
 *
 *  - 0x80 to 0x8F: Arguments: none
 *                  Effect:
 *                      Delay by (delay_lut[cmd_byte - 0x80]) ticks. See below for delay values. All delay related commands update the "last_delayed_ticks" variable.
 *  - 0x90:         Arguments: none
 *                  Effect:
 *                      Delay by 'n' ticks (last_delayed_ticks). 'n' is equivalent to the amount of ticks priorly delayed on this track.
 *  - 0x91:         Arguments: (1 signed byte) relative_delay
 *                  Effect:
 *                      0x91 behaves the same as 0x90 but also has a *signed* byte argument which adds/subs relative_delay bytes to the last_delayed_ticks
 *  - 0x92:         Arguments: (1 byte) absolute_delay
 *                  Effect:
 *                      Delays by absolute_delay ticks.
 *  - 0x93:         Arguments: (1 ushort, little endian) absolute_delay.
 *                  Effect:
 *                      Same as 0x92 but with extended range due to having 2 argument bytes
 *  - 0x94 to 0x97: ~currently unknown~
 *  - 0x98:         Arguments: none
 *                  Effect:
 *                      If no loop point was set: End of Track
 *                      If a loop point was set: Goto loop point
 *  - 0x99:         Arguments: none
 *                  Effect:
 *                      Set loop point to current location
 *  - 0x9A to 0x9B: ~currently unknown~
 *  - 0x9C:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0x9D:         Arguments: none
 *                  Effect: ~currently unknown~
 *  - 0x9E to 0x9F: ~currently unknown~
 *  - 0xA0:         Arguments: (1 byte) octave_set
 *                  Effect:
 *                      Sets the "last_octave" to the value of octave_set. This is usually initially called on each track.
 *  - 0xA1 to 0xA3: ~currently unknown~
 *  - 0xA4:         Arguments: (1 byte) tempo
 *                  Effect:
 *                      Sets the speed in BPM
 *  - 0xA5 to 0xA7: ~currently unknown~
 *  - 0xA8:         Arguments: (2 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xA9:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xAA:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *                      Note:   Command 0xA9 and 0xAA seem to set something common. Iirc 0xA9 sets the lower 8 bits and 0xAA the higher 8 bits
 *                              This command seems to be quite frequently use, however, I didn't manage to reverse engineer what they do nor I could guess what they do. 
 *                              When converting songs to MIDIs dropping this effect seems to not make things sound bad. So I literally have no idea what this does. 
 *  - 0xAB:         ~currently unknown~
 *  - 0xAC:         Arguments: (1 byte) program_number
 *                  Effect:
 *                      Sets the tracks instrument to program_number.
 *  - 0xAD to 0xB1: ~currently unknown~
 *  - 0xB2:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xB3:         ~currently unknown~
 *  - 0xB4:         Arguments: (2 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xB5:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xB6 to 0xBD: ~currently unknown~
 *  - 0xBE:         Arguments: (1 byte) mod_wheel
 *                  Effect: Set Track's Modulation to value of mod_wheel
 *  - 0xBF:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xC0:         Arguments: none
 *                  Effect: ~currently unknown~
 *  - 0xC1 to 0xCF: ~currently unknown~
 *  - 0xD0:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xD1:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xD2:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xD3:         ~currently unknown~
 *  - 0xD4:         Arguments: (3 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xD5:         ~currently unknown~
 *  - 0xD6:         Arguments: (2 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xD7:         Arguments: (short) pitch_bend
 *                  Effect: Set Track's pitch to pitch_bend
 *                          I'm not sure how the Track get's it's bend range. Most songs seem to default to an +-8 semi-tones range but I've seen some that didn't follow that rule.
 *  - 0xD8 to 0xDA: ~currently unknown~
 *  - 0xDB:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xDC:         Arguments: (5 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xDD to 0xDF: ~currently unknown~
 *  - 0xE0:         Arguments: (1 byte) vol, range 0-127
 *                  Effect: Sets Track's volume to vol
 *  - 0xE1:         ~currently unknown~
 *  - 0xE2:         Arguments: (3 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xE3:         Arguments: (1 byte) exp, range 0-127
 *                  Effect: Sets Track's expression (like a 2nd volume controller) to exp.
 *  - 0xE4 to 0xE7: ~currently unknown~
 *  - 0xE8:         Arguments: (1 byte) pan, range 0-127 : 0 = left, 64 = center, 127 = right
 *                  Effect: Sets Track's pan position (like a 2nd volume controller) to pan.
 *  - 0xE9:         ~currently unknown~
 *  - 0xEA:         Arguments: (3 bytes) unknown
 *                  Effect: ~currently unknown~
 *  - 0xEB to 0xF5: ~currently unknown~
 *  - 0xF6:         Arguments: (1 byte) unknown
 *                  Effect: ~currently unknown~
 *  - 0xF7 to 0xFF: ~currently unknown
 *
 * For every event where I don't even know how many / what arguments they take means that I simply have never seen them occur and therefore didn't look them up yet.
 * None the less I don't think that I've skipped any important ones.
 *
 *
 * Misc Information for above:
 *
 * delay_lut = { 96, 72, 64, 48, 36, 32, 24, 18, 16, 12, 9, 8, 6, 4, 3, 2 }
 */
PMD EoS Rips:
Here is a package of all the MIDIs I ripped from PMD EoS. I used a custom tool to do the raw conversion. However, a lot of hand work went into making these rips sound good (the originals don't follow the GM standard). I currently don't provide my tool for the conversion because from the information you should be able to make your own and mine currently works really really unreliable.
https://dl.dropboxusercontent.com/u/28573353/Music/PMD-EoS-MIDI.7z
Attached Files
File Type: zip PMD-EoS-MIDI.zip‎ (806.4 KB, 35 views) (Save to Dropbox)
__________________
Visit my YouTube channel (music hacking and other stuff): http://www.youtube.com/user/theipatix
Reply With Quote

Relevant Advertising!

  #2    
Old November 29th, 2015 (6:07 PM).
Pokemon_XY Pokemon_XY is offline
 
Join Date: Aug 2015
Posts: 58
Wow, nice work. Will you write a tool to convert .smd & .swd files to .mid & .dls files?
Reply With Quote
  #3    
Old November 29th, 2015 (6:26 PM).
Incineroar's Avatar
Incineroar Incineroar is online now
the spirit of the fighter
 
Join Date: May 2009
Location: Iki Town
Age: 21
Gender: Male
Nature: Brave
Posts: 13,088
One of the issues that he had run into while developing this was that some songs used more than 16 channels, so ripping MIDI and DLS files with this might not be feasible.
__________________
Reply With Quote
  #4    
Old November 29th, 2015 (6:26 PM). Edited December 5th, 2015 by ipatix.
ipatix's Avatar
ipatix ipatix is offline
Sound Expert
 
Join Date: May 2008
Location: Germany
Gender: Male
Nature: Relaxed
Posts: 144
Quote:
Originally Posted by Pokemon_XY View Post
Wow, nice work. Will you write a tool to convert .smd & .swd files to .mid & .dls files?
Well. I already do have one for smd to mid. I'll post it once I've fixed a few bugs that cause random crashes.
The swd's however, I don't knot a lot about them yet. I didn't look into them yet how they store the instrument maps. Also, the swd's don't seem to contain the actual instruments. I'd need to grab them from somewhere else. I might look into that further in the future but for now I'll work on other things.

Update
So in the end my researcges were more or less not so much breaking news. There is even tools out for all of that stuff. Check the "ppmd_audioutil":
-> https://github.com/PsyCommando/ppmdu
__________________
Visit my YouTube channel (music hacking and other stuff): http://www.youtube.com/user/theipatix
Reply With Quote
  #5    
Old December 5th, 2015 (3:42 PM).
throwaway2 throwaway2 is offline
 
Join Date: Dec 2015
Posts: 1
Perhaps adding to this wiki may help?

https://github.com/TruePikachu/pmDJ/wiki/SMD_Format

I feel you have some understanding to the missing components of PsyCommando's research, and was wondering if you could fill those in?

https://github.com/vgmtrans/vgmtrans/issues/38
Reply With Quote
  #6    
Old December 6th, 2015 (9:00 AM).
ipatix's Avatar
ipatix ipatix is offline
Sound Expert
 
Join Date: May 2008
Location: Germany
Gender: Male
Nature: Relaxed
Posts: 144
Quote:
Originally Posted by throwaway2 View Post
Perhaps adding to this wiki may help?

https://github.com/TruePikachu/pmDJ/wiki/SMD_Format

I feel you have some understanding to the missing components of PsyCommando's research, and was wondering if you could fill those in?

https://github.com/vgmtrans/vgmtrans/issues/38
I don't think I know that much more but I've added the "unknown" commands that I've looked up and fixed a few other things.
None the less I hope that I helped at least somewhat ;)
__________________
Visit my YouTube channel (music hacking and other stuff): http://www.youtube.com/user/theipatix
Reply With Quote
  #7    
Old December 7th, 2015 (5:13 AM).
psy_commando's Avatar
psy_commando psy_commando is offline
 
Join Date: Mar 2014
Location: Quebec, Canada
Gender: Male
Nature: Modest
Posts: 36
Hey! TeamFail told me about your thread !
Its great to see someone else working on PMD!

I was wondering if we could maybe exchange information and work together to reverse this format ? I did a good chunk of the research work, and I disassembled most of the DSE code too, but I really don't know much about audio honestly. And you seem to know more than me about that topic!

Also, if you want to see something more up to date for the meaning of the events and etc you should take a look at this:
http://projectpokemon.org/wiki/Digital_Sound_Elements
http://projectpokemon.org/wiki/Dse_swdl
http://projectpokemon.org/wiki/Dse_smdl

That github wiki that truepikachu setup is a bit outdated and he seems to have given up on it for now. The wiki articles I just linked were made a few weeks ago, and I'm updating them whenever I find out something new.

And here's my research + tools thread on Project Pokemon, there might be some info in there, as it pretty much turned into a general PMD2 research thread:
http://projectpokemon.org/forums/showthread.php?40199-Pokemon-Mystery-Dungeon-2-Psy_commando-s-Tools-and-research-notes

Hopefully this is useful.
Reply With Quote
Reply
Quick Reply

Sponsored Links
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 2:30 PM.