Fourthly... I don't usually like to do this, but I'd like to officially request anyone willing to help to really help me with figuring out the more tricky opcodes. It would be nice to have the debug version of NO$GBA or any other emulator with debugging capability, so I can more easily figure out the codes myself. What free emulators include this functionality? Does anyone have recommendations? It would make all this easier for sure. I'll do some searching on my own, too, so I can do better than just edit and restart emulation.
Yay! I love people that like to help! =)Hi ! I've been lurking for a while, and decided to try to help.
Actually a LOT of that could be useful. I already know where the sprites are stored, but I am unsure of the complete file format therein. Still, understanding them, and how the Pokémon are mapped to each file would be useful. And that's something found out easier with what you're doing =DI'd like to encourage as much as I can development of tools for PMD EoS, given so little information is available! So we can all have an easier time ! I just began a little side project for PMD Explorers of Sky, a runtime(for now) sprite/data injector/editor. And, if I can find anything that you might find useful I could probably put it in this thread. Though, my main focus are the sprites and avatars.
I've looked into DeSmuME, and it does have few good features, but I don't understand all of them that well. I think I've looked into Cheat Engine, too. If it's the one used to debug windows programs. =P The problem with using that is the games scripting engine is coded in ARM, and the program emulating the DS is running in machine code. So I'd be debugging the machine code that emulates the ARM processor, that runs ther games code, that has the script engine. It's three steps inside. Granted, once I learn how it works I can basically make my own debugger, but... well... it would just be easier if I could debug the NDS ARM code and find the script engine, then figure out what each command does from there.DeSmuME 0.9.10 has some basic features that could help out kinda like Dolphin, namely a disassembler, a memory viewer, and it can run lua scripts. Then, I highly suggest you'd get Cheat Engine. It features a great memory search feature and has data breakpoints, which are like the best thing ever ! And Cheat Engine runs lua scripts, with which you can automate tedious tasks and even write a little program/UI for doing things in cheat engine, or just running your scripts etc..
This is actually similar to how I'm looking for command length and structure + how the format of various files works. Things that I want to find out but don't know how they are store or where they are include 'dungeon data', 'battle data', 'object data', and a few others.Then, the best way I found to find the data you're looking for is usually to make a few hypothesis as to how the data is stored in memory. There are only so many ways to store data, so chances are you'll guess right ! Throughout coding languages and platforms, some common data structures are universally used. Like tables, lookup tables, linked lists etc..
Hee! Its alright. I'm mostly an all around programmer that only got into reversing because of PMD. And yes, I think you can definitely help! Even if I'm not focusing on sprites, I already know a lot about how they are stored. I was a sprite-ripper before this! In truth, I'm kind of honored!I hope it doesn't sounds patronizing though, its not my intention.
And of course, I'm not a great hacker or anything really. I'm mainly a C++ coder that got into reversing a short while ago XD
So there might be easier ways of doing all this ! But I hope this was of some use to you at least !
Alright then. I'll post what I find over there ! Though, at this point I don't have much. I can't find any documentation about Crystal Tile2's method of displaying pixels which would help a little figuring out the whole thing. It also seems that some of the sprites get misaligned in that program..
Though, I would have to say any help not tool-based might be better put in my other thread instead. I might even make it an overall info thread. Who knows? I basically work on what interests me. =P
Yay! I love people that like to help! =)
Actually a LOT of that could be useful. I already know where the sprites are stored, but I am unsure of the complete file format therein. Still, understanding them, and how the Pokémon are mapped to each file would be useful. And that's something found out easier with what you're doing =D
I've looked into DeSmuME, and it does have few good features, but I don't understand all of them that well. I think I've looked into Cheat Engine, too. If it's the one used to debug windows programs. =P The problem with using that is the games scripting engine is coded in ARM, and the program emulating the DS is running in machine code. So I'd be debugging the machine code that emulates the ARM processor, that runs ther games code, that has the script engine. It's three steps inside. Granted, once I learn how it works I can basically make my own debugger, but... well... it would just be easier if I could debug the NDS ARM code and find the script engine, then figure out what each command does from there.
Things are just never that easy, are they? =P
Honored ? What do you mean ?This is actually similar to how I'm looking for command length and structure + how the format of various files works. Things that I want to find out but don't know how they are store or where they are include 'dungeon data', 'battle data', 'object data', and a few others.
Hee! Its alright. I'm mostly an all around programmer that only got into reversing because of PMD. And yes, I think you can definitely help! Even if I'm not focusing on sprites, I already know a lot about how they are stored. I was a sprite-ripper before this! In truth, I'm kind of honored!
Yes, I mean that file. I already "unpacked it" using Tinke. the sir0 files in particular are what I'm actually decoding at the moment. I know a looooooooot about it now. animations, initial locations, you name it. everything for that sprite is stored in the SIR0 file, which is within the .bin file, and I can give you the format of both.Alright then. I'll post what I find over there ! Though, at this point I don't have much. I can't find any documentation about Crystal Tile2's method of displaying pixels which would help a little figuring out the whole thing. It also seems that some of the sprites get misaligned in that program..
Though, I still managed to find a few interesting tidbits by looking at the raw data.
I guess you mean that m_ground.bin file ? Its really an odd file. It seems to contain "SIR0" files/chunks.
I guess its some form of container, with a "table of content" at the beginning. But it has a strange and long header.. There should be some form of identification for the every subfiles in there.
What did you have trouble understanding ? Maybe I could help ?
I am indeed currently only working on editing the games script, but there are a few commands that I have no idea what do, and I figure if I understand the scripting engine then I'll understand what every command does. It's all to understand parameters that I don't know what they modify, like jumps and why they don't work for everything everywhere.About that, I thought you were only editing the game's script ?
Because, you can still use DeSmuMe's Disassembler to see the assembly code directly, if you really want to.
But, if you're looking only for the part that runs the scripts for the dialogue and game logic, then its easier to just look for the data used by the software to run those scripts. At least IMO.
I will look into doing it this way. It may be the only way, after all, I was just hoping for an easy solution. Oh well. That's what I get for being the pioneer for hacking PMD EoT/D/S xDBut, if you have a good IDE and grab the source code for DeSmuMe and compile it yourself, and run it in debug, that would be optimal, even though it would require some work. There is also a couple of tutorials on how to setup visual studio and a windows port of the GNU GBD debugger to debug the actual ARM code, thanks to some piece of code in DeSmuMe that allows to do that.
By honored, I mean I nerver really expected people were going to be helping me with any of this. I'm not a super-genius or anything, I'm just a programmer and even then, just a mediocre one that cares more about efficiency and short code than anything else.Honored ? What do you mean ?
Its kinda sad PMD isn't more popular.. Its such a charming game, and it would have been reversed years ago if it had been more popular.. Seeing as most other pokemon game got reversed really quickly !
Actually the whole reason I got into this, is to encourage people to add in the missing faces and sprites for the non official starter pokemon! But, if I can help others along the way, why not ! :)
I'll try to dump some of my notes so far in the thread you've pointed out. They're not 100% sure things, but I guess it can help to try figuring those things out as a group !
Yes, I mean that file. I already "unpacked it" using Tinke. the sir0 files in particular are what I'm actually decoding at the moment. I know a looooooooot about it now. animations, initial locations, you name it. everything for that sprite is stored in the SIR0 file, which is within the .bin file, and I can give you the format of both.
Come to think of it, you should read through both of my threads. I explain how the .bin files work as well as a bit of the sir0 for images in this post of the other thread. Though at that point I didn't know everything that I know now.
I was speaking about DeSmuMe. I don't know much about SIR0, given I found very little on the web for some reasons..
In terms of DeSmuMe, I'd have to look at them again. But in terms of the SIR0 format, there's one main section. It's near the beginning, with all those FFs and F9, FC, etc. it's really weird and i cannot for the life of me logically infer what it signifies. I do know however, that the length of the section is exactly 10 * the length of the pointers to the images section (at least in the file for female Pikachu, the 26th packed file), which means there may be a connection there. I'm unsure at this point. I'm looking deeper into it myself.
I am indeed currently only working on editing the games script, but there are a few commands that I have no idea what do, and I figure if I understand the scripting engine then I'll understand what every command does. It's all to understand parameters that I don't know what they modify, like jumps and why they don't work for everything everywhere.
I will look into doing it this way. It may be the only way, after all, I was just hoping for an easy solution. Oh well. That's what I get for being the pioneer for hacking PMD EoT/D/S xD
By honored, I mean I nerver really expected people were going to be helping me with any of this. I'm not a super-genius or anything, I'm just a programmer and even then, just a mediocre one that cares more about efficiency and short code than anything else.
Naturally, this means I'm really surprised and excited that i get the ability to work with someone that could very well be better than I am. I just started this, after all. =P
As for why I started? It alllll started because I wanted to find a way to allow people to actually battle...(spoiler'd because it gives plot points away). and the first commamnd i learned was 0x77 from Explorers of Time.Spoiler:Team Skull
Yeah, I saw those palettes, but they seemed incomplete for each pokemon oddly. They just had some of the most preeminent color on the pokemon. Or at least, from what I saw they did. While the palette extracted from red rescue team using VBALink had the entire correct lines for most pokemon. And the palette looked the same as the one DeSmuMe would read from the EoS game.(I'm not even sure why it won't let me build a palette from the current scene and export it, instead of just displaying it..)
PS: The palette for each pokémon is stored in the SIR0 file, just after the image data. and the scheme is indeed basically the same, but there are a few differences.
PPS: I'll prolly update my other thread with SIR0 stuff in the future.
PPPS: i might actually remember where i saw those portraits. I'll do some searching later and find out if I was right.
Not sure what Z-param you mean here.And you probably figured whether it is or not by now, but the Z param might just be the alpha channel.
They aren't compressed. Unpacking is like taking items from a box they were packed in, like when you move from an old house to a new one. Decompressing is something completely different in Computer Science terms. If you look at the format, the number of files parameter is at offset 4, and that is why the files is wrong. Change the num files offset to 4, then change the "start offset" to 8 (because again, that's in the pack format). That way you'll get all the files. However, make sure you select "Start offset/length" as the kind. if you get all SIR0 files, then you unpacked it properly and should hit "apply". =) I explain this indirectly in the post I linked to at the bottom.I didn't attempt unpacking the Bin files using tinke though. I didn't even think they could be compressed, given the file was mostly readable.. That could explain a lot of things XD
But, why does it outputs only 149 files ? I'm probably doing something wrong.. :/
I opened them as a "pack" in Tinke and extracted the content..
I sincerely doubt it. If it was padding there would be no reason to have an offset to it!And those might be padding bytes maybe ? However, I never really found much evidences that the files were byte aligned at all. Or any reasons for them to be, given they're on a flash catridge, not on a physical disc :/
I do realize this. I'm a Computer Science masters student. =)Oh, well, its just that at one point I wasn't sure what you were after, when you started mentioning OP codes and the ARM architecture. I thought you meant literally cpu op codes, but scripts are much higher level than that. I just wanted to clarify that.
All part of my plan. Honestly, there's not much else I can gain from guessing. I know almost everything I can (scriptwise) from guessing already. It's quite a lot, but it's not enough to really make a good hack. Given what I know, you can do almost anything cutscene related, except ruin other scripts. I don't know the details on doing that, sadly, even if I do know the commands used. That is why I need the debugger. So I can figure out why it just... doesn't work for certain things.But honestly, you'd have to have a good knowledge of ARM assembly, and look through pages and pages of opcode to really have a lot to gain from that. I was mostly only saying that, as far as debugger goes, this is probably one of the best way to do it.
In terms of programming knowledge I'm rather advanced. But this is my first venture into figuring out how the scripting works in anything =P, So in that respect I'm a newbie/intermediate.Oh, well, for one you're probably the only person I stumbled on still working on something for PMD:EoS ! That shows you've got motivation! And in the end, skills can be developed and improved, but motivation is another story..
I wouldn't hold my breath about me being much better ! XD Now that we discussed a little, I guess were pretty close in terms of knowledge. Though, you seem to be much farther ahead than I though in term of reversing the actual game files.
Incomplete? I have the complete palette in my sir0 files. 16 colors, the format of the palette is RRGGBB??*16. I open the file (currently) in YY-CHR and use that palette (which I manually remove the ?? bytes), adding the required number of bytes to make it a "full palette" of 256 colors, then load it into YY-CHR, letting me view the "official" colors for each pokémon.Yeah, I saw those palettes, but they seemed incomplete for each pokemon oddly. They just had some of the most preeminent color on the pokemon. Or at least, from what I saw they did. While the palette extracted from red rescue team using VBALink had the entire correct lines for most pokemon. And the palette looked the same as the one DeSmuMe would read from the EoS game.(I'm not even sure why it won't let me build a palette from the current scene and export it, instead of just displaying it..)
I do have the full specs already written down but there is a lot of extra stuff in there right now. If you look at my linked post (relinked here for convenience) I explain the SIR0 files in it on a basic level. =)And that sounds good! If you can get the specs written down, I might be able to write a little program to display them!
Btw, I was wondering, what language are you using to code this in ? Because, I could probably write that in the same language ? I'm more partial to C++ for working with bytes and such, but it might be easier for cooperation to use the same thing ?
Not sure what Z-param you mean here.
They aren't compressed. Unpacking is like taking items from a box they were packed in, like when you move from an old house to a new one. Decompressing is something completely different in Computer Science terms. If you look at the format, the number of files parameter is at offset 4, and that is why the files is wrong. Change the num files offset to 4, then change the "start offset" to 8 (because again, that's in the pack format). That way you'll get all the files. However, make sure you select "Start offset/length" as the kind. if you get all SIR0 files, then you unpacked it properly and should hit "apply". =) I explain this indirectly in the post I linked to at the bottom.
I sincerely doubt it. If it was padding there would be no reason to have an offset to it!
I do realize this. I'm a Computer Science masters student. =)
Yes, but the only issue here is that opcodes commonly refer to CPU opcodes, or sometimes their mnemonics. So its slightly hard for someone relatively familiar with them to get what you mean. Well in my case anyways.
I say op codes because the scripting is very similar to assembler where the first byte is the op code and the remaining ones are parameters. It's scripting only because it isn't ARM assembler. Granted, my terminology is a bit wonky, but eh. I'm using basically the same terminology Hackmew used, I believe.
Alright, sounds good ! I wasn't sure what you needed the debugger for, and tried to find you simpler to setup alternatives, but you seem to have everything worked out already.
All part of my plan. Honestly, there's not much else I can gain from guessing. I know almost everything I can (scriptwise) from guessing already. It's quite a lot, but it's not enough to really make a good hack. Given what I know, you can do almost anything cutscene related, except ruin other scripts. I don't know the details on doing that, sadly, even if I do know the commands used. That is why I need the debugger. So I can figure out why it just... doesn't work for certain things.
Well, same with me for reversing scripts XD
In terms of programming knowledge I'm rather advanced. But this is my first venture into figuring out how the scripting works in anything =P, So in that respect I'm a newbie/intermediate.
Well, I was mainly talking about those shades of color before the actual sprites. I never found the full palette..
Incomplete? I have the complete palette in my sir0 files. 16 colors, the format of the palette is RRGGBB??*16. I open the file (currently) in YY-CHR and use that palette (which I manually remove the ?? bytes), adding the required number of bytes to make it a "full palette" of 256 colors, then load it into YY-CHR, letting me view the "official" colors for each pokémon.
Right, but I meant the sprite animations, the headers, the palettes, etc.. In this post you linked, you only discuss the pixel format for the images.
I do have the full specs already written down but there is a lot of extra stuff in there right now. If you look at my linked post (relinked here for convenience) I explain the SIR0 files in it on a basic level. =)
I'm sorry if I sound difficult, but that, and the pixel format leaves me a lot to figure out again. And I say "again", because you mentioned having found out about those.SIR0:
Some type of data
picture
data for above picture
(repeat above ad nauseum)
some pointer data
I did a lot of C# already. Its a nice tool !
As for language, I use C#. I refuse to program in C++ if I don't have to, and I honestly hate using it. C# is so much cleaner and better in so many ways. If you don't know C#, I highly recommend it! You'll never go back to C++, I guarantee it =P
There should be 593 exactly. I'm really not sure where you got the extra 6 files. Maybe you mistyped?Well, I just assumed it was compressed, because I don't really see any advantage to grouping all those files into one, if its not for compression. Unless, they're just loading everything straight into memory..
Thanks for the offsets ! I got 599 of them now ! Honestly, I never really used Tinke all that much, and I'm still not very familiar with the formats on the DS.
As for the link, I can't seem to find those offset you just gave me. I know you've said indirectly, but I'm not sure what you mean in this case. :/
3,360 bytes in total (in the 26th file (holds female pikachu)), an offset to it and after it, pointed to by the same offset group. The offsets behind it are offsets pointing to location data, the offsets after it are pointers to animation data. I'm thinking its some sort of flags for how to change the sprites, or map sprites to directions or something, since in the file I checked the size is EXACTLY 10*the LENGTH of the pointers for the images. All the evidence points to it being used. I just don't know how. When I'm done I might run the game with that section zeroed out and see what happens.*EDIT*: Nevermind about the 0xFF thing I wrote. I thought you meant something else.. XD Given there are so many extremes like 0 and FF, maybe its some kind of opacity map ?
It's okay. In this case its the mnemonics. They made their own assembly language! =PYes, I was just explaining why I was asking this. Its hard to understand eachothers if we twist new meanings for words, so I was just trying to understand what you meant. I didn't mean to belittle you or anything. Sorry if it sounded that way.. :(
Yes, but the only issue here is that opcodes commonly refer to CPU opcodes, or sometimes their mnemonics. So its slightly hard for someone relatively familiar with them to get what you mean. Well in my case anyways.
Of course, I don't mean to tell you what to do. And you just cleared everything up anyways.
I'm not sure its a virtual machine... but I'm almost POSITIVE it was compiled by a compiler. All the evidence points towards it, and it isn't even a very GOOD compiler. Quite a bit of redundancy, but it explains a lot of things. Like all the u?XXXX.ssb files! everything has the same 'structure', and honestly would be pretty easy (comparatively) to reverse back to the source. (which is usually the case with scripting languages, and languages like Java and C# which ARE basically VMs)Maybe the dev used some kind of virtual machine to run their game scripts ? Or used some scripting language that was compiled into bytecode?
I do, but I'd have to tweak it to post on here. It's LONG because I am literally labeling all the different offsetts with what they do and its probably a bit confusing as is. but it's far more complete. It would be MUCH better to ... Actually, I'll use pastebin. https://pastebin.com/8btvKGDuWell, I was mainly talking about those shades of color before the actual sprites. I never found the full palette..
<snip>
Right, but I meant the sprite animations, the headers, the palettes, etc.. In this post you linked, you only discuss the pixel format for the images.
<snip>
...are you sure you don't have some notes somewhere you could just copy paste in here ?
Firstly... Ahh, language wars. This will be spoiler'd since its mostly off topic.I did a lot of C# already. Its a nice tool !
</snip>
With that said, I really don't mind using C#, if it would make collaboration easier ! :)