Research: Expanding the Storage System

[M.Hussain]

Didn't expect me did you?
Anyways, hey everybody! As you read by heading of this topic the main issue that's going to be discussed is how to actually expand number of pc boxes.

How the need for this issue ultimately began:

Spoiler:

For starters the it began a long a time ago about when people started releasing methods to increase the number of Pokemon in their rom (i.e. 420 plus) but the problem emerged as if one where to implement this where would we keep all the Pokemon in the first place( so people ignored the issue and or delayed it for some other time)? And we expanded the Pokemon data so much that we can literally future proof later generations but the problem still remains, how do we store them?

How the issue came to me:

Spoiler:

Well the issue was pretty much in my mind but grew kind of larger as I started helping on a rom base with more than 1000 Pokemon entries. So the problem was mentioned there as well and on many other rom hacks and that is that is the dev gonna add more storage boxes but the question is ignored because there is no way to do that yet.... let's change that shall we?

Why didn't I post this on a quick resource ask thread:

Spoiler:

Well because the issue was too large to let be ignored as desperate cry of help that may or may not get picked up... plus I did some research on the issue so there's that.

What I have on the matter at hand:

Spoiler:

A lot of things really, for starters I have a a lot of offsets that I can't seem to find where it all is linked to, so here's a complete info of the data I found(i.e don't try to create connections between all of them yet.
1.

These might not be helpful enough but they could be used to track down which data is using these i.e track the pc boxes data down.

2.https://bulbapedia.bulbagarden.net/wiki/Save_data_structure_in_Generation_III#Sections_5-13_-_PC_buffer
this is something I need data to be looked at because this conatins parts on how game saves pc data on save file, if we can figure out from where it get's or send it's data from we might be able to find the pc box structure.

3. 83 --> 1a6955( this is offset for the pc script, again we need find out where the offset is linked from.

4.While I was looking around any data I could find I found out that the pc not actually accessed by a script per say but a script that embeded into a tile

and you can place the tile any where and as along as you can press a infront of it pc script will activate( if we can find out what is embedded within the tile we might again figure out the location. of the pc box data.

What I believe the structure of the pc storage box is:

Spoiler:

I feel as is if it is a huge table occupying a good amount of the rom that has multiple sections within the huge section. The first part of the pc box structure would contain configurational settings(i.e what background it has or what box number it is(0-13) the second part just would be huge and contain mainly coordinate data( how it is divided and between a grid and how data of 30 Pokemon is to be stored( I believe that no data is saved on the table yet it's just an outline mainly, them main data is saved in the save file)) The third and the last part would mainly be the table end part and links to a another pc storage table, ultimately the entire structure is just 14 repeating tables each containing the above 3 parts.

What would happen if the above is true:

Spoiler:

Well if the structure of the pc storage I believe is true then the table could be analyzed and part of the start and end of each table figured out, we can copy the table and place it in free space and link the existent 14 tables with the a new one and hopefully have another table(i.e storage box) and the process could be repeated several times until we have enough boxes for all the Pokemon.

What's stopping the above from happening:

Spoiler:

The location of the pc storage structure is still not found yet.

Ending notes:

Spoiler:

Again this is all hypothesis and research and the hypothesis comes from what I know about the save structure. I feel as we are really close to figuring out how to expand just we are missing the link. This is why I am posting my findings and hypothesis here. Let's make this thing happen shall we.

extra:

Spoiler:

...

 

[pokefreak890]

I wonder why nobody commented on this yet so far this is a good find we will use anything in order to get expand PC storage and yeah I know what u guys are saying "But it will waste a lot of RAM" now the question is that is it worth using that RAM or not?

And also the base he's talking about is mine glad I got u more into PC storage research :)

 

[azurile13]

I'm a little unsure of what exactly you're searching for. You mention that you're looking for the pc structure...but that already exists? Its in the bulbapedia link you gave. That's it - that's the structure. Almost every routine related to the pc is going to have a pointer to the boxes saveblock (0x03005010 iirc) or will call/be called by a routine that does. There are a few exceptions (like when you're dragging a pokemon around with that hand), but its a really good place to start. Just start with routines with pointers to the saveblock and you'll find that most of them are pretty simple. Read wallpaper, write name, etc. Even the pokemon ones are typically something like r0= box number, r1 = which slot in the box. For the majority of routines, all you need to change is a box comparison. Even the transition between box 1 and box 14 is just a single comparison iirc.

I think the reason box expansion hasn't been done is that it takes quite a bit of space. But if you ask me, the better way to go about it...instead of looking for more space, we should be using space more efficiently. We really don't need to be devoting 80 bytes to each Pokemon, especially in a ROM hack.

 

[Deokishisu]

This thread is pretty much useless as-is. This has been discussed to death and the OP doesn't bring anything new (or even particularly relevant) to the PC expansion idea.

It would be different if he proposed something actually insightful like, "Could we compress the Pokemon data by leaving out padding, checksum stuff, language, and game origin and regenerate that info upon withdrawl?" But that's only 9-bytes per Pokemon saved, which is enough for 3 extra Pokemon per box, or an extra 53 Pokemon stored across all boxes. That only brings the total capacity up from 420 to 473.

You could save more space if you eliminated the ribbon field and the "EVs" for Contest Stats as well, which gives you an extra 10-bytes per Pokemon saved. Combined with the above 9-bytes per slot saved, that's enough for an extra 9 Pokemon per box, or an extra 130 Pokemon stored in the same amount of space. That only brings the total capacity up from 420 to 550. But this comes at a cost for people who want to use the contest feature if this is implemented in Emerald.

If you really wanted to get radical with compressing space, you could make it so that there's a separate index table of all holdable items in an attempt to save another byte (the idea being that you'd only need one byte to store all usable/possible hold items, you would exclude all non-holdable or non-usable as a hold item items from this table. So things like Potions would be excluded but Berries would not.) by having a routine convert an item's "traditional" index number to the new "hold item" index number and back when giving or taking an item from a Pokemon. You could save a level-byte and only "save" the experience that the Pokemon has gained in its current level to possibly free up a byte, instead of recording total experience gained and calculating a Pokemon's level from that. (Pokemon in the Fluctuating Experience group will need more that 65,535 experience points to go from 99 to 100. I see no problem with messing with the lookup table and lowering the experience needed for just this one level from 68,116 to do this.)

If you could save just one byte from rewriting how experience is done, then add the byte you save from rewriting held items, that's an extra two bytes that are gone. Combined with everything above, that gives you a 59-byte data structure for Pokemon in the PC, and 149 extra Pokemon total in the same amount of space for a grand total of 569 in the PC. There's also an unknown two bytes in the Growth substructure. If that's actually unused, then that's a 57-byte data structure, and 169 extra Pokemon stored with a grand total of 589 in the PC.

Looking at the data structure, I honestly can't think of anything else that could be compressed or rewritten to take up less space. You'd have to start removing things that are crucial to gameplay to do so (unless the personality value can be broken up another way to save a byte somewhere). You could remove the met location and Pokerus status for an extra 2 bytes I guess, so that's like 21 extra Pokemon with everything else (for a total of 190 extra, or 610 grand total), but I find that that's something that I wouldn't really want to lose.

I'm not familiar with the saveblock, but if there's a way to fit more Pokemon in empty, unused, or repurposed regions, it may be worth it to "lose" 25-bytes worth of data for an even larger boost in storage capacity.

As is, even in 720+ hacks, I'm okay with the 420 storage capacity we're given. There's no way that I'll want to do anything like a living dex in a ROMHack, and I can't see myself wanting to complete the Pokedex in a ROMHack. Would it be cool to have a radical storage expansion? Sure. Am I clamoring for it right now? No. Especially because I want to use the Contest Stats and Ribbon fields in my Emerald hack, and because most of these compressions would break trade compatibility if not converted into the full 100-byte party format properly.

EDIT: Keep in mind that on top of all this compression and data loss nonsense, you'll have to rewrite how the PC interprets stored data so that it's not trying to show 80-byte "slots", or pick up 80-bytes in the hand. This isn't just a mechanical rewrite, but a graphical one as well, seeing as you'll be needing to add "extra boxes" graphically as the screen will not fit 13-14 extra Pokemon per existing box.. Not only that, you'll need a routine that converts a boxed Pokemon to its full, 100-byte format when withdrawn, and to its lossy 55-byte format when deposited. You'd have to put dummy values in the things that we've eliminated, like language and met location, as well as convert the held item back into its standard two-byte index number, and regenerate the total experience based on current level and current level's experience. It'd be a ton of work. On top of that, we didn't hit on a total storage number with our above compression that is divisible by 30, so the last box will only be able to be partially filled, which is annoying.

EDIT2: azurile13 below me reminded me that the current PP bytes would be unneeded, freeing up four more bytes. That'd be a 53-byte data structure with 213 extra Pokemon with a 633 grand total WITHOUT getting rid of met location and Pokerus, and a 51-byte data structure with 238 extra Pokemon with a 658 grand total if met location and Pokerus are lost. The 633 number is pretty friendly with our boxes of 30, giving us 21 boxes with only 3 "wasted" slots left over (within which you could put, like, 3 extra daycare-type slots or something). Doing all of that but saving Contest Stats, Ribbons, and met location is unfortunately only 121 extra slots (541 grand total) with a 62-byte data structure. It lines up nicely with 18 total boxes with only one "wasted" slot left over, but those ten bytes in the data structure for contests are huge...

EDIT3: Maybe a rewrite of how contest stats work would be better. Like, save ONE contest stat, and it's just the Pokemon's overall aptitude for contests (which can be the Feel stat, which controls how many PokeBlocks it can be given). Then, that stat is modified based on the Pokemon's Nature to the individual contest categories. So a Naughty Pokemon would do well in the Smart and Cool categories, and have its Feel stat amplified for those, and poorly in the Cute and Tough categories, and have its Feel stat reduced for those. Contests still kind of work, and each Pokemon has a specific aptitude for a contest category based on its Nature (which is sort of already a thing based on the flavors of PokeBlocks Pokemon like, which is Nature-based.) The different Natures could have overlaps in affinities, but the modifier could be different between them, giving good variation. It could be really extreme too, say a Modest Pokemon will have a severely reduced Feel stat in Cool contests, which mimic the player's general inability to max out all of a Pokemon's contest stats in the vanilla game. There are only 32 obtainable Ribbons in Generation 3, is there a way to reduce the space that Ribbons take up? Perhaps by only saving the highest contest Ribbon for a category and regenerating the lower ribbons upon withdrawal? That makes only 12 (7 non-contest) Ribbons to save, which could be done in two bytes, but I don't know how you would determine the level of the contest Ribbons within only the 9 bits left for contest Ribbons (there may not be enough space). If there isn't enough space, Ribbons would take up three bytes, with likely leftover bits that could go to restoring things like Poke Ball caught in or Game Origin or whatever you'd want to repurpose those bits for.

Without messing with Contest Stats and with reducing Ribbons to only two bytes and saving met location, that's a 60-byte data structure with 140 extra slots with a 560 grand total. With reducing the Contest Stats to just Feel, Ribbons to two bytes, and saving met location, that's a 55-byte data structure with 190 extra Pokemon, with a 610 grand total. Three byte Ribbons with full Contest Stats and met location is 61-byte data structure with 130 extra and a grand total of 550. Three byte Ribbons with just Feel and met location retained is a 56-byte data structure with 180 extra and a grand total of 600. This number is an even 20 boxes with none leftover, and is probably the ideal solution for Emerald without losing all contest data or messing with OTs as azurile13 suggests. It also gives us some bits back for beloved features like Ball caught in, which I'm sure people will appreciate keeping. 52-byte data structure for Firered (removing ribbons and contest stats but keeping met location) is 226 extra, a 646 grand total, which is 21 boxes and 16 slots leftover. Probably the ideal for Firered without messing with OTs is a 53-byte structure (which adds back in ball caught in with bits leftover) which is, again, 21 boxes with only 3 slots leftover. Actually, is it okay that the structure is not word-aligned?

 

[azurile13]

This is something I wanted to do anyways, so here's what I would use:

struct pc_pkmn {
    u8 original_trainer;
    u8 friendship;
    u8 move_lower[4];
    u8 move_upper;
    u8 pp_bonuses;
    u8 pokerus;
    u8 met_location;    //7 bits
    u16 egg_moves;
    u8 ev[6];
    u8 nickname[10];
    u32 experience : 28;    //only requires 21 bits
    u32 markings : 4;
    u32 species : 10;
    u32 item : 10;
    u32 pokeball : 5;
    u32 level_met : 7;
    u32 iv_hp : 5;
    u32 iv_atk : 5;
    u32 iv_def : 5;
    u32 iv_spd : 5;
    u32 iv_sp_atk : 5;
    u32 iv_sp_def : 5;
    u32 egg : 1;
    u32 hidden_ability : 1;
    u32 pid;
};
/* removed:
size: 0x2C
OTID, OT Name, OT gender, etc
Language
Checksum
????
Growth Unused Short
Conditions
*/

u8 = 8 bits, u16 = 16 bits, u32= 32 bits
and : is for bitfields

It could be optimized a bit more, but that's the general idea. It starts with the obvious eliminations such as the unused shorts in growth/unencrypted, language, checksum, current pp. It also eliminates things that aren't needed for FireRed such as contest stats and ribbons. I think the most controversial elimination, but also the one that saves the largest number of bytes, comes from the fact that I really don't care about compatibility with other games. This allows you to create a new byte (it can actually be less than a byte probably) called "original trainer." Rather than having an OTID, OT Gender, and even OT Name, everything in is calculated on the fly. 0 would mean it is the player's pokemon so you read off of the player's literal ID, their gender, their name, etc. Other values would use a dedicated lookup table because actually, in-game trades all have preset values for things like OTID, so it can be read from the ROM. With all of the eliminations, you can reduce Pokemon to 44 bytes a piece, which is pretty nice. I think that most people could actually reduce it down to 40 by eliminating other stuff like pokerus, the egg_moves short I'm using, and other random bits. But even with the reduction to 44 bytes, I calculated that you could have 24 boxes (actually more, but 24 seems like a more natural number) and actually use less space than the current system.

 

[Deokishisu]

I've edited my above post with new insights. Check out EDIT2 and EDIT3 for more math and reduction.

Cut for brevity...

azurile13, by reducing held items to one byte (how I suggested above) and eliminating markings (which are pretty useless, right?) you save seven more bytes. If you introduce a level byte and save only the current level's experience (calculating the total amount upon withdrawal) I thiiiiink you can get away with reducing experience to two bytes (Pokemon in the Fluctuating Experience Group will need more that 65,535 experience points to go from 99 to 100. I see no problem with messing with the lookup table and lowering the experience needed for just this one level from 68,116 to do this.) So that's another whole byte saved, bringing you to eight bytes saved and a 36-byte data structure. That'd be 513 extra Pokemon! A 933 total, 31 boxes with 3 slots left over. By eliminated the other four bytes as you say, that's a 32-byte data structure. That'd be a whopping 630 extra Pokemon with a 1,050 total. 35 full boxes with nothing left over. Like you said, it could probably be reduced further from here by interpreting bits instead of bytes, but you'll probably only get like, what, an extra box out of it? Maybe two (but I doubt it). It's not really worth it, in my opinion, to go higher at this point with current Pokemon totals.

 

[azurile13]

I've edited my above post with new insights. Check out EDIT2 and EDIT3 for more math and reduction.



azurile13, by reducing held items to one byte (how I suggested above) and eliminating markings (which are pretty useless, right?) you save seven more bytes. If you introduce a level byte and save only the current level's experience (calculating the total amount upon withdrawal) I thiiiiink you can get away with reducing experience to two bytes (Pokemon in the Fluctuating Experience Group will need more that 65,535 experience points to go from 99 to 100. I see no problem with messing with the lookup table and lowering the experience needed for just this one level from 68,116 to do this.) So that's another whole byte saved, bringing you to eight bytes saved and a 36-byte data structure. That'd be 513 extra Pokemon! A 933 total, 31 boxes with 3 slots left over. By eliminated the other four bytes as you say, that's a 32-byte data structure. That'd be a whopping 630 extra Pokemon with a 1,050 total. 35 full boxes with nothing left over. Like you said, it could probably be reduced further from here by interpreting bits instead of bytes, but you'll probably only get like, what, an extra box out of it? Maybe two (but I doubt it). It's not really worth it, in my opinion, to go higher at this point with current Pokemon totals.

Wow, okay. Read through your posts. Ignored the contest stuff because I don't care about them :P Have fun with this poorly formatted post.

Graphics - not a problem. Even if there were more boxes, each of them would be the same (30 pokemon). There isn't really a need to "add" extra boxes as an entire box is generated as you press left and right. In fact, I'm looking at it now. Quite a bit of the asm is a lot simpler than you'd expect. Like.

ROM:0808D474                 STRH            R0, [R1]
ROM:0808D476                 CMP             R0, #0xD
ROM:0808D478                 BLE             loc_808D47E
ROM:0808D47A                 MOVS            R0, #0
ROM:0808D47C                 STRH            R0, [R1]

You know what that is? That, as far as I can tell, is the only thing handling the fact that pressing the right button while you're in box 14 takes you to box 1. In the case of that one, you'd literally change the 0xD to your new number of boxes and move on. The nice thing is that most of the routines are pretty nicely formatted, so even changing the size of pokemon isn't too big a deal. As far as I can see, the biggest pain seems to be that there are a loooot of them to change.

I think you're mixing up bits and bytes.

u32 markings : 4

means it takes 4 bits, not 4 bytes. So you don't save thaaat much there.

Also, although an interesting idea, sadly introducing a level byte doesn't actually save anything. A dedicated location for level would cost 7 bits, then a requirement of 0xFFFF experience for a level would cost 16 bits. That's a total of 23 bits. In the current format that counts total experience, as indicated by my comment in that struct, you'd actually only need 21 bits. 2^21 = 2,097,152, more than any Pokemon's total experience.

The overall issue is that with the 44 byte thing I put above, I've already put almost everything into bit format. You're actually overestimating the number of bytes that can be further reduced from that 44 number by double-dipping on some of the bits I've already saved :P

The format I gave above could be easily be reduced to 40 bytes if it didn't include some things specific to my ROM Hack. I think that it could be reduced to 36 bytes by jumping through a number of hoops. I'm guessing that 32 or 28 bytes are absolutely impossible without losing very real things, though I'd love to be proven wrong. Having a size that isn't word aligned, although technically possible, would probably be something to avoid.

Btw in calculating your new box sizes, don't forget to factor in box names (9 bytes) and wallpapers (1 byte each). I mean those could teeechnically take a little less space, but mehhh. Also, don't fret too much about using a little more space than the current system uses. Using a few hundred extra bytes isn't too problematic. It's when you're in the upper thousands range that I'd say it is too costly.

 

[Deokishisu]

Wow, okay. Read through your posts. Ignored the contest stuff because I don't care about them :P Have fun with this poorly formatted post.

Graphics - not a problem. Even if there were more boxes, each of them would be the same (30 pokemon). There isn't really a need to "add" extra boxes as an entire box is generated as you press left and right. In fact, I'm looking at it now. Quite a bit of the asm is a lot simpler than you'd expect. Like.

ROM:0808D474                 STRH            R0, [R1]
ROM:0808D476                 CMP             R0, #0xD
ROM:0808D478                 BLE             loc_808D47E
ROM:0808D47A                 MOVS            R0, #0
ROM:0808D47C                 STRH            R0, [R1]

You know what that is? That, as far as I can tell, is the only thing handling the fact that pressing the right button while you're in box 14 takes you to box 1. In the case of that one, you'd literally change the 0xD to your new number of boxes and move on. The nice thing is that most of the routines are pretty nicely formatted, so even changing the size of pokemon isn't too big a deal. As far as I can see, the biggest pain seems to be that there are a loooot of them to change.

I think you're mixing up bits and bytes.

u32 markings : 4

means it takes 4 bits, not 4 bytes. So you don't save thaaat much there.

Also, although an interesting idea, sadly introducing a level byte doesn't actually save anything. A dedicated location for level would cost 7 bits, then a requirement of 0xFFFF experience for a level would cost 16 bits. That's a total of 23 bits. In the current format that counts total experience, as indicated by my comment in that struct, you'd actually only need 21 bits. 2^21 = 2,097,152, more than any Pokemon's total experience.

The overall issue is that with the 44 byte thing I put above, I've already put almost everything into bit format. You're actually overestimating the number of bytes that can be further reduced from that 44 number by double-dipping on some of the bits I've already saved :P

The format I gave above could be easily be reduced to 40 bytes if it didn't include some things specific to my ROM Hack. I think that it could be reduced to 36 bytes by jumping through a number of hoops. I'm guessing that 32 or 28 bytes are absolutely impossible without losing very real things, though I'd love to be proven wrong. Having a size that isn't word aligned, although technically possible, would probably be something to avoid.

Btw in calculating your new box sizes, don't forget to factor in box names (9 bytes) and wallpapers (1 byte each). I mean those could teeechnically take a little less space, but mehhh. Also, don't fret too much about using a little more space than the current system uses. Using a few hundred extra bytes isn't too problematic. It's when you're in the upper thousands range that I'd say it is too costly.

My bad, I saw u32 etc. and assumed that you were using a full 4 bytes for stuff. It was late and that was dumb of me. Nevermind about that! And thanks for the correction on the experience thing. I guess I should be thinking on a bit level as well for maximum compression.

But wow, generating "more" boxes is that easy? Ridiculous. With the box names and wallpaper, I'd just hardcode them to save those bytes if it meant that I could get a nice, round number out of the boxes. If not, eh.

Thanks for being awesome and explaining, by the way.

EDIT: Just realized that XYORAS have 31 boxes. If we can get the data structure down to 36-bytes, we'd have the same storage capacity as the current gen games.

EDIT2: When I calculated the box sizes, I did so using only the current space allocated for Pokemon, so the box names and wallpapers weren't considered. That's ten bytes per box though that could be done away with if they're made uncustomizable and done with a lookup table. If keeping the customization of that, the leftover slots would have to be used for the new box names and wallpapers I guess, in configurations that allow for leftover Pokemon slots.

 

[DizzyEgg]

EDIT: Just realized that XYORAS have 31 boxes. If we can get the data structure down to 36-bytes, we'd have the same storage capacity as the current gen games.

EDIT2: When I calculated the box sizes, I did so using only the current space allocated for Pokemon, so the box names and wallpapers weren't considered. That's ten bytes per box though that could be done away with if they're made uncustomizable and done with a lookup table. If keeping the customization of that, the leftover slots would have to be used for the new box names and wallpapers I guess, in configurations that allow for leftover Pokemon slots.

This is something I wanted to do anyways, so here's what I would use:

struct pc_pkmn {
    u8 original_trainer;
    u8 friendship;
    u8 move_lower[4];
    u8 move_upper;
    u8 pp_bonuses;
    u8 pokerus;
    u8 met_location;    //7 bits
    u16 egg_moves;
    u8 ev[6];
    u8 nickname[10];
    u32 experience : 28;    //only requires 21 bits
    u32 markings : 4;
    u32 species : 10;
    u32 item : 10;
    u32 pokeball : 5;
    u32 level_met : 7;
    u32 iv_hp : 5;
    u32 iv_atk : 5;
    u32 iv_def : 5;
    u32 iv_spd : 5;
    u32 iv_sp_atk : 5;
    u32 iv_sp_def : 5;
    u32 egg : 1;
    u32 hidden_ability : 1;
    u32 pid;
};
/* removed:
size: 0x2C
OTID, OT Name, OT gender, etc
Language
Checksum
????
Growth Unused Short
Conditions
*/

u8 = 8 bits, u16 = 16 bits, u32= 32 bits
and : is for bitfields

Ah, bitfields, that's a very interesting idea. I think it's up to a hacker which things could be removed from the struct and which not. As a Emerald hacker, I'd like to have ribbons and condition stats.
Anyway, Deokishisu suggested shrinking it to 36 bytes. Hm...I suppose if one would hack FR, this could be doable.

struct pc_poke{
u32 pid; //+4
u8 nickname[10]; //+ 10 
u8 happiness; //+1
u8 pp_bonuses; //+1
u8 evs[6]; // +6
u8 location_caught; //+1
//okay now let's have some bitfield fun
u32 exp : 21;
u32 species : 10; max species would be 1024 then, I think it's not that much
u32 held_item : 9; amount of held items would have to be 512 max, I think that's enough
u32 markings : 4; each bit for each of the marking
u32 move_1 : 10;
u32 move_2 : 10;
u32 move_3 : 10;
u32 move_4 : 10;
u32 ivs : 30; each iv takes 5 bytes
u32 egg : 1;
u32 ability : 2; with support for hidden abilities
u32  trade_index : 4; where 0 means it's not traded, values 1 - 15 signify  from which trade you got the poke, so it would read data from the trade  table
u32 level_caught : 7; 
u32 poke_ball_caught_in : 6; with support for more balls
u32 pokerus : 3; value tells the amount of days pokerus will be active
u32 obedience : 1;

All in all the bitfield would take 138 bits. So more than 4 bytes, we'll count 5.
Now, this gives us 28 bytes total. (yay, it's word aligned!).
Now, some technical stuff:
OT ID, OT nickname, all that stuff will be either read from the trainer data or from trade table(depending on the trade value) The pc will heal poke's pps. Ribbons and contest stats do not exist. Language is always English. Checksum will have to go, meaning bye bye poke encryption.(or get modified)
So currently we've got 80 * 14 * 30 + 14 * 1 + 14 * 10 = 33754 bytes is the saveblock3's size.
With my method, we'd get something around 20 000 bytes, so I assume we could store about 1000 pokes with no RAM space wasted. Though, the PC routines would have to get severely rewritten. Also, I may have omitted something important.

 

[Deokishisu]

Ah, bitfields, that's a very interesting idea. I think it's up to a hacker which things could be removed from the struct and which not. As a Emerald hacker, I'd like to have ribbons and condition stats.
Anyway, Deokishisu suggested shrinking it to 36 bytes. Hm...I suppose if one would hack FR, this could be doable.

struct pc_poke{
u32 pid; //+4
u8 nickname[10]; //+ 10 
u8 happiness; //+1
u8 pp_bonuses; //+1
u8 evs[6]; // +6
u8 location_caught; //+1
//okay now let's have some bitfield fun
u32 exp : 21;
u32 species : 10; max species would be 1024 then, I think it's not that much
u32 held_item : 9; amount of held items would have to be 512 max, I think that's enough
u32 markings : 4; each bit for each of the marking
u32 move_1 : 10;
u32 move_2 : 10;
u32 move_3 : 10;
u32 move_4 : 10;
u32 ivs : 30; each iv takes 5 bytes
u32 egg : 1;
u32 ability : 2; with support for hidden abilities
u32  trade_index : 4; where 0 means it's not traded, values 1 - 15 signify  from which trade you got the poke, so it would read data from the trade  table
u32 level_caught : 7; 
u32 poke_ball_caught_in : 6; with support for more balls
u32 pokerus : 3; value tells the amount of days pokerus will be active
u32 obedience : 1;

All in all the bitfield would take 138 bits. So more than 4 bytes, we'll count 5.
Now, this gives us 28 bytes total. (yay, it's word aligned!).
Now, some technical stuff:
OT ID, OT nickname, all that stuff will be either read from the trainer data or from trade table(depending on the trade value) The pc will heal poke's pps. Ribbons and contest stats do not exist. Language is always English. Checksum will have to go, meaning bye bye poke encryption.(or get modified)
So currently we've got 80 * 14 * 30 + 14 * 1 + 14 * 10 = 33754 bytes is the saveblock3's size.
With my method, we'd get something around 20 000 bytes, so I assume we could store about 1000 pokes with no RAM space wasted. Though, the PC routines would have to get severely rewritten. Also, I may have omitted something important.

30 Pokemon with the vanilla 80-byte structure is 2400 bytes per box. 30 Pokemon with a 28-byte structure is only 840 bytes per box. That gives you an extra 55 and change Pokemon in the same box space, or a full 780 extra Pokemon saved across the space of all 14 boxes. That's a total of 1,200 Pokemon stored or 40 full boxes with no slots left over. You'd have to find some space to store the extra 26 box names and wallpaper settings, or hardcode them so that it's not an issue.

 

[azurile13]

Ah, bitfields, that's a very interesting idea. I think it's up to a hacker which things could be removed from the struct and which not. As a Emerald hacker, I'd like to have ribbons and condition stats.
Anyway, Deokishisu suggested shrinking it to 36 bytes. Hm...I suppose if one would hack FR, this could be doable.

struct pc_poke{
u32 pid; //+4
u8 nickname[10]; //+ 10 
u8 happiness; //+1
u8 pp_bonuses; //+1
u8 evs[6]; // +6
u8 location_caught; //+1
//okay now let's have some bitfield fun
u32 exp : 21;
u32 species : 10; max species would be 1024 then, I think it's not that much
u32 held_item : 9; amount of held items would have to be 512 max, I think that's enough
u32 markings : 4; each bit for each of the marking
u32 move_1 : 10;
u32 move_2 : 10;
u32 move_3 : 10;
u32 move_4 : 10;
u32 ivs : 30; each iv takes 5 bytes
u32 egg : 1;
u32 ability : 2; with support for hidden abilities
u32  trade_index : 4; where 0 means it's not traded, values 1 - 15 signify  from which trade you got the poke, so it would read data from the trade  table
u32 level_caught : 7; 
u32 poke_ball_caught_in : 6; with support for more balls
u32 pokerus : 3; value tells the amount of days pokerus will be active
u32 obedience : 1;

All in all the bitfield would take 138 bits. So more than 4 bytes, we'll count 5.
Now, this gives us 28 bytes total. (yay, it's word aligned!).
Now, some technical stuff:
OT ID, OT nickname, all that stuff will be either read from the trainer data or from trade table(depending on the trade value) The pc will heal poke's pps. Ribbons and contest stats do not exist. Language is always English. Checksum will have to go, meaning bye bye poke encryption.(or get modified)
So currently we've got 80 * 14 * 30 + 14 * 1 + 14 * 10 = 33754 bytes is the saveblock3's size.
With my method, we'd get something around 20 000 bytes, so I assume we could store about 1000 pokes with no RAM space wasted. Though, the PC routines would have to get severely rewritten. Also, I may have omitted something important.

Wut. Lol you dropped 100 bits. How did you get 138 bits = "more than 4 bytes, we'll count 5." Your struct size is 41 bytes, not 28.

 

[DizzyEgg]

Wut. Lol you dropped 100 bits. How did you get 138 bits = "more than 4 bytes, we'll count 5." Your struct size is 41 bytes, not 28.

Oh shit, I divided by 32 not by 8. xD Yeah, my bad.

 

[thedarkdragon11]

Just want to ask, what if, aside from removing the contest stuffs, one will also remove the external trade (not in-game trade) in FR to be replaced with an expanded storage, how many slots in total will be available?

 

[Deokishisu]

Just want to ask, what if, aside from removing the contest stuffs, one will also remove the external trade (not in-game trade) in FR to be replaced with an expanded storage, how many slots in total will be available?

Wait, what? The issue isn't ROM space, it's SRAM space. What azurile13 is suggesting is to remove support for non-thissavegameonly OTs, freeing up the space normally used to save a Trainer's Name, ID and SID. Technically, you wouldn't have to necessarily cut off external trade support, it's just that that information would be lost if the Pokemon is deposited in the PC.

 

[azurile13]

Alright, attempt two! I think you'll see that I'm reeeally fighting for every last bit I can save. Split up for easier reading:

struct pc_pkmn {
    u32 species : 10;
    u32 nature: 5;
    u32 gender : 1;
    u32 pokeball : 5;
    u32 markings : 4;
    u32 formerly_met_location : 7;

    u8 shiny : 1;
    u8 level_met : 7;

    u8 friendship;
    u8 move_lower[4];
    u8 move_upper;
    u8 pp_bonuses;
    u8 ev[6];
    u8 nickname[10];

    u32 experience : 21;
    u32 item : 9;
    u32 pokerus : 1;
    u32 obedience : 1;

    u32 iv_hp : 5;
    u32 iv_atk : 5;
    u32 iv_def : 5;
    u32 iv_spd : 5;
    u32 iv_sp_atk : 5;
    u32 iv_sp_def : 5;
    u32 slotted_ability: 1;
    u32 hidden_ability: 1;
};

New size: 36 bytes.

I removed things that were specific to my ROM Hack and did a bit of cleaning. Also removed PID, and the idea would be to remove it entirely from the game. Instead, 5 bits = nature, 1 bit = shiny, 1 bit = gender, 1 bit = slotted_ability. Wurmple evolution is read off of the slotted ability bit because it doesn't have a 2nd slotted anyways, and Unown are 28 different species because in ROMs with expanded dexes, it is effectively taking up the space of 28 species anyways. For the Magikarp guy on Route 12 and Heracross guy on Six Island, I'm sure something else can be read. The only real loss is Spinda spots. You could read off of IVs or something? Idk. I don't know what to tell someone that is super hardcore about Spinda spots.

The hacky innovative method of the day is making better use of the 7 bits that were previously "met location." First of all, we subtract 0x88 from met location: https://bulbapedia.bulbagarden.net/wiki/List_of_locations_by_index_number_(Generation_III) ignoring Hoenn locations so it only needs 7 bits. Then, values above Celadon Dept. Special Area are used to indicate an in-game trade because they have a default met location anyways. From there, the other original trainer lookup stuff proceeds as normal. Its just that it no longer costs any bits :D

- pokerus: just remembered that it doesn't ever disappear in FireRed, so 1 bit is sufficient. If we wanted it to be able to change to that black dot, I'd probably take a bit from obedience? If we wanted to add a real time clock and number of days left…I really don't know :/ I literally do not see another bit that I'd be willing to give up
- egg: instead of a bit, use level met = 0x7F or something like that
- items: reduced to 9 bits, now requires a lookup table

As I said earlier, I think 36 is the smallest possible size of a Pokemon without real loss in functionality.

Size of saveblock3: 0x083D0
If we were trying to match Gen VI:
(36 bytes per pkmn) x (30 pkmn per box) x (31 boxes) = 33480 = 0x82C8
This isn't enough space to save all of the box names, wallpaper, and current box, but as Deokishisu said, it can neatly fit all 31 boxes of pokemon, which makes me happy. I don't know if it would even be worth fighting with the size of the saveblock, because it would be easy to fit the box names in another saveblock. It doesn't really hurt anything to have them elsewhere except for organization.

 

[Deokishisu]

Cut again for brevity...

That's really awesome, but, damn. Losing the PID is huge. At this point, the Pokemon might as well be in Gen 1 or 2 format with some extras. I'm guessing you plan on keeping PID outside of the PC, and Pokemon are converted to the lossy 36-byte structure when caught and sent to or deposited into the PC? Then, a bogus PID is regenerated when withdrawn? That way you wouldn't have to mess with Pokemon generation itself. But hey, it's definitely perfect for the Firered hacker who wants to ditch trade support and include all Pokemon or the Emerald hacker who doesn't use Contests and is also ditching trade support. Link battles will probably work perfectly if the full 100-byte structure is preserved while in the OW.

It's a shame that it's easier to do this than restructure the saveblocks to allow for a compressed, but otherwise intact, format to fit somewhere that allows for more storage space. I mean, you can shave off 12-bytes as-is without actually losing anything in the structure except Pokerus (which, realistically, none of your players' Pokemon will ever contract anyway) and that alone grants you 74 more slots in the existing space. And that's without doing it at the bit level, which would probably shave off more (I would do it myself at the bit level if I knew realistically how many bits the used vanilla data in each category actually takes up.)

 

[thedarkdragon11]

Wait, what? The issue isn't ROM space, it's SRAM space. What azurile13 is suggesting is to remove support for non-thissavegameonly OTs, freeing up the space normally used to save a Trainer's Name, ID and SID. Technically, you wouldn't have to necessarily cut off external trade support, it's just that that information would be lost if the Pokemon is deposited in the PC.

I'm aware about the ROM/SRAM space stuff... I just asked the wrong way but you almost got what I'm trying to ask and clarified me about the trade stuff too... So yeah, if the trade info (and the contest info, ribbon, etc.) will be removed, how many slots in total would be available (at least in FR)? And would that also mean that in-game trade info will also not be saved?

 

[wally-217]

This kind of stuff is all way beyond my usefulness so I don't have any valid research to input. However, I think it should be worth noting that, if someone wanted to "catch 'em all," then realistically you wouldn't need to have one of each 722 pokemon in your boxes. And I'm not even talking about releasing them back into the wild once you capture a new type of pokemon. Say you obtained a Goomy, you can evolve said Goomy all the way up to a Goodra and you'll still have the prior evolutions registered in your PokéDex.

There are 439 Fully evolved Pokémon, disregarding Mega Evolutions but including All forms (except Furfrou, Arcues and Genesect) and Magearna [Or 460 if Arceus and Furfrou forms are necessitated]
There are 423 fully evolved Pokémon if you also discount Legendary forms, which you can presume to only obtain one of.
There are only 385 fully evolved Pokémon if you discount all forms (Unown, Florges, Gastrodon, Basculin, Gourgeist and Wormadam).
So in reality you don't need space for every single pokemon, an extra couple would comfortably futureproof roms on this front. Personally, I would say that 18 boxes would be the ideal as that leaves one for each type and a healthy 540 total spaces, or 19 if one wishes to include a box for all Unown forms. It would also allow one box of each wallpaper type (16 in total or 17 with the "Friends" wallpaper in Emerald". Anything more than that doesn't outweigh the stripped functions imho and I don't think trade OT details should be sacrificed as it's going backwards and then effects lotto chances and other things.

As for the logistics, well, I'll leave that to you. I just thought I'd chip in my two cents.

 

[azurile13]

That's really awesome, but, damn. Losing the PID is huge. At this point, the Pokemon might as well be in Gen 1 or 2 format with some extras. I'm guessing you plan on keeping PID outside of the PC, and Pokemon are converted to the lossy 36-byte structure when caught and sent to or deposited into the PC? Then, a bogus PID is regenerated when withdrawn? That way you wouldn't have to mess with Pokemon generation itself. But hey, it's definitely perfect for the Firered hacker who wants to ditch trade support and include all Pokemon or the Emerald hacker who doesn't use Contests and is also ditching trade support. Link battles will probably work perfectly if the full 100-byte structure is preserved while in the OW.

It's a shame that it's easier to do this than restructure the saveblocks to allow for a compressed, but otherwise intact, format to fit somewhere that allows for more storage space. I mean, you can shave off 12-bytes as-is without actually losing anything in the structure except Pokerus (which, realistically, none of your players' Pokemon will ever contract anyway) and that alone grants you 74 more slots in the existing space. And that's without doing it at the bit level, which would probably shave off more (I would do it myself at the bit level if I knew realistically how many bits the used vanilla data in each category actually takes up.)

When I made the post, I was thinking of creating literal gender/shiny bits and stuff in the 100 byte structures. It actually isn't hard to find everywhere the game looks for PID because it seems to always use a get_attr instead of quickly ldr'ing it. But it occurs to me now that there's the factor of battle structs which I'm not too familiar with. It also creates "fake" pids when doing stuff like showing pictures in the Pokedex, and I'm not sure how difficult it would be to find all of those. So for the time being, yeah, I suppose that creating "fake" PIDs would be the way to go. Just as long as there is no way in which the Pokemon could "change" in a noticeable way. I'm pretty sure Spinda spots would be the only thing that would require changing, but if someone knows of another difference, let me know.

I'm aware about the ROM/SRAM space stuff... I just asked the wrong way but you almost got what I'm trying to ask and clarified me about the trade stuff too... So yeah, if the trade info (and the contest info, ribbon, etc.) will be removed, how many slots in total would be available (at least in FR)? And would that also mean that in-game trade info will also not be saved?

Using the 36 byte per Pokemon I wrote out in my last post would allow for 31 boxes = 930 Pokemon, same as Gen VI. In game trade info would be saved. See my initial post where I discussed original trainer.

This kind of stuff is all way beyond my usefulness so I don't have any valid research to input. However, I think it should be worth noting that, if someone wanted to "catch 'em all," then realistically you wouldn't need to have one of each 722 pokemon in your boxes. And I'm not even talking about releasing them back into the wild once you capture a new type of pokemon. Say you obtained a Goomy, you can evolve said Goomy all the way up to a Goodra and you'll still have the prior evolutions registered in your PokéDex.

There are 439 Fully evolved Pokémon, disregarding Mega Evolutions but including All forms (except Furfrou, Arcues and Genesect) and Magearna [Or 460 if Arceus and Furfrou forms are necessitated]
There are 423 fully evolved Pokémon if you also discount Legendary forms, which you can presume to only obtain one of.
There are only 385 fully evolved Pokémon if you discount all forms (Unown, Florges, Gastrodon, Basculin, Gourgeist and Wormadam).
So in reality you don't need space for every single pokemon, an extra couple would comfortably futureproof roms on this front. Personally, I would say that 18 boxes would be the ideal as that leaves one for each type and a healthy 540 total spaces, or 19 if one wishes to include a box for all Unown forms. It would also allow one box of each wallpaper type (16 in total or 17 with the "Friends" wallpaper in Emerald". Anything more than that doesn't outweigh the stripped functions imho and I don't think trade OT details should be sacrificed as it's going backwards and then effects lotto chances and other things.

As for the logistics, well, I'll leave that to you. I just thought I'd chip in my two cents.

You're not wrong. But Pokemon has always allowed for more slots than what you'd need to have each family in your box. Going up to 31 boxes is merely keeping up with the standard.

Fortunately, however, this is something the user could decide. If I were to do this (not saying I would), I would open source it and allow people to define the pc struct and number of boxes as they choose. So if they want to keep OTID, contest stats, or whatever, they can by added it to the struct at the cost of less boxes. Everything would be written using the struct and a single "number_of_boxes" define, so changing around what is saved wouldn't be too difficult.

Also, to keep some perspective. Even right now, almost all ROM Hacks break compatibility with other games. Adding Pokemon, adding attacks, getting rid of encryption. There are so many things that break compatibility with other ROMs. So, in terms of a FireRed Hack (which doesn't even have lottos) losing OT details of other ROMs is very minimal. Another thing is that when I say "lose compatibility," it isn't even that major. It wouldn't even be so much as a bad egg. All that will happen is that the OTID of the other person's Pokemon will match your own.

 

[Deokishisu]

Also, to keep some perspective. Even right now, almost all ROM Hacks break compatibility with other games. Adding Pokemon, adding attacks, getting rid of encryption. There are so many things that break compatibility with other ROMs.

Yeah, but a hacker who wants to add all that but maintain compatibility would likely try and go the Time Capsule route and filter what heads into the link rooms when connected to a vanilla game. At the very very least, a bunch of hacks keep link functionality so that their player's can link to each other while playing the same hack which the OT thing partially botches. I'm not saying that your solution isn't great and perfect for most people, I'm just saying that those that are worried about compatibility will naturally have workarounds to keep all of the new innovations while maintaining it. Those hackers are probably going to feel that losing OT info is too big a loss.

But like you said, once the knowledge of how to compress, hack the boxes to support the new format, and decompress/regenerate when withdrawn is out there, people can tailor this to their needs. So the point is moot (unless someone somehow magically comes up with a better way to do this that we're just not seeing that preserves more with less).