Assuming you mean the Contests that RSE had, I have no idea how they work internally at all. I think they would be a cool thing to add at some point, but I have done absolutely zero research into actually doing it.
If you mean something like the Bug Catching Contest, I haven't done that yet either, but it would be a lot simpler to add. You'd basically just like...
- Make sure the player only has one Pokemon in their party, by just checking the party count RAM address.
- Come up with an extra battle menu, that is the same as the normal battle menu, but has the Safari Ball option instead of the Pack option. Haven't looked into the specifics of this yet.
- The lazy way is to end the contest if the steps run out OR if the Player has 2 Pokemon in their party (meaning they caught one). If you want to make it so they can swap out, that might get trickier. Probably need to come up with a routine that checks to see if they have 3 Pokemon, and asks if they want to swap their New Pokemon for the one they had. If they don't, it just decreases the party count back to 2. If they do want to swap, it would copy the Mon3 data over top of the Mon2 data, then decrease the party count.
- Make up a routine to judge the contest results. This would probably be the most complex part, depending on how you want scoring to work. I've looked into the way Gen 2 does it and the way Pokemon Gaia does it (though Gaia is a Gen 3 hack, and the code is written in C, its still an example of how the logic works).
EDIT: For the logic I described to work with swapping out the new one for the previous one, the player would have to be forbidden from rearranging their party during the Bug Catching Contest, so I'll probably want to come up with a better way to do it before I actually implement it. But that's my initial brainstorming/notes on it, anyway.
Also, did a bit more looking, and it looks like the problem with EXP on Capture was that wWhichPokemon gets used during the GainExperience routine (naturally) but the RemoveItemFromInventory routine expects that value to be the number of the item slot to remove. That's why it was removing a seemingly random item from your pack. So you could do something like this.
Go here and add this:
Then go right above this line and add this label:
And see if that works. That *should* make sure you don't lose your place in the bag menu, so it removes the proper item at the end of the routine, and then properly give EXP to the Pokemon that fought (and all of them if you have the EXP ALL). I haven't actually tested that yet, but it looks like it should work. Let me know if it works for you or not.