2nd Gen Help! Save file Corruption...

So, I recently got a copy of Pokemon Crystal for my birthday. It was secondhand from ebay, and said that it saves, which it does.
In game, saving works normally, as does the clock; the save battery is not dry.

However, though I can save and either turn off and on or soft reset and still continue my game, removing the cartridge causes the save file to corrupt and I have to start a new game. Every time. It may be useful to note that I am playing on a Secondhand Gameboy Advance SP AGS-001 though I own pokemon Fire Red (also secondhand) and have had no problems playing it.

Does anybody know a way I can fix this issue other than leaving the cartridge in forever? I realise that corrupted save files cannot be recovered, but can I avoid this situation and still be able to remove the cartridge?

Thanks in advance.

Sounds like the battery might not be in properly

Even though it saves fine when left in? Should I just return the game for another copy?

Change the battery. It is seriously not that big of a deal. There are plenty of videos on youtube. If you dont have the screwdriver use the improves melted pen (also on youtube) or do it like I did, with two pins. Removing the battery is also not that hard, in case you break the little metal hands, you can just tape them with the electrical tape. The important thing is that everything is in contact.
There are different methods, but do change the battery, risk of losing the entire save file is not worth it.

I had this issue with one of my copies of Silver. I could save just fine, but then I'd pull out the game or wiggle it too much while it was still in my GBA and it would corrupt the save file. Your battery is probably just too loose. It's working, at least. You'll just need to open up the cartridge and use a stronger tape. Electrical tape should work good and there are plenty of videos online to help show you how to open up and replace batteries on games. The person that replaced it before you probably used scotch and you're going to want a stronger hold that electrical tape can provide.