Caves in general annoyed me. Partly because of all those Zubats ambushing me, partly because of some irrational worry I would be stuck in there and never find my way out again... I always carried loads of escape ropes with me. xD
Also, looking back the fight against Lance annoys me now. Back then, it was not annoying, just a challenge. I always managed to beat him down. What I didn't realize back then was that Dragon types are weak against Ice, seeing as Lance was the only trainer to actually have dragons. And I had a friggin Articuno in my team and never used it!!!