Well in Y/R/B, for me, it was freakin Brock. The fact that he's first when you're all low-leveled has always made me mad xP I always get so mad when I beat Geodude, and then Onix has to come and kill me.
Misty kinda makes me mad, too, I have a hard time beating her Starmie - she ALWAYS has it do bubblebeam and more often than not, I black out.
On G/S/C, I must agree with the majority of you all and say Whitney and her damn Miltank! D: STOP ROLLOUTING ON ME.