As many others have said, the R/S/E rivals were the most irritating. Wally was more of a rival when he wasn't actually supposed to be one, but even then he was awful. They were rivals who were just disappointing to battle, since I'd be able to take all of their pokemon out with one pokemon. -_-; They weren't exciting at all, I didn't even enjoy beating them.
As for my favourite, I'd have to go with the guy from G/S/C. While he was sort of scary and mean, he wasn't like Gary who was just irritating to me. He was pretty cool, and he wasn't easy every battle - I actually looked forward to battling him, and sometimes even trained my pokemon a little extra before I battled him since I knew I'd need it.
Gary and the D/P rivals aren't as bad as May/Brendan, but they aren't that exciting. The hyper guy and Lucas were both incredibly generic character, with absolutely no depth. They also bugged me a lot since whenever I'd be force to do a double battle with them, their first pokemon would absolutely weak... usually I'd just use a pokemon with surf to knock it out, but in my most recent game I could not because I didn't have a water pokemon on my team. :|
Hopefully if there's a 5th generation of games will have rivals more like Silver or even Gary, who are actually challenging and have a bit of personality. Although I'm still hoping for a G/S remake so I can see Silver again. :D