Well, for gym leaders:
*bulbagarden.net image removed*
Brawly's not annoying though. He's just pretty tough. I can usually take out his Machop without taking too much damage, and his Meditite without taking any. But then Makuhita obliterates me. Usually my tactic has to involve good guesswork to get some stat boosts in whilst fighting Meditite. Which leads to many, many losses.
As for the Elite Four:
*bulbagarden.net image removed*
Phoebe, again, doesn't annoy me in herself. Her pokémon are pretty good, and I'm hoping in the re-makes she'll have a re-match team featuring Dusknoir, as it's like Dusclops but more damaging. She's also nicely sprited, and makes a change from the usual creepy Ghost-type users. What irritates me is the amount of PP I have remaining after the battle - Dusclops are durable and both have Pressure, Banette loves to use Grudge (one of them, that is. The other can't use it), and Sableye uses Double Team. This leads to the most important item to take on the Hoenn Elite Four being a Leppa Berry.
And Champions?
*bulbagarden.net image removed*
Definitely Wallace. Steven has extremely strong, high-levelled pokémon, and can only be battled once. He also battles alongside you. This therefore makes him cool. Wallace, however...whilst it's nice to have a connection between him and Juan, do we really need two flamboyant pretty-boys with water-types? His Ludicolo used to obliterate me, until I figured out that Flying-type moves were super-effective. Wailord is also tough, but only in that I usually rely on a super-powered Thunder to dispatch it, and if that misses, Water Spout is lethal. His Milotic is a good battler, and Whiscash is strong too. Tbh, all that really annoys me about his team is Tentacruel - we face them all the time, anyway. Gyarados is common, but powerful, unlike most encounterable Gyarados.
So, yeah, whilst none of them really annoy me, certain parts of the experience battling these three do frustrate me at times.