I'm laughing at all the people who are complaining about Lance and his Dragonite. I spent about twenty minutes in Victory Road getting my Feraligatr to Lv. 45, and he was at 47 when I actually got to Lance. I spammed three Blizzards in a row and didn't miss once.
Anyways, the most annoying would be a tie between Falkner and Rival at the west end of Azeala Town (I believe that's the place...). At Falkner I had a Lv. 13 Totodile and Lv. 12 Hoothoot, but that freaking Pidgeotto and it's Roost got me everytime. It took me ten tries and twice as many Leers from Alpha (Totodile). Then the Rival... I just couldn't freaking beat his Bayleef! My only Pokemon with an advantage over it was Gamma (Hoothoot) and all she knew was Peck.
Funnily enough, all those big obstacles that people always moan about are the ones I train for for sometimes an entire half-hour, and then they're no trouble. Example: Blue, Lance, and Whitney were all beaten on the first try. I'm currently gearing up for the massive training period for my battle with Red.