Strange...the first time through, I got obliterated by Cilan's Lillipup, struggled with Lenora in general (Watchog was too much once I had taken down Herdier), but then found the others fairly easy - Druddigon even defeated Haxorus.
However, this second time through, perhaps due to my obsessively keeping my pokémon at respectable levels, rather than being overpowered, I struggled against Burgh (well, I say "struggled", I mean I can actually remember what pokémon he had, and Whirlipede hurt Pidove a little), kept getting obliterated by Elesa, struggled against Clay (although his Krokorok only ever KO'd my Sandile once), had to revive Emboar because nothing could take a hit from Brycen's Beartic, and actually used a whole team of 5 instead of just 3 pokémon against Drayden. Although I did obliterate Cress and Lenora, and closely defeat Skyla first time.
I would say there isn't really a hardest one, although Drayden's dragons are pretty tough, particularly if they get to use Dragon Dance. It all depends on your team, really - my first team was full of exceptional members (Rather like Man Utd. in the late 90s), such as Sawk and Victini. Whereas my current team is more of a unit, all benefitting eachother to usually leave a battle with only one of them surviving (more like current Man Utd.)