Red and Blue: Yeah, I found Gary was pretty hard. His Blastoise just killed my team. Mainly because it could use Blizzard all it wanted (they didn't have limited PP for the opponents back then). I also remember Blizzard being more accurate back then. I think my Lapras was the only one that could take that Blastoise down, at least during my first Elite Four fight.
Gold and Silver: Okay, Lance's team was kind of boring. He has three Dragonites, for Pete's sake. I know there aren't very many Dragon Pokemon out there, especially in the 2nd generation of Pokemon games... but a Dragonair or Kingdra would have been nice. It just gets pretty tedious trying to take down the same Pokemon three times in the same battle (although their movesets do vary a little). As for difficulty, his team was probably the easiest to take down out of all the champions' teams I've faced. It may be that their levels are kind of low... or it may be that so many of them are weak to Ice or Water attacks.
Ruby and Sapphire: I don't remember much from this battle, because it's been ages since I've played the Ruby version. I think I mostly used my Swampert and Gardevoir to take down Steven's team. I really do not remember having any sort of extreme difficulty with this battle.
Emerald: Okay, Wallace's team is a pain. I hate his Ludicolo the most. Especially after his Wailord uses Rain Dance. This ups Ludicolo's speed to a point where even my fastest Pokemon can't go first. Then we bring in the Double Team, and things just get really annoying.
Other things that annoy me about Wallace are his tendency to use poisoning moves, all the Full Restores he has, and Milotic's Recover.
The E4 champion I find the hardest is probably Wallace from Emerald.
I have not yet gotten a chance to battle Gary in Fire Red, and I don't own any versions of Pokemon for the DS.