Been a while since I've last played through a third generation game, so trying hard to remember which E4 members I've been having trouble with. I definitely remember having huge trouble with Agatha after helping Lorelei on 4 island in FR/LG, but to be honest, the difficulty was raised a huge amount at that point, so they were all pretty hard after that. But yeah, Agatha has always been relatively hard for the fact that she, if I remember correctly, had some relatively bulky and speedy Ghost types, so my Alakazam would usually take some pretty critical hits, in case it didn't take them down in the first blow. The fact is, I'm not very diverse when it comes to choosing teams, so I would often have to rely on an Alakazam and a typically underleveled Sandslash to help in those battles, and well. They usually weren't that successful with her Gengars and Haunters.
As for the Hoenn E4, I seem to recall having quite some trouble with both Glacia and Phoebe. I don't know what it is with the Ghost type trainers, but I rarely ever use Dark or Ghost type Pokemon myself, so I wouldn't have Pokemon that were strong against them, and as such, her Pokemon weren't quite that easy to take out. She wasn't a complete threat, but the last time I played through R/S, I honestly had to rely on my Kyogre to help me through it all, but that was mostly because I was underleveled.
Glacia was honestly made less hard when I had both an Aggron and a Breloom to help me, but the final Walrein made it all hell. It was incredibly bulky, and had moves that were super effective against both of those, and as I often breeze through all of her other Pokemon, she'll usually have a bunch of Full Restores saved for just that Walrein, stalling the battle for such a long time, and draining out a lot of my PP. She was basically just the one who was there to wear me out for the final battle by stealing all of my revives, potions and PP, but sometimes critical hits would save my buttocks ;)
I honestly find both of the E4's to be hard, and am so happy that Surf is a thing, because it's saved me so many times when I've been against a typing which I didn't have a Pokemon which was strong against, and Surf is basically just a killer in those situations :p Long live Water types!