Are we counting the champions as part of the Elite 4? I've always pretty much considered them part of it, even if they weren't specifically an E4 member :p In any case, my pick would be Hoenn. I've just always struggled to get through that one. I mean, I don't think I've beaten it one single time yet in which I didn't have to resort to using the title legendary. And whether that's due to poor decisions when it comes to choosing Pokemon, a lack of effort put into training or w/e, that's just always been the case :p I don't think there's anyone specific that I really had any trouble with, I just found the entire E4 to be relatively challenging.
I also think the Sinnoh Elite 4 was pretty darn hard, and I don't think I made it through it on the first try during my first playthrough. The only reason for that though is Cynthia. It's been quite a few years, but from how I remember it, all the actual E4 members were pretty easy, whereas Cynthia stood at the end to wreck anyone who was starting to feel confident :p The combination of Milotic, Garchomp and Spirtomb was pretty deadly, so yeah.
If I had to list them in order of difficulty, it'd probably go somewhere in the line of:
Hoenn
Sinnoh
B2/W2 Unova
Kanto
Johto
The only one I actually found to be genuinely easy was Johto, and that was entirely because of the low levels. The games progressed incredibly fast, and it wasn't really that hard to prepare yourself for the E4. As long as you've got an Ice type to take out Lance, Koga will be the only one that'll really pose a problem, and that's entirely because of his dirty poison tactics. So yeah, it was pretty hard to choose between Sinnoh, Kanto and Unova, because like I said, Sinnoh was only hard because of Cynthia, and same thing goes for the two others really :p