I personally think that the difficulty depends on three things:
1) Your knowledge of the system. While the veteran player could easily breeze through the installments knowing weaknesses, reistances, etc., a new player will have no idea what they're doing, and will probably find it difficult- or at least more difficult.
2)How you train/your methods while you play. This is a worthwhile example; for example, Emerald was easy for me in the beginning, in fact all the way to the champion, because I pummeled through it with Blaziken and Rayquaza. But obviously, when it came to the Battle Frontier, I was a total screwup. In B2/W2, which is the more notable example, it was more difficult for me to fully storm through the game but I eventually made it with my team. and now my team is the most badass thing on six slots. Or at least, it is in game.
3)The competitve aspect. Whether you're going competitive or not definitely changes the level of difficulty, no matter what, and... well, I suck at that kind of stuff. Nuff' said.
---
Now, the way I play, I find it pretty easy. But that's just me.