I personally had the easiest time with X and Y. They're not bad games by any means, but I feel that there's a bit too much hand holding. Even with Black and White's abhorrent amount of tutorials, I still found it to be a good challenge, especially towards the end. X and Y however, I don't think there was a single gym leader, Elite Four, or any other boss in the game that I struggled with except for maybe Lysandre's Mega Gyarados and even then, it's not like I lost. Just had to rethink my strategy. The Champion of the game is a complete joke as well.
In my experience, I've found Platinum and Challenge Mode Black 2 and White 2 (even the normal mode can be difficult, but I was just playing challenge mode yesterday, so it's more fresh on my mind than normal) to be the most difficult games. Never overbearing in any way, but definitely more challenging than other games in the series and some trainers did send me packing a couple of times, like Cyrus in Platinum. Some trainers in Challenge Mode White 2 even knocked out at least one of my Pokemon, causing me to heal after almost every battle. Plus, the dungeons in that game tend to be very long and demanding.
I don't find most Pokemon games to be very difficult anymore though, but I do still think Platinum and BW2 still stand out as some of the hardest. RSE was difficult for me as a kid though, so I guess that deserves an honorable mention, however I no longer struggle with it so it clearly was just an age and experience thing.