well, it depends on what players want. tbh i think 3ds era was one of the worse ones, but it may bu purely personal, while i feel like, even with their flaws, pokemon sword and shield were better than 3ds games. the thing is that imo the major change was the Internet. when gen 5 games were out, not everyone had wi fi and if they had, it wasn't a good connection. i do remember that it was such a rare case that my connection actually worked.
When gen 6 was out, more and more people had wi-fi, that was actually working ok. and u could access it from the bottom screen, which was immediate. the second thing that had the same development was the competitive. before gen6 it was extremely hard to even figure out EV and IV and how to train a pokemon. Gen 6 put that EV training in the bottom screen. People actually noticed for the first time, maybe. Then, online battles, new mechanics. This is what changed. Newer games aren't concentrated only on the single player thing, but on the online things as well, but they also gave much more space to competitive. Sword and shield comes from this kind of view, indeed think about dynamax raids, dynamax adventures or how extremely easy is making competitive pokemon or how fast the game is, in order to let people interested in competitive end the story as soon as possible.
i miss the old games, because they were somewhat more difficult and had decent stories. imo, even without megas, z moves and dynamax, pokemon games are really enjoyable, even competitively. what i like about new games tho is the fact that they're trying to destroy all classical cliches of the pokemon series, like the third title or gym leaders or the fact that legendary pokemon couldn't evolve. On the other hand, something i hate about new games is the pokedex. pokemon was born as a monster collector and it just makes no sense that you don't let me collect all pokemon. Alright, being honest, few people actually collect all pokemon, but this doesn't justify the choice. it's also very sad, not every pokemon has a pokedex page. that's silly imo, even because, just like they had done before, they could have copied and pasted descriptions and it was done.