Does Game Freak really have to continue doing this? If so, why?
Have to? No.
That said a new battle gimmick is a very easy and cheap way to generate buzz and to help the battling feel different from game to game. If the gimmick du jour makes new designs then you effectively get free press forever. A bunch of the people who like them will make alternate fan forms and a bunch of people that don't will go on a hate rant that still fundamentally drives engagement.
Plus, Pokemon isn't really just a game anymore. It's very much a franchise first, and a game second. Possibly third. Which means that the games need to cater to the whole franchise instead of being their own thing in a vacuum. This means that the same gimmick that would be nothing but a buzz generator for the game is now the central concept of the new generation of cards, as well as central story beats to the plot of the anime and any mangas.
The throwing away of gimmicks is probably more the games going "I don't want to have balance a bunch of things but I still want to innovate". Because while the cards are healthy-ish enough the way they are (and if they aren't, they can just ban them from the meta - it's normal for TCG players), and the anime can do whatever they want, the games actually have to find a way to balance all these gimmicks if they want them at all.
I also think that a lot of people don't realize that throwing away mechanics wholesale is actually a pretty normal thing in the general JRPG industry. Pokémon is an outlier, not the norm here.
The Final Fantasies tend to have different mechanics among them.
Zelda will change everything except the bare minimum of the franchise identity (There is a Link and there's a Zelda, everything else is up to whoever is in charge of directing the game) every single game.
No one even knows how many Digimon there are because they never bothered to keep any monster continuity between the games.
Even Pokemon has had a few massive changes before that most likely just predate the time of a lot of people here: when Gen 3 came they got rid of backwards compatibility and "Gotta Catch'em All". The entire reason people still thought "Gotta Catch'em All" was still a thing before Dexit is because of the anime making this sentence heavily associated with Pokemon during the height of Pokemania.
Do you think we would ever get a mainline, non-remake game with good old fashioned battling?
Probably not, for the many reasons I outlined above. I also don't think it'd be necessarily a good thing but then again, there are a lot of other lower hanging fruits they could tackle before "doesn't have a gimmick" become a very noticeable sticking point, so whatever.