Grumble, grumble... I just wrote up some great responses for each of my favorite designs and my Firefox crashed as I tried to post. So enjoy each of my favorites with not quite as lengthy responses.
Kanto: Cinnabar. Loved the idea of having to answer trivia questions in order to face Blaine, even if they weren't all that difficult.
Johto: In Gen II, it's Ecruteak. Partly because it's the only one I can remember outside of Goldenrod and Violet City's, but it did have an interesting puzzle. In Gen IV, though, it's Chuck's waterfall gym in Cianwood.
Hoenn: I loved the ice puzzle in Sootopolis' gym, except when you screwed up right at the very end.
Sinnoh: My favorites were either Hearthome or Canalave City's gyms. And with Crasher Wake being a wrestler, Veilstone's gym in Platinum would've suited him perfectly (well, other than the fact he's Water-type).
Unova: Both of my favorites are from BW2, and they're Skyla's gym with all of the mega fans in Mistralton, and Clay's gym in Driftveil with all of the conveyor belts in the dark.