Silph Co. - And Sabrina's gym for that matter. The moving and teleporting titles got annoying and confusing.
Fuchsia Gym- The invisible walls always threw me making me spend a long time to get to Koga. And in Yellow where is Pokemon are overleveled compared to Erica's who had levle 30 Pokemon. Then Koga has Pokemon 48-50? Completely unfair.
The music in Lavender Town in general annoyed me, I hated going through there.