When I was 14 (when Red/Blue were released) I would talk about it with my buds, but a) generally not where others could hear, and b) not in a way in which people could use the "nerd" label. We just....discussed it.
As for why people frown upon (especially older) Poke fans - it's basically because they don't understand it. I, personally, blame two things for that. Firstly, Nintendo, for promoting the games as being childish, and having the more 'cuddley' monsters like Pikachu and Squirtle stand at the helm of their ad campaigns.
Secondly, the cartoon. Although most non-Pokemon fans actually enjoy the cartoon (who doesn't appreciate the sheer cheesiness of Japanese anime?), it still gives off babyish vibes thanks to (again) the inclusion of kiddy Pokemon. Pikachu in every episode. [Shudder]
I'm just a fan behind closed doors really, even though I'm nowhere near as hyped about it as when it first came out. Still, it gives no-one any ammo to dis, yo'. ;)