It's good that other people also question this. Personally, I think both genders should be allowed, yeah. I mean, I honestly don't see the difference between a topless man and woman. Okay, so you'll always get guys (most likely hormonal teenage ones) who'll be like "ZOMG BOOBS ARGHHHH", but most people aren't quite so easily distracted.
Besides, it's unfair to let one gender do it and exclude the other one. Even walking around in a bra is frowned upon (I honestly don't know if it's illegal in the UK - probably). I mean, come on.
But the reason people don't want to see it is because it's out of their comfort zone. Their socially constructed comfort zone.
Embarrassment about nudity really is just something we're pretty much trained into.
As for seeing it as inherently sexual... I've walked through a nudist beach full of guys who'd be widely considered as attractive and I honestly can't say I paid any attention. It's just a meat suit. Who cares?