That is true, but you cannot deny the fact that there was still hatred towards gays by this minority even though the Roman lexicon has no word for homosexual.
There's always going to be people with dissenting views no matter how homogenous the population is. My point is that, historically, there were societies that did not condemn homosexuality and at the same time did not have a label for it. This goes to show that labels do matter in that they affect the way people perceive and behave. I'm not going to argue over technicalities about how you couldn't consider those societies to be tolerant of homosexuality because you could find a guy or two who stigmatized it.