Just because someone feels offended, doesn't automatically make something said insulting. It works both ways, not just the one you think. Again, you are basing something off your own experience. Nice that every gay person you know and everyone else feels it's insulting, but you can't possibly make statement like this just based on that. It doesn't apply to every person just because it's like that with you and the people around you.
And it also sounds as if you're saying that I just bluffed that my gay best friend doesn't mind because you said you are surprised that I said that. No, you should be surprised (if you should be at all) that he doesn't care at all, and that's true. He doesn't mind it, he uses it himself and he knows the difference and therefore doesn't care.