I've heard my share of racist comments, from co-workers to acquaintances, and even family. It's not overt racism, but it's racism nonetheless. I doubt they even realize they're being racist when they make these comments, until it is pointed out to them and they are shocked and at once apologetic.
As for the anti-racism activists, the louder they get, the better off we are, I think. That is because it drowns out the message of intolerance, which is always a good thing. Everyone has the right to speak their mind, but that doesn't mean the people around them can be forced to listen.