Hooters' "image" is young, buxom women? Why do they get to say what their image is when it's something like that? Shouldn't their be limits on what your "image" is. 'Cuz if not then, hey, I want to start a business that has an image of "whiteness." Sorry, dark skinned people don't fit into that image so I'm not hiring any. I'm not actually starting a business. That was sarcasm.
You seem to be under the impression that the image is just arbitrary. In fact, it's designed that way because they are A. Part of the product and B. required to appeal to the target audience. This is not discrimination, as its something that men are, by and large, attracted to. The same would not apply for your white/black example, because there is no reasonable reasoning for it.
Shouldn't a sports store or gym be allow to turn people down based on their weight? It directly impacts the image of the business in a way that will make customers less confident in the product(s). Now there is an image that needs to be retained in order to keep business. This is in parallel with the thread topic.