People know I have a "no big deal" attitude for things like being late, being honest/blunt to me, and I don't even mind being blown off if people tell me upfront that they'd rather do something else than see me.
However, the relationship ends when sneakiness, lies, gossip and prejudice start to surface in someone. It's not hard for me to tell someone "I'm not your friend anymore" when I find out they're homophobic or they spread false rumors about people.
For romantic relationships, what seems to be the deal-breaker for me is dependency and attachment. If my SO wants to be joined at the hip 24/7, I've got no problems telling them that if they don't back off, it's over.
For my family, I am slightly more tolerant. My mother is slightly homophobic, but she has learned that I won't listen to her prejudice. I will argue with her about her invalid points and flaws in logic regarding gay people, and she shuts up. I'm not going to disown my mother. If she started being outright offensive to my gay friends (she isn't, she actually has a couple gay friends,) it might be a different story.