For me I can always tell when someone isn't going to work for me if I start to feel regularly annoyed and start to hold feelings of animosity toward the other. Granted, I've had a couple of extremely unhealthy relationships, so that's probably a part of that. I am also aromantic which can cause some issues because I don't really desire intimacy.
But I think there are a lot of signs in general that a relationship isn't working out. If one or both of them don't want to spend as much time together (as in, they really just don't want to be around that person as opposed to having a healthy amount of time apart so as not to suffocate each other), one or both people in the relationship starting to show interest in other people or maybe even developing feelings for other people. A general lack of interest or desire for their partner. And as someone stated above, people who's goals don't line up very well. It's hard to maintain a relationship when two people want different things. Eventually that will create problems no matter how enamored you were with a person at the beginning.
I also think one of the biggest signs that a relationship isn't going to work is if there is constant fighting. Like, not typical arguments that happen because of simple disagreements, but legitimate fighting with yelling, name calling, finger pointing, blaming, etc... I think those things are a pretty clear sign that you no longer have respect for the other person and probably don't love them anymore (if you ever did at all).