I'm not too resolute in the rules I use but I know precisely what rules I will not use.
Unwarranted rules, like "NO MARY SUES ALLOWED!"and "DON'T BE AN ASSHOLE!", really turn me off from a roleplay. I call them "unwarranted" because they're usually self-explanatory. That and they're usually bolded/fully capitalized to emphasize their "importance" with some additional details afterwards. An example: "Use correct grammar. This isn't Twitter, you're not texting your "BFF" and you've (hopefully) gotten a substantial education at this point. "Ur" should be spelled "your", "u" you and don't you dare forget to capitalize. Finally, no smilies during dialogue. Do all these things and I'll have to kick you for being illiterate. I take roleplaying very seriously and I expect you all to do the same." This sort of aggression obviously targets less-experienced roleplayers. Instead of driving them away, give them constructive advice when they do show interest.
That and rules that micro-manage. The worst GM is the one that tells you exactly what you should be doing with your character. I remember roleplayer owners that acted disdainful about "bubbly, ditsy, kindhearted characters" because they were "sooo cliche", as if cliches and tropes are inherently bad. Forcing players to design their characters in an exact manner or change a minute detail like their sex ("We must have an even sex ratio!!") or even their name ("Amber is such an unoriginal name! Try something like "Maugerettiettta" or "Little Miss Ramhind"!!!) is pretty obsessive. It's understandable if every character plays a certain role (i.e. 18 Pokemon Trainers, each based off of a specific type) or a minor detail contradicts the universe ("She's a... blue-haired, black-eyed ancient Egyptian...?") because that's basing a character around a concept, not altering a character to fit some crazy criteria.