I'm guessing by "you" in that question, you're referring to the general public? (or are you actually asking what would happen to me? XD)
If a member of the community abused reputation, they can get infracted, yes. And if the problematic rep persists, have the power taken away all together.