Yes, but most people who are against homosexual marriage say that it DOES affect them because it is against their religion and that if they let the homosexuals get married they(the religious fanatics) are not doing their duty to their religion.
Note taken, however, in order for a person to have religious freedoms, that person must allow others religious freedoms.
For example, a Catholic should not be able to dictate that Jewish people should have to accept Jesus into their lives as their savior. Conversely, Jewish people should not be able to dictate that Catholics disregard the New Testament and the belief that the son of god has already inhabited the Earth.
Ultimately, you are correct to an extent, with the exception that it is justified with any amount of logic. Many people of faith overlook that their ability to act and believe in accordance to their faith depends on all other people having that same religious freedom. With that being said, I think that the majority of people who oppose same sex marriage look at homosexuality as something worse than many other sins listed in the bible. I wonder why???
Let's take up Greed or Masturbation for example. Both are spelled out as sins in the Bible. Why don't you see many Christian/Jews protesting and advocating for legislation to ban such sinful acts, and therefore fulfilling their religious duties?
It's because many Christians, like most people, desire to have many possessions and luxuries. Similarly, most Christians, and people, have the desire for "sexual release". However, most Christians are heterosexual, and do not desire to either have a romantic or sexual relationship with someone of the same sex. Therefore, it's clearly just a bunch of hypocrisy; although proclaiming to act in the best interest of their faith, in reality they are subjectively excluding whichever sins that they choose to act upon. Also, there are many people who are non-religious who are in opposition of same sex marriage because of its "moral" stigma, or it's just "gross".
We humans have selfish tendencies. I, myself, have overlooked a lot of my own prejudices over the years. It seems so easy, but in action, it seems so hard for us to not judge other people...even for those of us who are self-proclaimed accepting and non-judgmental people. However, we do have the ability to embrace each others differences, and I think we will continue to become more accepting people in the coming years.