A guy and a girl can both be attracted to guys and they get called homosexual and heterosexual even though the thing they have in common is their attraction toward males. Maybe I'm a little weird, but when I'm attracted to someone I don't check my gender to make sure I'm using the right term. I suppose if your own gender defines you to a large extent you can't help but think of your gender in relation to that of someone you're attracted to and then use whatever term describes that relationship, but if, say, you were intersexed? Now you're in a pickle. You would have to use a term like 'androphilic' (attracted to males) to accurately describe yourself, but how many people use a word like that, let alone know what it means?
Er, short version: There are all kinds of people and labels suck.