Actually I go more for the signatures, unless the person has the same avatar for months, and suddenly change it...
Nica helped me a lot about not caring about avatars, since she changes hers eight times per week XD
But, really, I have known a lot of people who care about the avatar: I once had a more girl-ish one and I was called "her" (despite the fact of having never changed the "male" from my profile O_o).