Rarity isn't about how many of them are in the set, it's about the likelyhood of pulling a given card from a pack. There could be 20 commons in a set and 20 holographics, and the ratio of pulling a holo from a pack can still be 1:4 as opposed to commons, which is more like 7:1.
On an aside, there is a kind of sub-rarity within cards, where you are more likely to get a specific rare card than you are any others. The best example I can give is from the Japanese Lost Link set, the most recent set released in Japan; there are four prime cards in that set, all with the same rarity symbol (ultra-rare), but you are much less likely to pull a Mew prime than any other prime.