Depends, a swear word to one person is a normal word for another.
For example, the 'N' word, for a black dude its a normal word, for a white dude its swearing and very offensive to black people.
But anyway, yeah, its the meaning to the word which makes it 'dirty'.