Exactly. I mean, my family is Christian, but if someone were to tell us "Happy Passover" (wrong season, yes), I wouldn't be bothered (and frankly, my family celebrates it anyway).
To be honest, if someone came up to me and told me "Happy *insert holiday I don't celebrate*" that would be awesome. I would really love it if someone told me "Happy Hanukkah" you don't even know omg. I would just love that seriously.
Just a different phrase for once would be nice! lol