Irritating them is just, well, irritating them. Which, to me, is a momentary thing that doesn't last after I've stopped. Ticking them off is more like something that would stick around and mess up their day.
Hopefully that makes sense. I'm not good at wording things, obviously.