Don't! He can be a bit self-centered at times, and VERY easily get annoyed at other people, but he's also very fair-minded. He's positive that Victoria's mood was because of something he said, so he's only annoyed with himself, and is going to try everything he can to make it up to her again.
However, if she continues to act cold and push him away after several attempts on his end of making it well again, he's going to get annoyed at her :P Because of his whole "Fairminded"ness, since he thinks that as long as you're willing to bend down and apologize, the other person can't expect any more from you, right?