I never normally reject, but since this play-through of mine is a rough play-through, I decide to check everything out and do things opposite to what I'll do in my final and main play-through.
So I did do a lot of things in an anti-myself manner, like choosing another starter, choosing the opposite gender character, and even saying no to people at first when they want to exchange numbers, just as an experiment to find out what they reply with, and later exchange numbers anyways.
And yeah, I was surprised at how Pokemon game characters could come up with rude comments like that...
So I guess it's better to exchange numbers on first time, because that way you will know that you were nice, making you feel nice, and you won't have to unlock their rude counterparts.
And I never have a problem with them calling me. In fact, I need them to call me more, because they're not calling me much, and I need them to do so for battles.
For anyone who does have any problem with calls, just don't receive them. It's not like you have to...
Besides, making a wild Pokemon appear, battling others, talking to others, reading signposts and interacting to things, and even talking to your Pokemon following you stops the call.
The only person who bothered me from the calls was the in-game mom, since she was the only one calling too much telling me again and again about the fact that she bought something.
From then on, I ignored her calls, and later on I stopped saving my money and withdrew all I saved. She doesn't call anymore after that.