To be honest i didn't think that the rivals was that hard, when I just made sure to use a good counter for what they had.
If i had to pick one i would say the last battle against N, would be the hardest, with the last one against Cheren coming in at second place.
The reason the seem somewhat easy to me, is that when i was playing Gold when it was just out, i didn't really know much about tactics and how to use this at your advantage, back then i was more a fan of pure power and just having a level advantage, and beating them even if my own pokemon was weak against my rival.
Which ofc resulted in some harder battles for me.