Most difficult badge to achieve would have to be Whitney's from Johto League, without a doubt.
Whitney's team, as most of us know, had consisted of Milktank, which was quite difficult to take down on many of our ends, including mine. Its defenses were rather top-notch, as well as health, allowing it to endure a ton of hits. On top of this, this Pokémon also knew the move Milk Drink. With this, it was able to recover its health any time we wanted to, which was frustrating to see (whenever we got to get its HP down, one way or another, it'd find a way to use Milk Drink). If it weren't for that, I'm sure things wouldn't have been as challenging.
Aside from here, I feel Tate and Liza from Hoenn were also tough. Their Pokémon, especially Lunatone and Solrock, were pretty bulky; regardless of how many times you hit, they would never cease to get around to surviving and striking back. It was pretty surprising, actually. Due to similar reasons, I also found Wallace to be sort of difficult; some of his team members were quite bulky.