Skarm gets destroyed by rain boosted hits, really. Plus alot of Hydration Vaps have Work Up IIRC.
Uh grass types (Virizion, Ferro, Celebi, Shaymin [Natural Cure) and the like) and Rotom-W poop on Vappy. Taunt Jellicent outspeeds Vap and cant Taunt Toxics and Rests and wear Vap down. Sub Mind Latias can set up on Vappy as can Chesto Rest DD Kingdra. STalker Gyara can PHaze it and Rest off any and all damage. Or just change the weather back on predicted Rests. There is also Trick which ruins it.
Heatran and Chandelure wall p much all Venusaur sets without EQ which is exceedingly rare. Sp.Def Dragonite is annoying for it too and can wall even +2 LO Sludge Bomb with SR up iirc. If it lacks Sludge Bomb other dragons wall it too. Blissey/Chansey also beats it quite handily and Chansey walls even the Mix sets too. P2, Sp.Def Deoxys-D and Cress can work in a similar manner. You can also use ScarfTar to change the weather back, outspeed and crush it.
The usual anti weather stall core is Heatran, Gastrodon/Jellicent and Skarm irrc so you could try that.
Hape that helps a little.