A good start would be to go to smogon and read the "checks and counters" sections of the analyses in their strategydex. The section has detailed info on what Pokemon can stop the Pokemon in question. This can be helpful for both using a Pokemon offensively, and defending against a certain Pokemon. The best method, though, is to have some foresight in your battles. If you see a really threatening Pokemon like Cloyster, try to avoid getting into situations that would let it set up easily if the rest of your team can't take it on after a Shell Smash. For example, don't use a Pokemon that can't hurt Cloyster to kill something on your opponent's team if your team is really weak to Cloyster in its current state. Just make it air tight for Cloyster to set up, because it's dead weight when it can't.
Additionally, if you are in a situation where Cloyster threatens a KO and you want to switch, weigh the consequences of staying in and possibly losing the Pokemon that you have vs. Cloyster, to the consequences of Cloyster setting up a Shell Smash. Yes, this is a prediction and therefore a guess but there are cases when staying in on Cloyster is the better choice. You wouldn't want to lose the match entirely because you switched out a Salamence that can OHKO Cloyster but does nothing against the rest of the opponent's team out of it and having Cloyster set up SS. Even if you predict wrong, you can force Cloyster out and buy time to set up a Pokemon of your own and hopefully sweep [note that this mitigates Cloyster since you're on the offensive and he has to respond to you, hell, he might even fodder Cloyster]. Also, priority moves like Bullet Punch and Mach Punch really help against Shell Smashers.