On PokeCosmo, in general, my attitude towards people who claim that my staff rotational decisions is, "Are they doing their job? If so, STFU." To tell you the truth, every time I rotate the staff (demodding someone and recruiting some other person), I get whining that my decision is stilted, either because I demodded someone who's popular (e.g. Alakazam, friendly member well-loved by most regulars... but demodded because he was a lazy SOB who didn't do anything), modded someone who's a bit questionable (e.g. Spherical Ice, modded because she was the most intelligent person other than Ramen Pirate to frequent the spriting forum), or didn't even consider someone else who's popular (e.g. Blastoise). You really can't let that kind of news get to you because if you do, you're going to be pulling this sort of stunt a lot, and that's really not healthy for a board, especially a small one like PokeCosmo in your case or PFU in mine. (I mean, what you're basically doing is pissing a lot of people off on both sides and inciting fights. Your job as administrator is to keep the peace, not to destroy it because you want to save face.)
Next time, if you're questioned about who's in and who's out, just ignore it. Yeah, people are going to whine and ***** about your decisions. It's inevitable. Eventually, people start forgetting that they were ever pissed off about your decisions -- or if they don't, then oh well. In the end, it's your decision, and if your higher staff (i.e. fellow admins and whatnot -- or even the mods) aren't staging a coup against you and throwing you off the staff, then you know you're doing something right.
Maybe I'm just missing something fairly important here, but if it's just the fact that some people are accusing you of nepotism, then I think that entire mess really could've been avoided. If it isn't, then never mind. Apologies for the long rant. *shrug*