My problem is this: How does one do horror in an RP?
This is an interesting concern.
Horror is a mood you hope to arouse out of the audience, and that can be achieved in a variety of ways. I think a dependence on what has arguably become stereotypically "scary" situations will not suffice, as just the situation alone does not carry the mood as it does delving into the ideas of
why a circumstance is scary, and the emotions that come of it.
Let's take your example. Six players are now separated into two groups by a cave-in. A possible outcome of it is death. Is that scary?
Well, what's the significance in having the uniform group before, and what's the drawback now in being separated? Since death is a possible outcome, in what way could that outcome be met from this separation? This is all assuming that the cave-in is, of course,
a critically bad thing, or something that will create the domino effect
into something that is bad. But why the assumption? We have to give some agency to the characters and the overall setting before we can give any weight to this event.
Maybe every character in this group has something essential to each other's survival, and moving on would certainly hinder or even impede progress without the others' skills/items/etc. That creates a sense of loss. Knowing you cannot move on without this or that else you will meet demise in some way can arouse a sense of dread, which is prominent in horror.
Maybe on one side of the cave-in, there is a leaking, noxious gas, and now there is a lethal build up because this large exit is blocked by debris. Now, this half of the group has to do something to get themselves out of the situation or die by the end of a time frame (however that is to be counted by the GM--perhaps a post limit per player or your own timekeeping posts, etc.). This creates a sense of urgency. Knowing you cannot move on in a hazardous situation that will kill you if you act incorrectly or without proper prioritization arouses dread.
Maybe there is a quiet sound like whispers and gurgles in the pitch-black distance further down the cave--and that direction is the only way to proceed, as no one has the strength or tools to break through the debris. Perhaps the group was warned that this cave was unexplored past a certain point, a point they long ago crossed. Perhaps this was a known place of safe refuge for scavengers long ago. Perhaps the group knew nothing of this cave whatsoever. Whatever the case may be, no one knows what those sounds are, but it's there, it could come closer, and if push comes to shove it is in the direction of the only
possible exit--
that they think they know of. This creates a sense of the unknown or nondescript. Not knowing and having that ignorance loom over you yet needing to take an action that could lead to your death arouses dread.
There's more to it still. Mood in writing can be a bit tricky, but it can be affected by indulging in creative sensory details. You can use sensory details to build up the atmosphere of a situation. Elicit an emotional response in describing and interacting with setting, or playing around with descriptions of intangible things like creeping feelings to unnerve, shock, make uncomfortably intimate, or uncomfortably distance a reader to a character or setting. Tone in your post as a GM or a players' post can also work to create a sense of horror--either by keeping uniform or using a more critical level of dissonance and deconstruction through the tool of the character and possibly the voice in narration.
TL;DR Horror is horror not because an event or action is in itself "scary", but in the senses you evoke through circumstances that are greater than and surround the characters. Exploring the connection between understanding the importance of the situation, the nature of the emotions that rise out of it, and tinkering with the sensory descriptions (and even the lack thereof) of both can help give it legitimacy. It's also well worth it to keep in mind that the horror genre capitalizes on not so much the
mortality of man, but our
vulnerability.
