If anything, I think it would be one of the minor characters, in which they could easily pry out of the manga and everything would still flow as it does. But as for the main characters, I think it would be difficult to take any of them out without changing the story.
Now, if we're talking about a character never coming back, I can possibly think of a few people. However, I'm sure all those I'm thinking of have already gone up and disappeared for good, or for a long time, haha~