I think the main reason people are opposed to giving so much power to a small number of people is because they're worried about oversight. I think that's a valid concern. I'm sure there are also ways to address it, though I think that's a topic that would need to be discussed at length.
The US education system teaches that consolidated power is always bad and democracy is always good. I disagree with that notion. There have been plenty of benevolent and quite effective dictators throughout history, plenty of monarchs that acted in the best interest of their own people, plenty of ruling families that have reigned over nations that prospered. And yes, there have been fewer of those than of the bad sort, but look at the history of mankind. We've been warlike for millennia and many countries were ruled by those who fought their way to the top, killing any that stood in the way and seizing power. Not the sort usually dedicated to their peoples' well-being. Many other such rulers came into their position because of their heritage. Not all of these kinds of governments have to be that way. There can still be some form of election, it just doesn't have to be one directly from the people. Maybe election by experts, or by those who do the most for society.
Anyway, we're hardly a democracy here in the US, anyway. Whether you think it's a good thing or a bad thing, you'd be naive to believe that our government is run by the direct will of the people and not the direct will of those with the most money. I'll admit that it has largely worked out so far and that we've generally prospered under what our plutocracy. However, that only serves to further emphasize my point: democratic government isn't the only effective form of government.
Why is democracy treated as some inherently good concept? Why is the will of all people held so sacred? People are a mixed bag. There are great ones and there are nasty ones. Your average one is, well, average. Ignorant about many things and often reasonably self-centered. A democratic government is going to be chained down by the worst elements in society. A non-democratic system isn't necessarily going to be better, but it has the potential to. Look at your average representative or senator in Congress. They "care" only as far as it gets them elected and gets them more money. There are outliers on both sides, people who are downright corrupt to the core and people who genuinely care about doing the right thing. But the average rep is going to be right in the middle because that's how democracy works.
Anyway, many people seem to have the false notion that dictatorships, oligarchies, monarchies, any government that consolidates power in one person or a few people are necessarily authoritarian. This simply isn't true. There's no reason a powerful ruler can't give their people the same sorts of freedoms we enjoy in our society right now.
Sorry if this was a bit rambling, I was kind of considering the idea more as I went along.