In the games the region is scattered with trainers magnitudes stronger than the E4 so you can advance beyond them, so anyone could defeat them really. Which makes me think that the levels that the E4 are at is not meant to display their skill - in my headcanon, they pick levels based on your Pokemon. They're testing your skill, seeing if you can out-strategize them at the same level.
On topic though, I would say the most prestigious place in the Pokemon world is the professor. He chooses who becomes a trainer, who has the potential to be great, and he's respected by everyone in the region.