For some video games (notably Tales of Games) party members have the magical ability to shrink. Think about it, where are they when you're on the field, and only one character (whether it be the main character or - if you can - a character you should to be the field character) is wandering on the field. Everyone else in the party must have shrank and went into the character's pocket, or something along those lines.
Then there is the whole battling aspect, where you only have a few people battle. What do the others do? Sit on the sideline and drink tea and cheer them on? Or perhaps in some cases they cook the food you're going to eat after the battle (again, Tales of). We know for sure (least in Symphonia) that only four people fight, because Gnome mentions it after you beat him.