Your right but you don't have to put every single Super Hero from a Marvel team into the Avengers.
XMEN - Wolverine or Beast (Well Beast was the first of the XMen Team to join the Avengers)
Fantastic Four - At least one of the FF members...more specifiably Mr Fantastic or Human Torch or better, She Hulk!
Yeaaa... I read too much comics...