I'm not too worried about that really. There are other artists I like (that have an accumulation of albums already), and I'll listen to one album and then another, and they'll be different, but I'll still like them. I recently downloaded a different Mars Volta album, that everyone claims to be the worst (and the other album I had been listening to was their two acclaimed best albums). I personally find nothing wrong with the "bad" album. I think it's really good. So, in regards to FtP's next album, I don't think I'll end up hating anything on it. Didn't happen when Kings of Leon released Come Around Sundown last October either, so yeah. Not worried. :)
That in mind though, I don't like when fans are all like "ohh hurr hurr this album sucks I wish they'd do what they did in the (insert first album here) days" That is SO annoying. And I have a feeling we'll have those Foster the People "fans" in the future saying "they suck now Torches was the best hurr hurr." Maybe it will be the best, but artists grow and mature over time. If they didn't change over time to some extent, all bands would be Nickelback. No offense to anyone that likes them...they have a few select good songs. lol