I know English really well and have a passable command of Spanish. I have studied Japanese for almost 3 years (self-study) and studied French for 1 year (class).
From this, I would say Japanese is easily the hardest. The other 3 languages have overlap in vocabulary and the grammar isn't too different (just minor things like adjective order and using possession to denote age).
Japanese pronunciation is simple, but it is just too different grammar wise. The distinction of politeness, giving/receiving verbs, no articles, and usage of particles aren't difficult, but is different from what I am used to. I would say the hardest thing about Japanese is that it takes too long to learn, not that any of the concepts are difficult. The most time consuming part is the ever infamous Kanji, which amounts to more than 2500 characters if you want to achieve a reasonable amount of literacy (in my opinion, some may argue for a lower number). It is fun and rewarding though, I suppose. Although, I have pretty much given up at this point.
What was your favorite class that you took in high school or college?