From what I've heard in school, English may very well be the hardest language to learn. It is so different from other languages in many regards. For one thing, every "rule" in English has an exception(ie. "I before E except after C" works for piece, but not neighbour. XD). And this rule itself has an exception, in that "Q must always be followed by U" is the only English rule that stands 100% of the time (Iraq isn't English, folks, XD).
Another thing that I've noticed is silent sounds(ie. the words "know," "gnaw," etc), and different letters sounding the same way(ie. "F" & "PH," "EE" & "EA," "K" and hard "C").
So unless you grow up speaking it, English can be quite a challenge for some. Although those who speak the other Germanic languages(French, Italian, German, etc) should have an easier time than most.