It is such a tricky subject but I will try to show you my perspective and share my experience of learning English.
I am a Pole and Polish is very different from English. We don’t have certain grammar elements that are at the core of English. They are articles, phrasal verbs, and perfect tenses. The other important fact to know about Polish is the word order. It is quite flexible whereas the word order in English is much more fixed. In Polish and many other languages, you have one word which can be used in different contexts. English, it seems, has a different word for every situation. Let me give you an example. Doors can be locked or closed. In Polish, you have one word for these two adjectives. I think it is much easier.
I used to teach English in Madrid, Spain. I noticed that a lot of my Spanish students didn’t realize how lucky they were to speak Spanish as their first language. Why? They have a lot in common. Spanish and English have much more in common than Polish and English. Yet the Spanish struggle more with learning English than the Poles. I have been learning Spanish for a long time now and it never ceases to amaze me how your native language could either help you or not help you at all with learning English. It is incredible!
Billions of people around the world learn English. Their native languages are very different. I am, by no means, a language expert but to my understanding European languages have influenced or are spoken as first languages in many territories around the world. It potentially makes it easier to learn English as opposed to some Asian languages like Japanese or Korean which have nothing in common with English or other European languages.
Another problem English poses is pronunciation. English words are written one way but are read the other. Again, some languages don’t follow this pattern. Students find it so difficult sometimes to pronounce words that may sound easy at first glance, for example, leopard. It is not pronounced as a combination of leo and pard!
The last thing I believe that makes English so difficult to learn is the number of accents and regional dialects. When I was traveling around the UK, I noticed that every 30 to 50 km I could hear a different English accent. It was a problem at times. In the US alone there are more than 40 different accents!
Certain components of English can be unpredictable and counter-intuitive to use but once you have laid a strong foundation it becomes easier to learn.