You are right that it happens, but people do care and enforcement is improving in at least several countries like the UK and Norway.

The Olympics and World Cup have come to Western Europe (e.g. England, France, Norway, Spain) over and over again

Never been to Norway or Spain, but England and France are generally okay w.r.t. migrant labor. Not saying there isn't some hostility to foreigners, because ceterus paribus you'll get many more job interviews with a CV that says "Jean-Pierre Petit" than "Mohammed Karkar" in France, but there is no "temporary worker visa" that we can use to basically exploit peoples from poor countries. If you get a visa that lets you work, you generally get the same protections/wages as locals :)

Are the immigrants' passports confiscated so that they cannot leave?

If you genuinely believe that, why don't you move to Qatar?