It depends on the environment - many years ago I used to have temp job in the summer working on a large industrial plant that had a nice office building where the managers and admin staff were based. There were no signs saying "temp staff keep out" - and you did occasionally have to go in there but it was pretty clear to me that you couldn't go and hang out in there - particularly as the temps got all the muckiest, smelliest jobs in all weathers.
I feel like you don't have any first hand experience with the kind of classist horseshit that is endemic to these kinds of work environments.