I can't speak for TFS, so maybe they do something daft, but it's usual that when you get latest from the server you simply won't get the files you aren't allowed to. You won't even see them. It's access control at a finer grain than the repository level, applied on the server side.

(with git always giving you the whole repository, there's not much you could do with git, but but many systems don't do that.)