If anyone is arrogant it's those that think they can make information publicly available but then still control where it is shared.

You don't get to define "netiquette" to whatever you want it to be.

robots.txt in particular has been a giant mistake that only seves to entrench Google. Anyone that wants to archive or index the web as it appears to humans SHOULD ignore it.