Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I’m not fully subscribed to the idea that all public scraping of websites is bad, or that this project is a productive contribution. It would be nice to have search engines other than Google, and that necessitates bots being able to index your website (in some respectful manner).
 help



It looks like the tool lets anybody that robots.txt allows through. IOW it doesn't stop all public scraping, just the scraping you want to stop.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: