I don't see a reason why a good bot operator couldn't build a parser lib in a di... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

codingminds on July 17, 2025 | parent | context | favorite | on: I was wrong about robots.txt

I don't see a reason why a good bot operator couldn't build a parser lib in a different language and put it on a public repo.

Shouldn't be that hard if someone WANT to be good.

elric on July 17, 2025 [–]

Sure, but it's always easier to use a tool that's been tried and tested.

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact