If you've got your robots.txt set to tell search engines to go away, or at least... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		rossdavidh on Jan 29, 2025 \| parent \| context \| favorite \| on: AI haters build tarpits to trap and trick AI scrap... If you've got your robots.txt set to tell search engines to go away, or at least don't look at those pages, then you probably aren't worried about that, I assume?

dylan604 on Jan 29, 2025 [–]

there's a difference between an ai-bot and a search engine bot though

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact