Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You shouldn't really serve aggressive scrapers any kind of error or otherwise unusual response, because they'll just take that as a signal to try again with a different IP address or user agent, or a residential proxy, or a headless browser, or whatever else. There's no obligation to be polite to rude guests, give them a 200 OK containing the output of a Markov chain trained on the Bee Movie script instead.




Unless your output is static, you’d then be paying the cost of running the markov generator.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: