Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Does Internet Archive have a distributed residential IP crawler program? I would enthusiastically contribute to that.

There must be some mechanism to prevent tampering in such a setup.



The Internet Archive does not, but Archive Team does: https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior


Yes! I'm running an instance right now.


No, IA does everything above board and even honors invalid DMCA takedowns.


> There must be some mechanism to prevent tampering in such a setup.

Trivial as long as they terminate the TLS on their end, not yours. So you'd just be a residential proxy.




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: