Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

whats the unique value proposition of this compared to the bajillion other web scrapers out there?


Most basic scraping libraries require you to input a bunch of regexs or css selectors to manually specify what you want to extract from a page. They require custom coding for each page you want to scrape. This library is totally automatic - you just pass in an html page and it returns the most 'texty' text on the page with no custom coding.

There are of course other libraries like this (boilerpipe, Goose, etc), but they tend to be written in Java and Python. The very few existing Node solutions didn't fit my needs so I hacked this together. So for people looking for a quick and simple Node solution, this might be useful.


there is the boilerpipe library which I used to do a test on New Scientist a while back - we wanted to use ML to identify usefull clusters of content.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: