I started out scraping sites manually, and started automating more pieces (a lot of sites use wordpress, so they are pretty structured). I'm working on a talk on the subject, so I'll have an article soon that explains better :)