What are some really interesting web crawling projects?

A nice little project that I've done in the past is a simple 'sites similar to X' recommendation engine. You crawl sites, strip out any html tags, and build word lists for the content on each one, then you just need some metric for comparing how similar they are to one another (I used the http://en.wikipedia.org/wiki/Jaccard_index to calculate the similarity of the word lists for each site) The user can then just enter a url and get a list of the most similar sites (could be a console app, or I did mine with a web front end) It's a heap of fun to tweak, and you end up with something pretty usable at the end of it.

I've got an ongoing partially unresolved inquiry into converting sitemaps into mindmaps [1], which in ideal conditions would involve crawling a website, parsing the directory structure into a nested outline, and then inserting the crawled webpage titles into the outline.  A cloud based (web) API would be the ideal form for this.  I can further recommend my Meta Guide webpage, "100 Best Web Crawler Videos" [2]. [1] http://www.meta-guide.com/home/ai-engine/mindmap-conversion [2] http://www.meta-guide.com/home/about/best-of-the-best-videos/100-best-web-crawler-videos

http://Import.io is by far the best crawling software. Unfortunatly it Will be no more free starting next month (over priced, Too expensive).The best thing to do is learning Python And use beautifulsoup and/or all the frameworks using this library. It has no limit, with some research you can crawl any website even if it's known to be "uncrawlable" (google results, linkedin, ...)