Screen scraping etiquette

For what it's worth, that sounds like a pretty standard terms of service. How can you be expected to not cache pages? There's nothing really illicit about it as long as you don't deny service to others or republish the data. I'd do it via tor, space out the requests to be nice, and let it run for a couple of days .Just use the user-agent for a browser, as well. This idea might conflict with others' scruples, but I feel my answer is the best balance of ethics and utility.

Don't email them, just go ahead and do it. You're being more than nice enough already by doing it during off-peak hours and spacing it out. But don't announce yourself with the user-agent, just in case they're insane.

I'd say hold off on providing your info (use a default user-agent) until (if) they complain. You're already being nice about it, spreading the requests and all, so I think you're in "ask for forgiveness, not permission" territory.

I'll echo the "just go ahead and run it off-hours" comments. Either they're going to limit you very, very quickly, or they won't at all. At least, that was my experience when extensively screen-scraping del.icio.us to gather a corpus for a class project; they didn't take kindly to our assault and locked us down pretty fast. So try it, see if it works - it probably will - and be 'nicer' only if you have to be.

Tomorrowful has it. Throttle your requests and do it during offpeak hours. If they are unhappy with your visits, there won't be any emails or conversation; they'll just block your IP. This is one of those instances where "don't ask, don't tell" is actually a good policy. Residents of my house have done this repeatedly with 20K+ record scrapes from a variety of sources and this has consistently worked to achieve, err, the desired end.

I have scraped some very large sites in the past -- and usually do something under a hit a minute. The only time I have been threatened was when I used a non-standard user agent (that had my email address in it). I would discourage doing that.

From the moral PoV, it's pretty clear that they don't want you to grab their content by automated means. You can twist your interpretation of their ToS however you like, but don't kid yourself - you will be connecting using other than a web browser, and you will be making use of the content in a way they will see as being contrary to their copyright. Having said that, your plan sounds OK and more respectful of their server load and bandwidth than they would be if the situation was reversed. Just remember that their concern isn't with the load on their servers, it's with controlling their data. Because of this, I'd fake a real browser ID and not give them your name / contact details. If they catch on and want that info, they can hassle your ISP for it. Depending on your ISP & the relevant laws, it's a layer of protection between you and them. I do a daily screen scrape of around 200~400 pages from a site which is actively and overtly hostile against such activities - javascript randomisation and encryption of all the data on the pages, limiting requests to a handful per day from any one IP address, both the site owner & data supply companies actively pursuing federal court legal action against people even suspected of running scrapers for personal use only, etc. A bit of lateral thinking led to a way that gets around even this level of corporate paranoia and protectionism to grab ~300 pages/hr - but I'm afraid that if I explain it they'll notice, close that loophole, and set their QCs on me...

As a guy who has run a bunch of websites, I suspect that a 30 second delay between requests will lead to absolutely nobody noticing- you'll be below the noise level. However, be aware that if you are only running the script for 12 hours of every day (to be non-peak), and are letting 30 seconds elapse between each page request, you'll take the better part of two weeks in order to get everything. Also, will you be pulling down any referenced graphics as well? If so, you may have more than 20,000 files to pull down, which will make this take longer.

I'd use a non-generic useragent. (e.g. RoboBot 1.0). If they block that, then you know they have a problem with it. And can decide what to do from there (either mask the UA or announce your intentions). Odds are they won't notice. (Incidentally, if you do decide to mask the UA, I suggest using GoogleBot)