Looking for unique web ad techniques.

What are some Good Books on Programming Advanced Web Bots?

• Note: This question has nothing to do with Botnets or anything of that illegal nature. I'm looking for some good books on programming advanced Web Bots. I've been programming bots for a number of years, but quite recently it's been challenging due to new measures in place to stop Web Bots accessing certain websites. One problem is when sites rely on JavaScript to set cookies, or elements on a page that are required for a web request on that site. I've got around this a number of times by using FireBug, and setting breakpoints in the code to discover various functions used to set these variables. It usually requires de obfuscation of the functions, but once done I can usually port those methods in C# and have my Web Bot produce these variables, hashes etc. That's one type of method that's being used on larger websites. I'm looking for books that go into this level of detail, but on other protection methods and advanced techniques for coding Web Bots. From what I've found there don't seem to be many. Data Mining is also interesting, and ties in with the Web Bot's although my knowledge of Data Mining isn't too great. I know a lot of the top Data Mining Developers use a lot of statistics and analysis. Something I've not been exposed to. But any C# related books on this topic would be excellent.

• Answer:

  If you want to crawl webpages you sort of have to write your own crawler. Or you can use something like eg Kapow to set up robots that do the crawling. This doesn't answer the question but I doubt you could find books on this subject.