How do I purge my search results?

Why should I exclude Internal Search results in the robots.txt?

• I have long heard that it is a best practice to exclude your internal search results within the robots.txt file of your website. This is verified by Matt Cutts who claims Google doesn't want search results within their search results (http://www.mattcutts.com/blog/search-results-in-search-results/). But how would Google obtain a link to my internal search results? There are only two options, Internal or External. Either "Google has begun entering search phrases into search forms" as this post from SEObook suggests (http://tools.seobook.com/robots-txt/) or Google is specifically speaking of external links which point to search results on your website. But there are two problems. 1. Why would Google enter search phrases into an Internal Search and index the results if they don't want them indexed in the first place? 2. If there are external links pointing to internal search results how would excluding them in the robots.txt keep them out of the index, since Google will still show URLs that are linked to in their index.

• Answer:

  I love this question. I'm going to speculate, but I feel fairly confident in my speculation. I highly doubt that Google is entering queries in site search bars. Why would they do that if they don't want search queries indexed? I believe Google is crawling pages where visitors go. We know that they collect user data "to make Google search better." Wouldn't part of that be indexing pages that users go to? If many users end up on a page that Google doesn't have indexed, why wouldn't they want to index it? My supporting evidence comes from several sites that have their queries indexed. See, for example, this search on DMOZ: http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=site%3Admoz.org+search%3Fq%3D&pws=0&changed_loc=1&bav=on.2%2Cor.r_gc.r_pw.&biw=861&bih=730&emsg=NCSR&noj=1&ei=4pxBTrrWDaOLsgLgrtzYCQ I can't find any links to most of these DMOZ SERPs - even on the few pages where my SEO tools found links, I'm not seeing them on the supposed (internal) linking page. The only answer that accounts for the 250k+ results returned for the example query I linked to is that Google indexes pages that users find. I suspect that they're more likely to index previously unknown pages from a high authority site.