Find solution
How do you remove your website from search engines?

How to temporarily remove a website from search engines?

  • We are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. Now we are already being crawled and indexed by bots (Google, Yahoo, and others) and we don't want that for our test version which is not ready and finalized yet. My question is related to indexing: Is it OK if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all? Can we use the 307 HTTP status (Temporary Redirect)? How long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?

  • Answer:

    is it ok if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all? Yes. Once you stop blocking the contents of the website it will be crawled again. But I would resubmit my site to Google and/or submit a XML sitemap just to make sure they know you now have content to index. can we use the 307 http status (Temporary Redirect)? That would work as well. how long doest it take spiders to refresh their db (either disallow robots or 307 redirect)? It varies. Newer sites will find they are crawled less frequently and changes made to the website are picked up slowly. I definitely wouldn't expect it to happen quickly. You'll find in questions asked here that it can take weeks or months to finally see results.

user8849 at Webmasters Visit the source

Was this solution helpful to you?

Other answers

If you don't want the site crawled at all, I'd suggest moving it to a "dummy" subdomain, and be sure to update the rules to NOT crawl it before putting the code there (and ideally ensure there are no links from the "live" site to the "demo" site. e.g. a347demo.example.com

scunliffe

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.