Do crawlers only index pages to a certain depth?
-
While doing a content inventory I've reached the sixth level of link depth, and I was wondering if this content will still get indexed.
-
Answer:
The Botify crawler and log analyser helps you to understand how search engines really crawl your website. It also gives you active and indexed pages. You can visit their website : http://www.botify.com
Anonymous at Quora Visit the source
Other answers
When we are analyzing Crawl data vs Logs data (ie how GoogleBot behaves) we see that GG usually stops after reaching depth 5 or 6 when your internal link structure is not optimized. Check this screenshot of Crawled page by Google vs Availble pages on your website by page depth. This graph is a common case : You can check our Log analyzer if you want here: http://www.oncrawl.com/seo-log-analyzer/
Francois Goube
It might, but you will have to provide additional links to those URLs so that search engine crawlers have more incentives to go there and index them. (HTML sitemaps, shrewd internal linking etc.) In general, crawlers do think of the depth level of the page as a certain marker of that page's importance, but there are hundreds of other signs that search engines take into account, so if it is too difficult for you to reconsider your URL structure now, just go for smart HTML sitemaps and internal linking. Here you can find some useful info on this topic http://blog.promodo.com/fast-site-indexing-how-to-speed-up-indexing-of-a-large-content-project and some useful things hereĀ http://blog.promodo.com/seo-for-developers
Anna Korolekh
It has to do with site architecture (internal links) and inbound links from other sites. It's also recommended you don't go more than 3 levels deep.
Mark Nicholson
Related Q & A:
- How do I scroll to a certain widget in a QScrollArea?Best solution by Stack Overflow
- What's a rare mineral, that only occurs in a certain part of the world?Best solution by Yahoo! Answers
- Do they make a certain kind of band-aid for a toe?Best solution by Yahoo! Answers
- Where can I see a video being used by a certain camera?Best solution by Quora
- How do I ask for a certain salary at a job interview?Best solution by Quora
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.