How to crawl same url using Scrapy?

Does Google crawl http://goo.gl shortened URLs?

• We are using Google's own url shortening service. All the target URLs which are shortened are publicly available but I don't want Google to index them. I understand that robots.txt is going to prevent indexing of such documents if I choose to. I was wondering, if Google starts crawling target URLs too. If Google is crawling, I don't mind it. I don't want them to appear in search results. Any ideas?

• Answer:

  No, shortened URL's will not appear in the search results. The content will. Google won't necessarily index your content, but using Google's URL shortener will ping the content, likely resulting in a crawl. If you setup the page correctly, you can almost definitely keep the page from being indexed. As a quick note of clarification: Google indexes content based on other pages linking to that content. Google crawls through the web and looks for links that have never been seen before. It is my understanding that if a URL shortened by Google has never been seen before, it will behave similarly to a new link. It doesn't add any SEO value, but Google bot will probably visit your site, simply because it is new content. I sometimes get my pages indexed faster by sharing them on Google+, because I know it pings the content, resulting in a crawl.