Should a digital marketer learn how to web crawl using Python? If yes, should he learn SQL next? If no, is there a better alternative or a reason no to pursue learning web crawling, Python, or SQL?
-
I always believed and knew about the superb combination of programming and marketing. I also believed that learning how to web crawl will help a lot in terms of effectiveness and efficiency of digital marketing campaigns. I have average coding skills in Python 2.7 and I'm planning to build a web crawler on my own and I plan to learn and use SQL next. Am I doing it right? Do you have any suggestions?
-
Answer:
Great question. I've once implemented a crawler that finds the "top" engineers from LinkedIn by using a primitive PageRank algorithm. Feel free to check it: http://github.com/alioral/shark
Ali Oral at Quora Visit the source
Other answers
I never developed a web crawler (yet) so I can't help on that. But on the SQL side: if you already know the technology fine, otherwise if you have some knowledge of javascript you may want to try http://www.mongodb.org/ since it's easier to learn (for javascript developers). It has a Python library as well of course (http://docs.mongodb.org/ecosystem/drivers/python/).
Francesco Alisetta
I am always of the opinion that if one can, one SHOULD learn how to accomplish seminal tasks ones self. Then you become a MASTER of your computer army. And Python and SQL are powerful tools well worth mastering. Web crawling is also a very worthwhile talent to pursue. So I absolutely can't fault you for investigating these tools, and I applaud your curiosity and determination. That having been said, there are easier ways to accomplish your information seeking goals which will not require you to learn Python and SQL. For a quick and dirty solution which will accomplish your marketing requirements (though they will not give you the power and control and ancillary benefits of leaning Python and SQL), take a llok at: Yahoo Pipes Scrapy Mechanize BeautifulSoup Twill
Chris Kemp
Try Udacity "Intro to computer science", the first 1-3 units are dedicated on how to build a crawler using Python. I am on mobile otherwise would have posted link to the course.
Parikshit Joshi
If a significant amount of your work involves web crawling, and you have some familiarity with Python already, I would recommend using a framework like Scrapy [1] to scrape web pages. As for learning SQL, I don't think it's necessary in this case. Scrapy allows you to save data in a number of common formats including CSV and JSON without any additional code, and I can't think of any common scenarios where you would need to use a database if you aren't accessing the data programmatically. Alternatively, if you want to avoid writing code, you should look into using an annotation based scraping tool such as Portia [2]. Portia allows you to visually create templates of pages you want to extract by clicking on elements on the page and mapping them to fields. Portia will take the templates you created and will generate a spider to crawl similar pages on the website. See the video below for an idea of how Portia works. Scrapy Cloud [3] integrates Portia into its platform, so you can run the spiders you create from the dashboard and download the results when the crawling job is finished. Here's a post demonstrating how Portia works with Scrapy Cloud: http://blog.scrapinghub.com/2015/04/07/scrape-data-visually-with-portia-and-scrapy-cloud/ There are also other services such as importio [4] and Kimono [5] which let you scrape websites through a clean user interface. You can see some examples of how Kimono is used here: http://builtwith.kimonolabs.com/ [1] http://scrapy.org [2] https://github.com/scrapinghub/portia [3] http://scrapinghub.com/platform [4] http://import.io [5] https://www.kimonolabs.com Disclaimer: I work for Scrapinghub.
Richard Dowinton
Related Q & A:
- How do I create a digital signature and how do I use it?Best solution by support.office.com
- Is there a better alternative to the Flip video camcorder?Best solution by Ask.Metafilter.Com
- I was thinking of buying a web cam, but I was woundering if you can use a digital camera instead of a web cam.Best solution by Yahoo! Answers
- What is a close alternative to a pocket bike?Best solution by Yahoo! Answers
- How to get a better package from a job offer?Best solution by Quora
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.