Find solution
What are some good tools for interactive web scraping and automation?

What are some good tools for interactive web scraping and automation?

  • Tools for automating or scripting web app access, including rich JavaScript apps. Commercial or open source. Clarification Looks like the answers tools are for classic "scraping" or parsing websites read-only. I'm looking for a solution for those cases where the user needs to load forms or even run JavaScript client-side functions.

  • Answer:

    Another python option is scrapy From http://scrapy.org/ What is Scrapy? Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.FeaturesSimpleScrapy was designed with simplicity in mind, by providing the features you need without getting in your wayProductiveJust write the rules to extract the data from web pages and let Scrapy crawl the entire web site for youFastScrapy is used in production crawlers to completely scrape more than 500 retailer sites daily, all in one serverExtensibleScrapy was designed with extensibility in mind and so it provides several mechanisms to plug new code without having to touch the framework corePortable, open-source, 100% PythonScrapy is completely written in Python and runs on Linux, Windows, Mac and BSDBatteries includedScrapy comes with lots of functionality built in. Check this section of the documentation for a list of them.Well-documented & well-testedScrapy is extensively documented and has an comprehensive test suite with very good code coverage [...]

Brooks Cutter at Quora Visit the source

Was this solution helpful to you?

Other answers

Ruby and Ruby gem Nokogiri

Paul Denlinger

BeautifulSoup is a Python library for HTML and XML parsing. Great for extracting data from web!

Samarth Bansal

I tried out WebQL once, I don't think they have a free version though.

Konrad Szpirak

For windows, tried HTMLAgility Pack, open source. Works well.

Earljon Hidalgo

The ONE that works well, Open Source, Free, and always kept updated! Anyone can download it and use it at no charge. It is called Easy Data Feed http://www.easydatafeed.com/        Development                   ·           Open Source             ·           Sample Projects             ·           Visual Studio Support             ·           Programming Language             ·           Customize Core             ·           Native API Development                    Scraping             ·           Run Daily Schedules             ·           Set Scrape Limits             ·           Visit as Anonymous             ·           Set Thread and Speed             ·           Password Protected Scrape             ·           Solve Decaptcha             ·           Use Proxies                    Export             ·           Upload to FTP             ·           Save as CSV             ·           Split File             ·           Zip File             ·           Force Save             ·           Import/Export Profiles             ·           EDI Integration             ·           CRM Integration             ·           Shopping Cart Integration                    Data Manipulation             ·           Multiple Profiles             ·           Add Custom Values             ·           Limit Length             ·           Strip HTML             ·           Convert Measurements             ·           Change Case             ·           Custom Profiles             ·           Find/Replace             ·           Custom Merge Rules             ·           Custom Clean Rules             ·           Custom Remapping Rules                    Images             ·           Unlimited Free Image Downloads             ·           Download Images Locally             ·           Unlimited File Storage             ·           PDF to JPG             ·           GIF to JPG             ·           Apply Watermark             ·           Transfer JPG from Body to FTP     You can read about how to use it here: http://www.easydatafeed.com/open-source/ They also have developers you can hire to do the job for you, their skype is “easydatafeed”

Sammir Diesuza

For scraping, There are hundreds of  Web Scraping tools today available for both commercial and personal use. If you’ve never done any web scraping before, there is one basic tool is available which is more flexible and has extra functionality. It is user-friendly tool and easy to perform. I have one best free open source solution for you. I think you should have to use it because it is a freeware and you can easily download it and feel the difference compared to other scraping tools. It is called as “Easy Data Feed” and it is available on http://www.easydatafeed.com Here is the list of its features: • Upload to FTP • Save as CSV • Split File • Zip File • Force Save • Import/Export Profiles • EDI Integration • CRM Integration • Shopping Cart Integration • Inventory Management You can read about how to use it here: http://www.easydatafeed.com/open-source/ They also have developers, you can hire them to do the job for you, and their Skype is “easydatafeed”

Mike John

Check out ParseHub - http://www.parsehub.com. It's a visual web scraper built to handle rich Javascript apps and to get data from forms, search, drop downs and even maps. I'm one of the founders of ParseHub and would love to get feedback from anyone that tries our tool!

Angelina Fomina

You will find many web scraping and automation services that brings you information from your targeted websites. I had tried different scraper tools such as ScraperWorld's LinkedIn scraper, Twitter scraper, Business directory extractor, Amazon product scraper etc. They all worked pretty well for me and provide accurate results, it save all extracted information at my database for further analysis and use. It arrange data in a csv file format to make it easy to use and analysis. These tools are user friendly and interactive..

Erma J. Deaton

Ho Thanh Son

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.