What are some tools that can help me get the HTML content of a page making heavy use of AJAX?
-
I'm working to create a web crawler in my internship. But I have found that some websites use a lot of AJAX and their HTML source code is not accessible (source code of pages). What are some tools that can help me make the source code readable and get the HTML rendered?
-
Answer:
What you are looking for is called a head-less browser. Try http://casperjs.org or http://phantomjs.org. But before to use such tools, just disable javascript in your web browser and try going to these websites you want to crawl. They may redirect you to non-js pages with the content rendered in the backend. Then you would just have to use these non-js URL.
Julien Buty at Quora Visit the source
Other answers
Julien provides good advice. I would add Selenium to the list. It is not headless, but it is very easy to use, has good documentation, and a lot of discussion on stackoverflow. It was created by Google to assist with testing.
Neil Aggarwal
Related Q & A:
- How to get all the content of a page?Best solution by Stack Overflow
- How do I display limited html content in a webview?Best solution by Stack Overflow
- Can you help me get some design ideas for a mask?Best solution by Yahoo! Answers
- What is the best website to help me get work abroad?
- Can my facebook friends see if i create a page?Best solution by Yahoo! Answers
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.