How to write code to go through web pages automatically and get the required information?

How do I write a code to go through web pages like a human automatically and get the required information?

  • I am new to programming and I want to write a code to go through different web pages automatically by providing the login credentials and fetch the data from the required page.

  • Answer:

    Basically you want to create a crawler / scraper to gathering informations from web pages. To compleate this goal you need to examiner the page DOM and evaluate if you need also JavaScript execution to retrieve all the required data. There are various methods to extract data from DOM, I suggest to use XPath with XQuery. For programming language comparison and a working example, have a look on http://programmers.stackexchange.com/a/280686/102576.

rudiecig700 at Stack Overflow Visit the source

Was this solution helpful to you?

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.