How do I retrieve a URL in Java?

Querying Google from Java program (URL, openStream())

• I try to send a query directly to Google with java program and receive the results(not using Google API). Here is the (part of) java code that I was created for the purpose. String urlName = "://www.google.com/search?q=java"; URL url = new URL(urlName); BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream())); String line; while ((line = in.readLine()) != null){ System.out.println(line); } The problem is that the above openStream call throws a FileNotFoundException even though the url (://www.google.com/search?q=java) is a valid one (i.e. a Browser will show the results if I put the url into its address field). FYI, I have tried to use the openConnection method of the URL class but it was also failed. Is there another way to succesfully send the query to Google and retrieve the answer using java program (without using the Google API)?

• Answer:

  Hi mrz I strongly suspect that there is nothing wrong with your code; rather it is Google's policy on robots and automated requests that is causing you to have problems. If you review the Google Terms & Conditions at ://www.google.com/terms_of_service.html you will find the following clause: "No Automated Querying "You may not send automated queries of any sort to Google's system without express permission in advance from Google." What's more, Google seems to take active steps to enforce this rule. In particular they seem to check that the User-Agent HTTP header does not contain any of the well-known software automation tools. For an anecdotal tale of how readily Google will deny access to robots you could look at: http://www.everything2.com/index.pl?node_id=1229892 Of course, there are a number of ways to try to get round this policy, but as you would still be in breach of Google's T&C, and as Google's programmers have probably thought of most of them anyway, I'm afraid I can't list any of them here. Sorry to be the bearer of bad tidings, and I wish you luck in the rest of your development project. Regards iaint-ga Search strategy: Visited ://www.google.com/ and clicked around to find the Terms & Conditions page!