How to prevent page redirect?

perl: fetching Web pages / redirect issue

• Hi - The script below fetches Web page content from specified URLs. I'd like to have code added to the grab function so that in case of a redirect I'll have a way of knowing what URL is being redirected to, by looking at the contents of @endurl. In the example below, for instance, "http://www.news.com" redirects to http://news.com.com", therefore $endurl[0] should be "http://news.com.com". $endurl[1] should "http://www.yahoo.com" since there is no redirect for that address. As an answer to this question, please add the necessary code to the grab function and post the entire modified, tested script. Thank you. Marc. #!/usr/bin/perl use LWP::Simple; require LWP::Parallel::UserAgent; @urls = ( "http://www.news.com", "http://www.yahoo.com" ); @content = &grab(@urls); # add code to the grab function below so that @endurl is filled with actual URLs that content is being fetched from. # in cases where there is no redirect, those will be the same as the $url[n]. # In this example, $endurl[0] should be "http://news.com.com" ( since this is # what www.news.com redirects to) and $endurl[1] should be "http://www.news.com" print "$endurl[0] \n"; print $content[0]; exit; sub grab { @results= (); $ua = LWP::Parallel::UserAgent->new(); $ua->agent("MS Internet Explorer"); $ua->timeout ($timeout); $ua->redirect (1); # prevents automatic following of redirects $ua->max_hosts(6); # sets maximum number of locations accessed in parallel $ua->max_req (2); # sets maximum number of parallel requests per host foreach $url2 (@_) { $ua->register(HTTP::Request->new(GET => $url2), \&callback); } $ua->wait (); return @results; } sub callback { my($data, $response, $protocol) = @_; #Comment this line to prevent show the url print $response->base."\n"; #majortom-ga's change my $oresponse = $response; while (defined($oresponse->previous)) { $oresponse = $oresponse->previous; } for ($i=0; $i<@urls; $i++) { # check $oresponse->base, not $response->base if ( index( $oresponse->base, $urls[$i]) != -1 ) { $results[$i].=$data; last; } } } ############

• Answer:

  Hello, marcfest-ga, It's a pleasure to hear from you again. I have made the change you asked for, and also made an improvement to the callback: instead of searching for $urls[$i] in $oresponse->base, which just happens to work if there is no <base> tag or the <base> tag happens to point to a deeper URL in the same web site, I now compare $urls[$i] directly to $oresponse->request->uri, which is the document actually asked for on this particular request, although it may not be the *original* document asked for due to redirects and we still have to deal with that as we always have by walking the list of responses. Here is the code with the new @endurl feature. Let me know if you have any questions! Thanks for the opportunity to work on this interesting program. Sources of information: "perldoc HTTP::Request", "perldoc HTTP::Response" * * * CUT HERE * * * #!/usr/bin/perl use LWP::Simple; require LWP::Parallel::UserAgent; @urls = ( "http://www.news.com", "http://www.yahoo.com" ); @content = &grab(@urls); # add code to the grab function below so that @endurl is filled with # actual URLs that content is being fetched from. # in cases where there is no redirect, those will be the same as the $url[n]. # In this example, $endurl[0] should be "http://news.com.com" ( since this is # what www.news.com redirects to) and $endurl[1] should be "http://www.news.com" print "ENDURLS ARE:\n"; for $u (@endurl) { print $u, "\n"; } print "END OF ENDURLS\n"; print $content[0]; exit; sub grab { @results = (); @endurl = (); $ua = LWP::Parallel::UserAgent->new(); $ua->agent("MS Internet Explorer"); $ua->timeout ($timeout); $ua->redirect (1); # ALLOWS automatic following of redirects $ua->max_hosts(6); # sets maximum number of locations accessed in parallel $ua->max_req (2); # sets maximum number of parallel requests per host foreach $url2 (@_) { $ua->register(HTTP::Request->new(GET => $url2), \&callback); } $ua->wait (); return @results; } sub callback { my($data, $response, $protocol) = @_; #Comment this line to prevent show the url print "URL: ", $response->base, "\n"; #majortom-ga's change my $oresponse = $response; while (defined($oresponse->previous)) { $oresponse = $oresponse->previous; } for ($i=0; $i<@urls; $i++) { # majortom-ga: instead of trying to find $urls[$i] in # $oresponse->base, which won't necessarily contain it # if a Content-Base: header or <base> tag is present, # we match it exactly in $oresponse->request->uri if ($oresponse->request->uri eq $urls[$i]) { #majortom-ga: record the URL of the last page #we were redirected to $endurl[$i] = $response->request->uri; $results[$i] .= $data; last; } } }