How to return the response from an asynchronous function?

perl function improvement

• Please look at the perl script below. It uses a function called "grab" to fetch URLs of the Web. I need the function's timeout feature fixed. Right now, when including a non existent server like "http://216.239.39.111/" (which is a bogus url to simulate a server that's down; it should trigger the timeout) in @urls, the script will hang for up to 3 minutes instead of timing out after 3 seconds. This may be due to a bug in the PUA Perl module. Please make it so the "grab" function times out appropriately, i.e. in this case after 3 seconds. You may have to stop using PUA to accomplish this. Make it so that when a request times out, the value "timeout" is assigned to the variable that would otherwise hold the fetched html. Please try to keep grab working as fast as possible. Thank you. Marc. #!/usr/bin/perl #Uncomment to get full debug info #use LWP::Debug qw(+ -conns); use LWP::Simple; require LWP::Parallel::UserAgent; require HTTP::Request; @urls = ( "http://www.yahoo.com/", "http://216.239.39.111/", ); $timeout = 3; # each request times out after 3 seconds) @content = grab(@urls); #This prints the contents of http://www.cnn.com print $content[0]; exit; sub grab { @results; $ua = LWP::Parallel::UserAgent->new(); $ua->agent("MS Internet Explorer"); $ua->redirect (0); # prevents automatic following of redirects $ua->max_hosts(6); # sets maximum number of locations accessed in parallel $ua->max_req (6); # sets maximum number of parallel requests per host foreach $url (@_) { $ua->register(HTTP::Request->new(GET => $url), \&callback); } $ua->wait ( $timeout ); return @results; } sub callback { my($data, $response, $protocol) = @_; #Comment this line to prevent show the url print $response->base."\n"; for ($i=0; $i<@urls; $i++) { if ( index( $response->base, $urls[$i]) != -1 ) { $results[$i].=$data; last; } } }

• Answer:

  Good morning marcfest, You've made a small error in using $ua->wait in your script. Since the call to wait occurs after your foreach loop, it does not impact the execution of any code within that loop. Instead, you have to use $ua->timeout() to set the timeout value while you're setting up the other constants governing the behavior of LWP::Parallel in your script: sub grab { @results; $ua = LWP::Parallel::UserAgent->new(); $ua->agent("MS Internet Explorer"); $ua->timeout ($timeout); # <--- ADD THIS $ua->redirect (0); # prevents automatic following of redirects $ua->max_hosts(6); # sets maximum number of locations accessed in parallel $ua->max_req (6); # sets maximum number of parallel requests per host foreach $url (@_) { $res = $ua->register(HTTP::Request->new('GET', $url), \$callback); } $ua->wait (); # <--- simply wait until all registered URLs are dealt with return @results; } LINK: http://search.cpan.org/~marclang/ParallelUserAgent-2.56/lib/LWP/Parallel.pm The CPAN page on LWP::Parallel was invaluable in answering this question for you. It has several examples that you may find useful in continuing to work with this script. -Haversian