LWP-UserAgent-GET.pl - Sending a GET Request

This section provides a tutorial example on how to send a GET request to a Web server and dump both request and response messages.

We have learned 3 classes, LWP::UserAgent, HTTP::Request, and HTTP::Response in previous tutorials, with some simple tests. Now let's try to write more real Perl program to send a GET request to any given URI and dump both request and response.

Here is my first version of LWP-UserAgent-GET.pl:

#- LWP-UserAgent-GET.pl
#- Copyright (c) 2014 by Dr. Herong Yang, http://www.herongyang.com/
#
   use LWP::UserAgent;
   
   my ($uri) = @ARGV;
   $uri = 'http://herongyang.com/' unless $uri;
   
   my $browser = new LWP::UserAgent();
   my $response = $browser->get($uri);
   my $request = $response->request();
   
   print($request->as_string());
   print($response->as_string());

   exit;

Test 1: When testing the program on google.com, I got the following:

C:\herong>LWP-UserAgent-GET.pl http://www.google.com

GET http://www.google.com
User-Agent: libwww-perl/6.05

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Connection: close
Server: gws
Content-Type: text/html; charset=ISO-8859-1
Expires: -1
Alternate-Protocol: 80:quic
Client-Response-Num: 1
P3P: CP="This is not a P3P policy! See http://www.google.com/suppo...
Set-Cookie: PREF=ID=fb56a6bf2677de5a:FF=0:TM=1393607257:LM=1393607...
Set-Cookie: NID=67=iolNVOo3ULZAHWxwv3P-LQhF5GRZvNQhgZyEedVVHOzGQzV...
Title: Google
X-Frame-Options: SAMEORIGIN
X-Meta-Description: Search the world's information, including webp...
X-Meta-Robots: noodp
X-XSS-Protection: 1; mode=block

<!doctype html><html itemscope="" itemtype="http://schema.org/WebP...
...

LWP-UserAgent-GET.pl worked as expected. Both request and response are printed out.

Test 2: When testing the program on facebook.com, I got the following:

C:\herong>LWP-UserAgent-GET.pl http://www.facebook.com

GET https://www.facebook.com/unsupportedbrowser
User-Agent: libwww-perl/6.05

HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: close
Content-Type: text/html; charset=utf-8
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust N...
Client-SSL-Cert-Subject: /C=US/ST=California/L=Palo Alto/O=Faceboo...
Client-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Link: <https://www.facebook.com/unsupportedbrowser>; media="handhe...
Link: <https://fbstatic-a.akamaihd.net/rsrc.php/yV/r/hzMapiNYYpW.i...
P3P: CP="Facebook does not have a P3P policy. Learn why here: http...
Refresh: 0; URL=/unsupportedbrowser?_fb_noscript=1
Set-Cookie: reg_fb_ref=https%3A%2F%2Fwww.facebook.com%2Funsupporte...
Title: Update Your Browser | Facebook
X-Content-Type-Options: nosniff
X-FB-Debug: zKaAOvfvJwkkb2nP7VcGQwahBGkRIZ1frxAdPRL6Ngs=
X-Frame-Options: DENY
X-Meta-Charset: utf-8
X-Meta-Referrer: default
X-Meta-Robots: noodp, noydir
X-XSS-Protection: 0
... 

<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><script>function envFlush(a){functio...

It worked too. But there were 2 surprises in the output:

With this information, I can improve my LWP-UserAgent-GET.pl program to make facebook.com happy now, see the next tutorial.

Last update: 2014.

Table of Contents

 About This Book

 Perl on Linux Systems

 ActivePerl on Windows Systems

 Data Types: Values and Variables

 Expressions, Operations and Simple Statements

 User Defined Subroutines

 Perl Built-in Debugger

 Name Spaces and Perl Module Files

 Symbolic (or Soft) References

 Hard References - Addresses of Memory Objects

 Objects (or References) and Classes (or Packages)

 Typeglob and Importing Identifiers from Other Packages

 String Built-in Functions and Performance

 File Handles and Data Input/Output

 Open Files in Binary Mode

 Open Directories and Read File Names

 File System Functions and Operations

 Converting Perl Script to Executable Binary

 Using DBM Database Files

 Using MySQL Database Server

 Socket Communication Over the Internet

 XML::Simple Module - XML Parser and Generator

 XML Communication Model

 SOAP::Lite - SOAP Server-Client Communication Module

 Perl Programs as IIS Server CGI Scripts

 CGI (Common Gateway Interface)

 XML-RPC - Remote Procedure Call with XML and HTTP

 RPC::XML - Perl Implementation of XML-RPC

 Integrating Perl with Apache Web Server

 CGI.pm Module for Building Web Pages

LWP::UserAgent and Web Site Testing

 What Is LWP::UserAgent?

 What Is HTTP::Response?

 What Is HTTP::Request?

LWP-UserAgent-GET.pl - Sending a GET Request

 LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

 http-equiv="Refresh" Meta Tag not Followed

 LWP-UserAgent-POST.pl - Posting Form Data

 post() Method not Following Redirect Location

 LWP-UserAgent-POST-Redirect.pl - Posting with Redirects

 What Is HTTP::Cookies?

 LWP-UserAgent-Request.pl - GET, POST and Cookies

 LWP-UserAgent-Request.pl - Login to facebook.com

 HTTP::Cookies save() not Saving Temporary Cookies

 LWP-UserAgent-Request-Cookies.pl - Sending Request with Cookies

 References

 PDF Printing Version