LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

This section provides a tutorial example on how to send a GET request to a Web server, follow up with redirects, and dump request and response messages of all redirects.

Here is the improved LWP::UserAgent test program:

#- LWP-UserAgent-GET-Redirect.pl
#- Copyright (c) 2014 by Dr. Herong Yang, http://www.herongyang.com/
#
   use LWP::UserAgent;

   my ($uri) = @ARGV;
   $uri = 'http://herongyang.com/' unless $uri;
   
   my $browser = new LWP::UserAgent();
   $browser->max_redirect(5);
   $browser->agent("Mozilla/5.0 (Windows NT 6.1)");
   
   my $response = $browser->get($uri);
   my $request = $response->request();
   my @redirects = $response->redirects();

   foreach my $res (@redirects) {
      my $req = $res->request();
      print($req->as_string());
      print($res->as_string());
   }
   print($request->as_string());
   print($response->as_string());

   exit;

3 changes are made on the program:

Now try this new testing the program on facebook.com, I got the following:

C:\herong>LWP-UserAgent-GET-Redirect.pl http://www.facebook.com

GET http://www.facebook.com
User-Agent: Mozilla/5.0 (Windows NT 6.1)

HTTP/1.1 301 Moved Permanently
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: close
Pragma: no-cache
Location: https://www.facebook.com/
Content-Length: 0
Content-Type: text/html; charset=utf-8
...


GET https://www.facebook.com/
User-Agent: Mozilla/5.0 (Windows NT 6.1)

HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: close
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust N...
Client-SSL-Cert-Subject: /C=US/ST=California/L=Palo Alto/O=Faceboo...
Client-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Link: <https://www.facebook.com/>; media="handheld"; rel="alternate"
Link: <https://www.facebook.com/>; hreflang="en"; rel="alternate"
P3P: CP="Facebook does not have a P3P policy. Learn why here: http...
Refresh: 0; URL=/?_fb_noscript=1
Set-Cookie: reg_fb_ref=https%3A%2F%2Fwww.facebook.com%2F; path=/; ...
Title: Welcome to Facebook - Log In, Sign Up or Learn More
X-Content-Type-Options: nosniff
X-FB-Debug: HshLC4adnJI8cXDgh8tGUSyMrbso7p3Wfw1EDU61Swo=
X-Frame-Options: DENY
X-Meta-Charset: utf-8
X-Meta-Description: Facebook is a social utility that connects peo...
X-Meta-Referrer: default
X-Meta-Robots: noodp, noydir
X-XSS-Protection: 0

<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><script>function envFlush(a){functio...
    <link type="text/css" rel="stylesheet" href="https://fbstatic-...
...
<form id="login_form" 
action="https://www.facebook.com/login.php?login_attempt=1"
method="post" onsubmit="return window.Event && Event.__inline...
<input type="text" class="inputtext" name="email" id="email" value...
<input type="password" class="inputtext" name="pass" id="pass" tab...
...

As you can see, facebook.com server is happier now.

If we want to continue to play with the facebook web server, we need use LWP::UserAgent post() method to submit the login form. Continue with the next tutorial.

Last update: 2014.

Table of Contents

 About This Book

 Perl on Linux Systems

 ActivePerl on Windows Systems

 Data Types: Values and Variables

 Expressions, Operations and Simple Statements

 User Defined Subroutines

 Perl Built-in Debugger

 Name Spaces and Perl Module Files

 Symbolic (or Soft) References

 Hard References - Addresses of Memory Objects

 Objects (or References) and Classes (or Packages)

 Typeglob and Importing Identifiers from Other Packages

 String Built-in Functions and Performance

 File Handles and Data Input/Output

 Open Files in Binary Mode

 Open Directories and Read File Names

 File System Functions and Operations

 Converting Perl Script to Executable Binary

 Using DBM Database Files

 Using MySQL Database Server

 Socket Communication Over the Internet

 XML::Simple Module - XML Parser and Generator

 XML Communication Model

 SOAP::Lite - SOAP Server-Client Communication Module

 Perl Programs as IIS Server CGI Scripts

 CGI (Common Gateway Interface)

 XML-RPC - Remote Procedure Call with XML and HTTP

 RPC::XML - Perl Implementation of XML-RPC

 Integrating Perl with Apache Web Server

 CGI.pm Module for Building Web Pages

LWP::UserAgent and Web Site Testing

 What Is LWP::UserAgent?

 What Is HTTP::Response?

 What Is HTTP::Request?

 LWP-UserAgent-GET.pl - Sending a GET Request

LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

 http-equiv="Refresh" Meta Tag not Followed

 LWP-UserAgent-POST.pl - Posting Form Data

 post() Method not Following Redirect Location

 LWP-UserAgent-POST-Redirect.pl - Posting with Redirects

 What Is HTTP::Cookies?

 LWP-UserAgent-Request.pl - GET, POST and Cookies

 LWP-UserAgent-Request.pl - Login to facebook.com

 HTTP::Cookies save() not Saving Temporary Cookies

 LWP-UserAgent-Request-Cookies.pl - Sending Request with Cookies

 References

 PDF Printing Version