LWP-UserAgent-Request-Cookies.pl - Sending Request with Cookies

This section provides a tutorial example on how to convert all cookies into persistent cookies, save them to a cookie jar file, and attach them to subsequent requests.

In the previous tutorial, we learned that the save() method in the HTTP::Cookies class does not save any temporary cookies to the cookie jar file. One workaround option is convert all temporary cookies into persistent cookies in the client script.

Here is my enhanced version of the script: LWP-UserAgent-Request-Cookies.pl:

#- LWP-UserAgent-Request-Cookies.pl
#- Copyright (c) 2014 by Dr. Herong Yang, http://www.herongyang.com/

   use LWP::UserAgent;
   use HTTP::Cookies;

   my ($method, $uri, $data) = @ARGV;
   $method = 'GET' unless $method;
   $uri = 'http://herongyang.com/' unless $uri;
   $data = 'author=Herong&lang=Perl' unless $data;

   my $browser = new LWP::UserAgent();
   $browser->requests_redirectable(['GET', 'HEAD', 'POST']);
   $browser->max_redirect(5);
   $browser->agent("Mozilla/5.0 (Windows NT 6.1)");

   my $request = HTTP::Request->new();
   $request->method($method);
   $request->uri($uri);
   if (uc($method) eq 'POST') {
      $request->header('Content-Type' 
         => 'application/x-www-form-urlencoded');
      $request->content($data);
   }

   my $jar = HTTP::Cookies->new();
   $jar->load('HTTP-Cookies.jar');
   $jar->add_cookie_header($request);

   my $response = $browser->request($request);
   $request = $response->request();
   $jar->extract_cookies($response);

#- Convert cookies from old jar to new jar
#  print("List of cookies in the old jar:\n");
#  $jar->scan(\&listCookie);
   my $jarNew = HTTP::Cookies->new();
   $jar->scan(\&buildNewJar);
   $jarNew->save('HTTP-Cookies.jar');
#  print("List of cookies in the new jar:\n");
#  $jarNew->scan(\&listCookie);
   
   my @redirects = $response->redirects();
   foreach my $res (@redirects) {
      my $req = $res->request();
      print($req->as_string());
      print($res->as_string());
   }
   print($request->as_string());
   print($response->as_string());

   exit;

#- Convert all cookies to persistent cookies
sub buildNewJar {
   my ($version, $key, $val, $path, $domain, $port, 
      $path_spec, $secure, $expires, $discard) = @_;
   $maxage = 60*60*24*30; # lives for 30 days
   $discard = 0; # do not discard
   $jarNew->set_cookie($version, $key, $val, $path, $domain, $port,
      $path_spec, $secure, $maxage, $discard);
}

#- List each cookie in the jar
sub listCookie {
   my ($version, $key, $val, $path, $domain, $port, 
      $path, $secure, $expires, $discard, $hash) = @_;
   print("key = ($key)\n");
   print("val = ($val)\n");
   print("domain = ($domain)\n");
   print("path = ($path)\n");
   print("expires = ($expires)\n");
   print("discard = ($discard)\n");
}

#- Maintain cookies during redirect  
sub LWP::UserAgent::redirect_ok {
   my ($self, $req, $res) = @_;
   $jar->extract_cookies($res);
   $jar->add_cookie_header($req);
   return true;
}

Note that the enhancement uses the scan() method in the HTTP:Cookies class to loop through each cookie in the jar and convert all cookies into persistent cookies by setting $maxage = 60*60*24*30 and $discard = 0.

If we repeat the same test with LWP-UserAgent-Request-Cookies.pl, we should get:

C:\herong>LWP-UserAgent-Request-Cookies.pl 
   GET http://localhost/CGI-pm-Manage-Cookies.pl

GET http://localhost/CGI-pm-Manage-Cookies.pl
User-Agent: Mozilla/5.0 (Windows NT 6.1)

HTTP/1.1 200 OK
Connection: close
Server: Apache/2.2.25 (Win32)
Content-Type: text/html; charset=ISO-8859-1
Client-Transfer-Encoding: chunked
Set-Cookie: Chocolate_1=Another%20chocolate%20cookie%20for%20you%2...
Set-Cookie: Sugar_2=Another%20sugar%20cookie%20for%20you%21; path=/
Title: CGI-pm-Manage-Cookies.pl

...


C:\herong>type HTTP-Cookies.jar
#LWP-Cookies-1.0
Set-Cookie3: Chocolate_1=Another%20chocolate%20cookie%20for%20you%...
Set-Cookie3: Sugar_2=Another%20sugar%20cookie%20for%20you%21; path...
Set-Cookie3: datr=df0dU_W0r6vNIWj_NgpYQMJ7; path="/"; domain=.face...
Set-Cookie3: fr=0oswe6EQT4vYQsVEs.AWXgFm5hU3wMtefwkipG9OoNw3A.BTEl...
Set-Cookie3: lu=SgFEbQxflkf9cYQqNmEjSYmw; path="/"; domain=.facebo...
Set-Cookie3: NID="67=v4hLLg1a_TYSA0jgDZmyiEFmp2r3qWujpGhCRAx9by8hS...
Set-Cookie3: PREF="ID=39f3c35845eeda94:U=5282f19f1a13190a:FF=0:TM=...


C:\herong>LWP-UserAgent-Request-Cookies.pl 
   GET http://localhost/CGI-pm-Manage-Cookies.pl

GET http://localhost/cgi-pm-manage-cookies.pl
User-Agent: Mozilla/5.0 (Windows NT 6.1)
Cookie: Chocolate_1=Another%20chocolate%20cookie%20for%20you%21; 
   Sugar_2=Another%20sugar%20cookie%20for%20you%21
Cookie2: $Version="1"

The output shows that my enhanced script works correctly. All cookies are persisted into the HTTP-Cookies.jar file. So when you send a subsequent to the same Web site, cookies belongs to that site will be attached to the request.

Last update: 2014.

Table of Contents

 About This Book

 Perl on Linux Systems

 ActivePerl on Windows Systems

 Data Types: Values and Variables

 Expressions, Operations and Simple Statements

 User Defined Subroutines

 Perl Built-in Debugger

 Name Spaces and Perl Module Files

 Symbolic (or Soft) References

 Hard References - Addresses of Memory Objects

 Objects (or References) and Classes (or Packages)

 Typeglob and Importing Identifiers from Other Packages

 String Built-in Functions and Performance

 File Handles and Data Input/Output

 Open Files in Binary Mode

 Open Directories and Read File Names

 File System Functions and Operations

 Converting Perl Script to Executable Binary

 Using DBM Database Files

 Using MySQL Database Server

 Socket Communication Over the Internet

 XML::Simple Module - XML Parser and Generator

 XML Communication Model

 SOAP::Lite - SOAP Server-Client Communication Module

 Perl Programs as IIS Server CGI Scripts

 CGI (Common Gateway Interface)

 XML-RPC - Remote Procedure Call with XML and HTTP

 RPC::XML - Perl Implementation of XML-RPC

 Integrating Perl with Apache Web Server

 CGI.pm Module for Building Web Pages

LWP::UserAgent and Web Site Testing

 What Is LWP::UserAgent?

 What Is HTTP::Response?

 What Is HTTP::Request?

 LWP-UserAgent-GET.pl - Sending a GET Request

 LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

 http-equiv="Refresh" Meta Tag not Followed

 LWP-UserAgent-POST.pl - Posting Form Data

 post() Method not Following Redirect Location

 LWP-UserAgent-POST-Redirect.pl - Posting with Redirects

 What Is HTTP::Cookies?

 LWP-UserAgent-Request.pl - GET, POST and Cookies

 LWP-UserAgent-Request.pl - Login to facebook.com

 HTTP::Cookies save() not Saving Temporary Cookies

LWP-UserAgent-Request-Cookies.pl - Sending Request with Cookies

 References

 PDF Printing Version