Use Perl to access RESTful APIs

A visualization of the interconnectivity of routes across the global Internet
A visualization of the interconnectivity of routes across the global Internet | Source
Welcome to another Perl tutorial!
Welcome to another Perl tutorial! | Source

REST is representation state transfer

Welcome to the Information Age. These past few decades have seen generations of APIs (application programming interfaces) come and go.

This tutorial will introduce you to the concepts necessary to build your own Perl scripts to interact with online services that publish a RESTful interface. Although the focus of this document is to lead you through building your own client interface, many popular APIs have their own libraries implemented in Perl.

What is an API?

An API maps out the interoperability between software components. Generally, the API publisher is the established party, and the API consumer is the developer of some new idea. For example, to create a new application for Windows, a developer would need to interface with any number of Windows APIs. The same goes for a Mac developer with Mac OS APIs. What about mobile devices? Oy vey. That's quite a long list of APIs to learn.

In today's world of hyper-connectivity, users move between mobile and desktop environments many times a day. Independent developers are hard-pressed to learn each of the APIs required for Windows, Mac, iDevice, Android, etc. Since almost any given device is online, more developers choose web-based APIs for their applications.

Give it a REST

REST (representation state transfer) had its start in 2000 as a chapter in a PhD dissertation, but has since taken over the web as one of the dominant HTTP APIs of choice. Any number of recognizable brands make their services available via a RESTful API:

  • Amazon (Perl library Net::Amazon)
  • Google Glass (Perl example of OAuth)
  • Twitter (Perl library Net::Twitter)
  • Flickr (Perl library Flickr::API2)
  • Atom publishing protocol (Perl library XML::Atom::Feed)

Transport and Data

Compared to other online APIs, REST is a simple extension of HTTP, taking advantage of a rich set of existing technologies to provide transport between server and client. Furthermore, REST does not require a specific data format, so the content can be XML, JSON, HTML, etc. The most common formats are XML and JSON, but we will focus on XML below.

A RESTful scene from a fjord in Norway
A RESTful scene from a fjord in Norway | Source

Perl modules for HTTP, XML

Perl began in 1987 as a Unix utility for parsing through text files and tabulating summary reports. Through the years it has gained functionality through built-in features and add-on modules. Over 100,000 modules are available through CPAN (comprehensive Perl archive network). The same name is also a binary included with Perl implementations, cpan. My personal preference is to install cpanminus (binary name cpanm) due to its added feature of automatically installing dependencies.

Before continuing on to the tutorial, take some time to verify that cpan or cpanminus is functional on your system. (Pick one - you don't need both.)

  • Install cpan
  • Install cpanminus

With your module installer of choice, install the following modules that will be used in this tutorial. For cpan, use the command cpan install <module-name> and for cpanm use the command cpanm <module-name>.

  • LWP::UserAgent
    cpan install LWP::UserAgent
    cpanm LWP::UserAgent
  • XML::Simple
    cpan install XML::Simple
    cpanm XML::Simple

Library for WWW in Perl

The introduction of the hypertext transfer protocol in the late 1980s and early 1990s began the WWW revolution. The HTTP protocol classifies the client end of the transaction as the user agent. Generally speaking, this is the web browser (for example, the browser used to read this article).

When creating a Perl script that interacts with web protocols, the user agent becomes the script. Perl's LWP extension implements this functionality in the LWP::UserAgent module. Our first step towards building a RESTful Perl app is to understand the LWP interface.

When a web server is configured to require authentication before granting access to a particular resource, it will inform the user agent via the WWW-Authenticate header. See the code segment below for an example Perl script that displays this header.

Note: HTTP does not provide security unless configured to do so via HTTP over SSL (https). Be wary of sending any credentials to a URI beginning with http:// - you could be sending passwords in clear text over untrusted networks.

Grab the WWW-Authenticate header

#!/usr/bin/perl

# author: Jeff Wilson (aka JDubya)
# created: 2014
# license: Public Domain

use strict;
use warnings;
use LWP::UserAgent;

# Submit query to the URL of the RESTful interface
my $baseURL = q{https://restapi.mybrand.com/resource/};

# Spin up the browser object (via LWP::UserAgent)
my $ua = LWP::UserAgent->new(
    ssl_opts => { verify_hostname => 0 }, # not worried about SSL certs
    cookie_jar => {}, # keep cookies in RAM but not persistent between sessions
    );

my $searchResp = $ua->get( $baseURL );
my $realm = $searchResp->header('www-authenticate');
print "$realm\n" if defined($realm);

Set up transport and start grabbing data

Copy the code segment above to a text editor on your system. Change $baseURL to point to the server designated for the REST API you're interested in, and then run the script. If the results say, Basic realm="restapi", then "restapi" becomes the value for your $realm variable below.

my $netloc = "restapi.mybrand.com:443";
my $realm = "restapi";
$ua->credentials($netloc, $realm, $username, $password);

Look at the LWP::UserAgent documentation for the credentials routine. The code line above just caches the credentials scheme in the UserAgent object, but does not send them. Once set, the credentials are automatically send with the first request, whether via GET or POST.

In our fictitious mybrand.com scenario, the REST API documentation describes that we can grab a list of items for sale and their attributes by sending a GET request to a specific URL. Let's call $ua->get() and see what kind of data comes back.

my $url = "https://restapi.mybrand.com/sales/items/?attr=desc&attr=price";
my $resp = $ua->get($url);
die "Query failed: ".$resp->status_line unless ($resp->is_success);

Query credentials at runtime

sub GetCredentialsFromUser {
  print STDERR "Username: ";
  my $user = <>;
  chomp $user;
  die "Unable to proceed without a username" unless (length $user);
  print STDERR "Password for $user: ";
  # Turn off terminal's echo to read the password
  `stty -echo`;
  my $pw = <>;
  chomp $pw;
  # Restore the terminal echo after reading the password
  `stty echo`;
  die "Unable to proceed without a password" unless (length $pw);
  print STDERR "\n";
  return ($user,$pw);
}

Understanding the data

Extensible markup language (XML) is a near-ubiquitous data encoding format focused on merging machine readability with human readability. Perl has a few libraries devoted to parsing and generating valid XML code, but we only need a simple parser for this tutorial*.

Use Perl's built-in Data::Dumper library to pull apart the object returned by the XML parser.

use XML::Simple qw/:strict/;
my $xml = XMLin($resp->content, ForceArray=>1, KeyAttr=>undef);
use Data::Dumper;
die Dumper \$xml;

The output will show the hierarchy of embedded data structures returned by the XML parser. Each square bracket (such as "[" and "]") represents a reference to an array, and each curly brace (like "{" and "}") represents a reference to a hash.

*Note: Understanding that this tutorial is just an introduction to the concepts involved in REST, keep in mind that the documentation for XML::Simple recommends against basing any significant new projects on its library. See the documentation for recommendations of other libraries.

Sales items in raw XML

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<sales-item-list>
<sales-item serialnum="00001">
<attribute id="desc">Leprechaun Hat</attribute>
<attribute id="price">11.99</attribute>
</sales-item>
<sales-item serialnum="00002">
<attribute id="desc">Pot of Gold</attribute>
<attribute id="price">9999.99</attribute>
</sales-item>
<sales-item serialnum="00003">
<attribute id="desc">Rainbow</attribute>
<attribute id="price">0.99</attribute>
</sales-item>
<sales-item serialnum="00004">
<attribute id="desc">Shamrock</attribute>
<attribute id="price">3.99</attribute>
</sales-item>
<sales-item serialnum="00005">
<attribute id="desc">Wrist Watch</attribute>
<attribute id="price">79.95</attribute>
</sales-item>
<sales-item serialnum="00006">
<attribute id="desc">Pipe</attribute>
<attribute id="price">12.99</attribute>
</sales-item>
</sales-item-list>

Perl processed XML output by Data::Dumper

$VAR1 = {
          'sales-item' => [
                          {
                            'attribute' => [
                                           {
                                             'content' => 'Leprechaun Hat',
                                             'id' => 'desc'
                                           },
                                           {
                                             'content' => '11.99',
                                             'id' => 'price'
                                           }
                                         ],
                            'serialnum' => '00001'
                          },
                          {
                            'attribute' => [
                                           {
                                             'content' => 'Pot of Gold',
                                             'id' => 'desc'
                                           },
                                           {
                                             'content' => '9999.99',
                                             'id' => 'price'
                                           }
                                         ],
                            'serialnum' => '00002'
                          },
                          {
                            'attribute' => [
                                           {
                                             'content' => 'Rainbow',
                                             'id' => 'desc'
                                           },
                                           {
                                             'content' => '0.99',
                                             'id' => 'price'
                                           }
                                         ],
                            'serialnum' => '00003'
                          },
                          {
                            'attribute' => [
                                           {
                                             'content' => 'Shamrock',
                                             'id' => 'desc'
                                           },
                                           {
                                             'content' => '3.99',
                                             'id' => 'price'
                                           }
                                         ],
                            'serialnum' => '00004'
                          },
                          {
                            'attribute' => [
                                           {
                                             'content' => 'Wrist Watch',
                                             'id' => 'desc'
                                           },
                                           {
                                             'content' => '79.95',
                                             'id' => 'price'
                                           }
                                         ],
                            'serialnum' => '00005'
                          },
                          {
                            'attribute' => [
                                           {
                                             'content' => 'Pipe',
                                             'id' => 'desc'
                                           },
                                           {
                                             'content' => '12.99',
                                             'id' => 'price'
                                           }
                                         ],
                            'serialnum' => '00006'
                          }
                        ]
        };

Finalize the XML parser

As coded so far, each sales-item object is a hash of attribute and serialnum. Change the KeyAttr argument from undef to "serialnum", and the results get re-factored as keys based on the value of serialnum.

The code segment below shows the last piece needed to print out a list of sales-items, sorted by serialnum.

XML to CSV

my $parser = XMLin($xml,ForceArray=>1,KeyAttr=>"serialnum");

if (exists($parser->{error})) {
  die $parser->{error} unless ($parser->{error} eq "EndOfResults");
}

# Header row
print join(",",qw/SerialNum Desc Price/),"\n";

# Walk through XML parser's data structure
for my $serialnum (sort keys %{$parser->{"sales-item"}}) {
  my ($desc, $price);
  for my $row (@{ $parser->{"sales-item"}{$serialnum}{attribute} }) {
    my $id = $row->{id};
    my $content = $row->{content};
    if ($id eq "desc") {
      $desc = $content;
    }
    if ($id eq "price") {
      $price = $content;
    }
  }
  print join(",",$serialnum,$desc,$price),"\n";
}

Further comments on walking the XML

Let's take a second look at the output of Data::Dumper, which breaks down the data structure returned by the XML parser. Some of the annotations of hashes, arrays, and their references can be confusing.

After we go back and forth between the XML and the Dumper output, we'll take a deeper dive into the code segment that walks that data structure.

From XML to Perl data structure

So first, look back at the XML. The opening tag is sales-item-list ... but where is that in the Dumper output? It is hiding out as the anonymous VAR1, the root node, which points to a hash reference, indicated by the open curly brace "{".

The next tag in the XML is sales-item, with implied attribute serialnum and explicit attributes desc and price. The first label in the root hash reference is sales-item. and the data it points to is another reference, but this time it's an array, indicated by the open square bracket "[". This array reference points to a list of hash references.

The first hash reference is the encoding of the first sales-item's data with desc of Leprechaun Hat. In similar fashion, each sales-item's data is encoded as another hash reference, listed in the array reference that's pointed to by the label sales-item in the root hash reference.

It gets a bit tangled at first blush, but there's method to the madness - hang in there!

Using the output of Data::Dumper this way, we can figure out how to drill down into data structure returned by XML::Simple.

Pulling apart the data

How would you reference the desc and price of a sales-item with serialnum '00002'?

Begin at the array reference, then loop over the hash references until you encounter serialnum of '00002'.

Huh?

To de-reference an array reference in Perl, place the "@" sigil, as in @{$xml->{'sales-item'}}. To refer to single item in a hash reference, use the -> symbol, as in $row->{'serialnum'}. And for my final trick, ladies and gentlemen - the slice. To refer to more than one item in a hash, you're asking Perl for a list. So use the "@" sigil to de-reference the hash, and pass in a list of the named values you're looking for, as in @{$attr}{qw/id content/}.

Here's one way to do it ... below.

for my $row (@{$xml->{'sales-item'}}) {
  if ($row->{'serialnum'} eq '00002') {
    for my $attr (@{$row->{'attribute'}}) {
      my ($id,$content) = @{$attr}{qw/id content/};
      print "$id: $content\n";
    }
  }
}

More by this Author


Comments 2 comments

Chankey Pathak 21 months ago

A very simple yet effective tutorial on REST in Perl, thanks! :)


PETER GOTSIS 17 months ago

Awesome.... how to Use Perl to access RESTful APIs

    Sign in or sign up and post using a HubPages Network account.

    0 of 8192 characters used
    Post Comment

    No HTML is allowed in comments, but URLs will be hyperlinked. Comments are not for promoting your articles or other sites.


    Click to Rate This Article
    working