PAR::Tutorial(3) User Contributed Perl Documentation PAR::Tutorial(3)

PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR

This is a tutorial on PAR, first appeared at the 7th Perl Conference. The HTML version of this tutorial is available online as http://search.cpan.org/perldoc?PAR::Tutorial

% sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101"
Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1.
BEGIN failed--compilation aborted at sshnuke.pl line 1.
  • Q: "Help! I can't run your program!"
  • A1: Install Perl & "perl -MCPAN -e'install(...)'"
  • How do we know which modules are needed?
  • New versions of CPAN modules may break "sshnuke.pl"
A2: Install Perl & "tar zxf my_perllib.tgz"
Possibly overwriting existing modules; not cross-platform at all
A3: Use the executable generated by "perlcc sshnuke.pl"
Impossible to debug; "perlcc" usually does not work anyway

PAR, the Perl Archive Toolkit

Do what JAR (Java Archive) does for Perl
  • Aggregates modules, scripts and other files into a Zip file
  • Easy to generate, update and extract
  • Version consistency: solves forward-compatibility problems
  • Developed by community: "par@perl.org"
PAR files can be packed into self-contained scripts
  • Automatically scans perl script for dependencies
  • Bundles all necessary 3rd-party modules with it
  • Requires only core Perl to run on the target machine
  • PAR also comes with "pp", the Perl Packager:
    % pp -o sshnuke.exe sshnuke.pl # stand-alone executable!
    

  • PAR files are just Zip files with modules in it
  • Any Zip tools can generate them:
    % zip foo.par Hello.pm World.pm        # pack two modules
    % zip -r bar.par lib/          # grab all modules in lib/
    
  • To load modules from PAR files:
    use PAR;
    use lib "foo.par";             # the .par part is optional
    use Hello;
    
  • This also works:
    use PAR "/home/mylibs/*.par";  # put all of them into @INC
    use Hello;
    

PAR Loaders

  • Use "par.pl" to run files inside a PAR archive:
    % par.pl foo.par               # looks for 'main.pl' by default
    % par.pl foo.par test.pl       # runs script/test.pl in foo.par
    
  • Same thing, with the stand-alone "parl" or "parl.exe":
    % parl foo.par                 # no perl or PAR.pm needed!
    % parl foo.par test.pl         # ditto
    
  • The PAR loader can prepend itself to a PAR file:
  • "-b" bundles non-core modules needed by "PAR.pm":
    % par.pl -b -O./foo.pl foo.par # self-contained script
    
  • "-B" bundles core modules in addition to "-b":
    % parl -B -O./foo.exe foo.par  # self-contained binary
    

  • Recursively scan dependencies with "scandeps.pl":
    % scandeps.pl sshnuke.pl
    # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN
    'Crypt::SSLeay'       => '0', #  X   #
    'Net::HTTP'           => '0', #      #
    'Crypt::SSLeay::X509' => '0', # S    # Crypt::SSLeay
    'Net::HTTP::Methods'  => '0', # S    # Net::HTTP
    'Compress::Zlib'      => '0', #  X   # Net::HTTP::Methods
    
  • Scan an one-liner, list all involved files:
    % scandeps.pl -V -e "use Dynaloader;"
    ...
    # auto/DynaLoader/dl_findfile.al [autoload]
    # auto/DynaLoader/extralibs.ld [autoload]
    # auto/File/Glob/Glob.bs [data]
    # auto/File/Glob/Glob.so [shared]
    ...
    

  • Combines scanning, zipping and loader-embedding:
    % pp -o out.exe src.pl         # self-contained .exe
    % out.exe                      # runs anywhere on the same OS
    
  • Bundle additional modules:
    % pp -o out.exe -M CGI src.pl  # pack CGI + its dependencies, too
    
  • Pack one-liners:
    % pp -o out.exe -e 'print "Hi!"'   # turns one-liner into executable
    
  • Generate PAR files instead of executables:
    % pp -p src.pl                 # makes 'source.par'
    % pp -B -p src.pl              # include core modules
    

Command-line options are almost identical to "perlcc"'s
Also supports "gcc"-style long options:
% pp --gui --verbose --output=out.exe src.pl
  • Small initial overhead; no runtime overhead
  • Dependencies are POD-stripped before packing
  • Loads modules directly into memory on demand
  • Shared libraries (DLLs) are extracted with File::Temp
  • Works on Perl 5.6.0 or above
  • Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...

  • A common question:
    > I have used pp to make several standalone applications which work
    > great, the only problem is that for each executable that I make, I am
    > assuming the parl.exe is somehow bundled into the resulting exe.
    
  • The obvious workaround:
    You can ship parl.exe by itself, along with .par files built
    by "pp -p", and run those PAR files by associating them to parl.exe.
    
  • On platforms that have "ln", there is a better solution:
    % pp --output=a.out a.pl b.pl  # two scripts in one!
    % ln a.out b.out               # symlink also works
    % ./a.out                      # runs a.pl
    % ./b.out                      # runs b.pl
    

  • Of course, there is no cross-platform binary format
  • Pure-perl PAR packages are cross-platform by default
  • However, XS modules are specific to Perl version and platform
  • Multiple versions of a XS module can co-exist in a PAR file
  • Suppose we need "out.par" on both Win32 and Finix:
    C:\> pp --multiarch --output=out.par src.pl
    ...copy src.pl and out.par to a Finix machine...
    % pp --multiarch --output=out.par src.pl
    
  • Now it works on both platforms:
    % parl out.par                 # runs src.pl
    % perl -MPAR=out.par -e '...'  # uses modules inside out.par
    

  • Modules can reside in several directories:
    /                      # casual packaging only
    /lib/                  # standard location
    /arch/                 # for creating from blib/ 
    /i386-freebsd/         # i.e. $Config{archname}
    /5.8.0/                # i.e. Perl version number
    /5.8.0/i386-freebsd/   # combination of the two above
    
  • Scripts are stored in one of the two locations:
    /                      # casual packaging only
    /script/               # standard location
    
  • Shared libraries may be architecture- or perl-version-specific:
    /shlib/(5.8.0/)?(i386-freebsd/)?
    
  • PAR files may recursively contain other PAR files:
    /par/(5.8.0/)?(i386-freebsd/)?
    

MANIFEST
  • Index of all files inside PAR
  • Can be parsed with "ExtUtils::Manifest"
META.yml
  • Dependency, license, runtime options
  • Can be parsed with "YAML"
SIGNATURE
  • OpenPGP-signed digital signature
  • Can be parsed and verified with "Module::Signature"

This is not meant to be a flame
All three maintainers have contributed to PAR directly; I'm grateful
perlcc
  • "The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged." (from perldoc perlcc)
  • Guaranteed to not work is more like it
PerlApp / Perl2exe
  • Expensive: Need to pay for each upgrade
  • Non-portable: Only available for limited platforms
  • Proprietary: Cannot extend its features or fix bugs
  • Obfuscated: Vendor and black-hats can see your code, but you can't
  • Inflexible: Does not work with existing Perl installations

MANIFEST: Best viewed with Mozilla

  • The URL of "MANIFEST" inside "/home/autrijus/foo.par":
    jar:file:///home/autrijus/foo.par!/MANIFEST
    
  • Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:
  • No needed to unzip anything; just click on files to view them

META.yml: Metadata galore

Static, machine-readable distribution metadata
Supported by "Module::Build", "ExtUtils::MakeMaker", "Module::Install"
  • A typical "pp"-generated "META.yml" looks like this:
    build_requires: {}
    conflicts: {}
    dist_name: out.par
    distribution_type: par
    dynamic_config: 0
    generated_by: 'Perl Packager version 0.03'
    license: unknown
    par:
      clean: 0
      signature: ''
      verbatim: 0
      version: 0.68
    
  • The "par:" settings controls its runtime behavior

SIGNATURE: Signing and verifying packages

OpenPGP clear-signed manifest with SHA1 digests
Supported by "Module::Signature", "CPANPLUS" and "Module::Build"
  • A typical "SIGNATURE" looks like this:
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1
    SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS
    ...
    -----BEGIN PGP SIGNATURE-----
    ...
    -----END PGP SIGNATURE-----
    
  • Use "pp" and "cpansign" to work with signatures:
    % pp -s -o foo.par bar.pl      # make and sign foo.par from bar.pl
    % cpansign -s foo.par  # sign this PAR file
    % cpansign -v foo.par  # verify this PAR file
    

Framework for self-contained Web applications
  • Similar to Java's "Web Application Archive" (WAR) files
  • Works with mod_perl 1.x or 2.x
A complete web application inside a ".par" file
  • Apache configuration, static files, Perl modules...
  • Supports Static, Registry and PerlRun handlers
  • Can also load all PARs under a directory
One additional special file: "web.conf"
Alias /myapp/cgi-perl/ ##PARFILE##/
<Location /myapp/cgi-perl>
    Options +ExecCGI
    SetHandler perl-script
    PerlHandler Apache::PAR::Registry
</Location>

  • First, make a "hondah.par" from an one-liner:
    # use the "web.conf" from the previous slide
    % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \
         --add web.conf
    % chmod a+x hondah.par
    
  • Add this to "httpd.conf", then restart apache:
    <IfDefine MODPERL2>
    PerlModule Apache2
    </IfDefine>
    PerlAddVar PARInclude /home/autrijus/hondah.par
    PerlModule Apache::PAR
    
  • Test it out:
    % GET http://localhost/myapp/cgi-perl/main.pl
    Hon Dah!
    
  • Instant one-liner web application that works!

  • With LWP installed, your can use remote PAR files:
    use PAR;
    use lib 'http://aut.dyndns.org/par/DBI-latest.par';
    use DBI;    # always up to date!
    
  • Modules are now cached under $ENV{PAR_GLOBAL_TEMP}
  • Auto-updates with "LWP::Simple::mirror"
  • Download only if modified
  • Safe for offline use after the first time
  • May use "SIGNATURE" to prevent DNS-spoofing
Makes large-scale deployment a breeze
  • Upgrades from a central location
  • No installers needed

Also known as source-hiding techniques
  • It is not encryption
  • Offered by PerlApp, Perl2Exe, Stunnix...
Usually easy to defeat
  • Take optree dump from memory, feed to "B::Deparse"
  • If you just want to stop a casual "grep", "deflate" already works
PAR now supports pluggable input filters with "pp -f"
  • Bundled examples: Bleach, PodStrip and PatchContent
  • True encryption using "Crypt::*"
  • Or even _product activation_ over the internet
Alternatively, just keep core logic in your server and use RPC

  • To get the host archive from a packed program:
    my $zip = PAR::par_handle($0); # an Archive::Zip object
    my $content = $zip->contents('MANIFEST');
    
  • Same thing, but with "read_file()":
    my $content = PAR::read_file('MANIFEST');
    
  • Loaded PAR files are stored in %PAR::LibCache:
    use PAR '/home/mylibs/*.par';
    while (my ($filename, $zip) = each %PAR::LibCache) {
        print "[$filename - MANIFEST]\n";
        print $zip->contents('MANIFEST');
    }
    

  • GUI toolkits often need to link with shared libraries:
    # search for libncurses under library paths and pack it
    % pp -l ncurses curses_app.pl  # same for Tk, Wx, Gtk, Qt...
    
  • Use "pp --gui" on Win32 to eliminate the console window:
    # pack 'src.pl' into a console-less 'out.exe' (Win32 only)
    % pp --gui -o out.exe src.pl
    
  • "Can't locate Foo/Widget/Bar.pm in @INC"?
  • Some toolkits (notably Tk) autoloads modules without "use" or "require"
  • Hence "pp" and "Module::ScanDeps" may fail to detect them
  • Tk problems mostly fixed by now, but other toolkits may still break
  • You can work around it with "pp -M" or an explicit "require"
  • Or better, send a short test-case to "par@perl.org" so we can fix it

Installing XS extensions from CPAN was difficult
  • Some platforms do not come with a compiler (Win32, MacOSX...)
  • Some headers or libraries may be missing
  • PAR.pm itself used to suffer from both problems
  • ...but not anymore -- "Module::Install" to the rescue!
    # same old Makefile.PL, with a few changes
    use inc::Module::Install;      # was "use ExtUtils::MakeMaker;"
    WriteMakefile( ... );          # same as the original
    check_nmake();                 # make sure the user have nmake
    par_base('AUTRIJUS');          # your CPAN ID or a URL
    fetch_par() unless can_cc();   # use precompiled PAR only if necessary
    
  • Users will not notice anything, except now it works
  • Of course, you still need to type "make par" and upload the precompiled package
  • PAR users can also install it directly with "parl -i"

Additional resources
Any questions?

Here begins the scary part
  • Grues, Dragons and Jabberwocks abound...
  • You are going to learn weird things about Perl internals
PAR invokes four areas of Perl arcana:
  • @INC code references
  • On-the-fly source filtering
  • Overriding "DynaLoader::bootstrap()" to handle XS modules
  • Making self-bootstrapping binary executables
The first two only works on 5.6 or later
  • DynaLoader and %INC are there since Perl 5 was born
  • PAR currently needs 5.6, but a 5.005 port is possible

On 1999-07-19, Ken Fox submitted a patch to P5P
  • To _enable using remote modules_ by putting hooks in @INC
  • It's accepted to come in Perl 5.6, but undocumented until 5.8
  • Type "perldoc -f require" to read the nitty-gritty details
  • Coderefs in @INC may return a fh, or undef to 'pass':
    push @INC, sub {
        my ($coderef, $filename) = @_;  # $coderef is \&my_sub
        open my $fh, "wget ftp://example.com/$filename |";
        return $fh;        # using remote modules, indeed!
    };
    
  • Perl 5.8 let you open a file handle to a string, so we just use that:
    open my $fh, '<', \($zip->memberNamed($filename)->contents);
    return $fh;
    
  • But Perl 5.6 does not have that, and I don't want to use temp files...

... Undocumented features to the rescue!
  • It turns out that @INC hooks can return two values
  • The first is still the file handle
  • The second is a code reference for line-by-line source filtering!
This is how "Acme::use::strict::with::pride" works:
# Force all modules used to use strict and warnings
open my $fh, "<", $filename or return;
my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
return ($fh, sub {
    return 0 unless @lines;    
    push @lines, $_; $_ = shift @lines; return length $_;
});

  • But we don't really have a filehandle for anything
  • Another undocumented feature saves the day!
  • We can actually omit the first return value altogether:
    # Return all contents line-by-line from the file inside PAR
    my @lines = split(
        /(?<=\n)/,
        $zip->memberNamed($filename)->contents
    );
    return (sub {
        $_ = shift(@lines);
        return length $_;
    });
    

XS modules have dynamically loaded libraries
  • They cannot be loaded as part of a zip file, so we extract them out
  • Must intercept DynaLoader's library-finding process
Module names are passed to "bootstrap" for XS loading
  • During the process, it calls "dl_findfile" to locate the file
  • So we install pre-hooks around both functions
Our "_bootstrap" just checks if the library is in PARs
If yes, extract it to a "File::Temp" temp file
The file will be automatically cleaned up when the program ends
  • It then pass the arguments to the original "bootstrap"
  • Finally, our "dl_findfile" intercepts known filenames and return it

The par script ($0) itself
May be in plain-text or native executable format
Any number of embedded files
  • Typically used to bootstrap PAR's various dependencies
  • Each section begins with the magic string "FILE"
  • Length of filename in pack('N') format and the filename (auto/.../)
  • File length in pack('N') and the file's content (not compressed)
One PAR file
Just a regular zip file with the magic string "PK\003\004"
Ending section
  • A pack('N') number of the total length of FILE and PAR sections
  • Finally, there must be a 8-bytes magic string: "\012PAR.pm\012"

All we can expect is a working perl interpreter
  • The self-contained script *must not* use any modules at all
  • But to process PAR files, we need XS modules like Compress::Zlib
Answer: bundle all modules + libraries used by PAR.pm
  • That's what the "FILE" section in the previous slide is for
  • Load modules to memory, and write object files to disk
  • Then use a local @INC hook to load them on demand
Minimizing the amount of temporary files
  • First, try to load PerlIO::scalar and File::Temp
  • Set up an END hook to unlink all temp files up to this point
  • Load other bundled files, and look in the compressed PAR section
  • This can be much easier with a pure-perl "inflate()"; patches welcome!

Any questions, please?

PAR, pp, par.pl, parl

ex::lib::zip, Acme::use::strict::with::pride

App::Packer, Apache::PAR, CPANPLUS, Module::Install

Audrey Tang <cpan@audreyt.org>

You can write to the mailing list at <par@perl.org>, or send an empty mail to <par-subscribe@perl.org> to participate in the discussion.

Please submit bug reports to <bug-par@rt.cpan.org>.

Copyright 2003, 2004, 2005, 2006 by Audrey Tang <cpan@audreyt.org>.

This document is free documentation; you can redistribute it and/or modify it under the same terms as Perl itself.

See LICENSE.

2022-09-28 perl v5.36.0