.\" -*- mode: troff; coding: utf-8 -*- .\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. .ie n \{\ . ds C` "" . ds C' "" 'br\} .el\{\ . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "UUlib 3" .TH UUlib 3 2023-07-25 "perl v5.38.0" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH NAME Convert::UUlib \- decode uu/xx/b64/mime/yenc/etc\-encoded data from a massive number of files .SH SYNOPSIS .IX Header "SYNOPSIS" .Vb 1 \& use Convert::UUlib \*(Aq:all\*(Aq; \& \& # read all the files named on the commandline and decode them \& # into the CURRENT directory. See below for a longer example. \& LoadFile $_ for @ARGV; \& \& for my $uu (GetFileList) { \& if ($uu\->state & FILE_OK) { \& $uu\->decode; \& print $uu\->filename, "\en"; \& } \& } .Ve .SH DESCRIPTION .IX Header "DESCRIPTION" This module started as an interface to the uulib/uudeview library by Frank Pilhofer that can be used to decode all kinds of usenet (and other) binary messages. .PP After upstream abondoned the project, th library was continuously bugfixed and improved in this module, with major focuses on security fixes, correctness and speed (that does not mean that this library is considered safe with untrusted data, but it surely is safer than the poriginal uudeview). .PP Read the file doc/library.pdf from the distribution for in-depth information about the C\-library used in this interface, and the rest of this document and especially the non-trivial decoder program at the end. .SH "EXPORTED CONSTANTS" .IX Header "EXPORTED CONSTANTS" .SS "Action code constants" .IX Subsection "Action code constants" .Vb 5 \& ACT_IDLE we don\*(Aqt do anything \& ACT_SCANNING scanning an input file \& ACT_DECODING decoding into a temp file \& ACT_COPYING copying temp to target \& ACT_ENCODING encoding a file .Ve .SS "Message severity levels" .IX Subsection "Message severity levels" .Vb 6 \& MSG_MESSAGE just a message, nothing important \& MSG_NOTE something that should be noticed \& MSG_WARNING important msg, processing continues \& MSG_ERROR processing has been terminated \& MSG_FATAL decoder cannot process further requests \& MSG_PANIC recovery impossible, app must terminate .Ve .SS Options .IX Subsection "Options" .Vb 10 \& OPT_VERSION version number MAJOR.MINORplPATCH (ro) \& OPT_FAST assumes only one part per file \& OPT_DUMBNESS switch off the program\*(Aqs intelligence \& OPT_BRACKPOL give numbers in [] higher precendence \& OPT_VERBOSE generate informative messages \& OPT_DESPERATE try to decode incomplete files \& OPT_IGNREPLY ignore RE:plies (off by default) \& OPT_OVERWRITE whether it\*(Aqs OK to overwrite ex. files \& OPT_SAVEPATH prefix to save\-files on disk \& OPT_IGNMODE ignore the original file mode \& OPT_DEBUG print messages with FILE/LINE info \& OPT_ERRNO get last error code for RET_IOERR (ro) \& OPT_PROGRESS retrieve progress information \& OPT_USETEXT handle text messages \& OPT_PREAMB handle Mime preambles/epilogues \& OPT_TINYB64 detect short B64 outside of Mime \& OPT_ENCEXT extension for single\-part encoded files \& OPT_REMOVE remove input files after decoding (dangerous) \& OPT_MOREMIME strict MIME adherence \& OPT_DOTDOT ".."\-unescaping has not yet been done on input files \& OPT_RBUF set default read I/O buffer size in bytes \& OPT_WBUF set default write I/O buffer size in bytes \& OPT_AUTOCHECK automatically check file list after every loadfile .Ve .SS "Result/Error codes" .IX Subsection "Result/Error codes" .Vb 10 \& RET_OK everything went fine \& RET_IOERR I/O Error \- examine errno \& RET_NOMEM not enough memory \& RET_ILLVAL illegal value for operation \& RET_NODATA decoder didn\*(Aqt find any data \& RET_NOEND encoded data wasn\*(Aqt ended properly \& RET_UNSUP unsupported function (encoding) \& RET_EXISTS file exists (decoding) \& RET_CONT continue \-\- special from ScanPart \& RET_CANCEL operation canceled .Ve .SS "File States" .IX Subsection "File States" .Vb 1 \& This code is zero, i.e. "false": \& \& UUFILE_READ Read in, but not further processed \& \& The following state codes are or\*(Aqed together: \& \& FILE_MISPART Missing Part(s) detected \& FILE_NOBEGIN No \*(Aqbegin\*(Aq found \& FILE_NOEND No \*(Aqend\*(Aq found \& FILE_NODATA File does not contain valid uudata \& FILE_OK All Parts found, ready to decode \& FILE_ERROR Error while decoding \& FILE_DECODED Successfully decoded \& FILE_TMPFILE Temporary decoded file exists .Ve .SS "Encoding types" .IX Subsection "Encoding types" .Vb 7 \& UU_ENCODED UUencoded data \& B64_ENCODED Mime\-Base64 data \& XX_ENCODED XXencoded data \& BH_ENCODED Binhex encoded \& PT_ENCODED Plain\-Text encoded (MIME) \& QP_ENCODED Quoted\-Printable (MIME) \& YENC_ENCODED yEnc encoded (non\-MIME) .Ve .SH "EXPORTED FUNCTIONS" .IX Header "EXPORTED FUNCTIONS" .SS "Initializing and cleanup" .IX Subsection "Initializing and cleanup" Initialize is automatically called when the module is loaded and allocates quite a small amount of memory for todays machines ;) CleanUp releases that again. .PP On my machine, a fairly complete decode with DBI backend needs about 10MB RSS to decode 20000 files. .IP CleanUp 4 .IX Item "CleanUp" Release memory, file items and clean up files. Should be called after a decoidng run, if you want to start a new one. .SS "Setting and querying options" .IX Subsection "Setting and querying options" .ie n .IP "$option = GetOption OPT_xxx" 4 .el .IP "\f(CW$option\fR = GetOption OPT_xxx" 4 .IX Item "$option = GetOption OPT_xxx" .PD 0 .IP "SetOption OPT_xxx, opt-value" 4 .IX Item "SetOption OPT_xxx, opt-value" .PD .PP See the \f(CW\*(C`OPT_xxx\*(C'\fR constants above to see which options exist. .SS "Setting various callbacks" .IX Subsection "Setting various callbacks" .IP "SetMsgCallback [callback\-function]" 4 .IX Item "SetMsgCallback [callback-function]" .PD 0 .IP "SetBusyCallback [callback\-function]" 4 .IX Item "SetBusyCallback [callback-function]" .IP "SetFileCallback [callback\-function]" 4 .IX Item "SetFileCallback [callback-function]" .IP "SetFNameFilter [callback\-function]" 4 .IX Item "SetFNameFilter [callback-function]" .PD .SS "Call the currently selected FNameFilter" .IX Subsection "Call the currently selected FNameFilter" .ie n .IP "$file = FNameFilter $file" 4 .el .IP "\f(CW$file\fR = FNameFilter \f(CW$file\fR" 4 .IX Item "$file = FNameFilter $file" .SS "Loading sourcefiles, optionally fuzzy merge and start decoding" .IX Subsection "Loading sourcefiles, optionally fuzzy merge and start decoding" .PD 0 .ie n .IP "($retval, $count) = LoadFile $fname, [$id, [$delflag, [$partno]]]" 4 .el .IP "($retval, \f(CW$count\fR) = LoadFile \f(CW$fname\fR, [$id, [$delflag, [$partno]]]" 4 .IX Item "($retval, $count) = LoadFile $fname, [$id, [$delflag, [$partno]]]" .PD Load the given file and scan it for encoded contents. Optionally tag it with the given id, and if \f(CW$delflag\fR is true, delete the file after it is no longer necessary. If you are certain of the part number, you can specify it as the last argument. .Sp A better (usually faster) way of doing this is using the \f(CW\*(C`SetFNameFilter\*(C'\fR functionality. .ie n .IP "$retval = Smerge $pass" 4 .el .IP "\f(CW$retval\fR = Smerge \f(CW$pass\fR" 4 .IX Item "$retval = Smerge $pass" If you are desperate, try to call \f(CW\*(C`Smerge\*(C'\fR with increasing \f(CW$pass\fR values, beginning at \f(CW0\fR, to try to merge parts that usually would not have been merged. .Sp Most probably this will result in garbled files, so never do this by default, except: .Sp If the \f(CW\*(C`OPT_AUTOCHECK\*(C'\fR option has been disabled (by default it is enabled) to speed up file loading, then you \fIhave\fR to call \f(CW\*(C`Smerge \-1\*(C'\fR after loading all files as an additional pre-pass (which is normally done by \f(CW\*(C`LoadFile\*(C'\fR). .ie n .IP "$item = GetFileListItem $item_number" 4 .el .IP "\f(CW$item\fR = GetFileListItem \f(CW$item_number\fR" 4 .IX Item "$item = GetFileListItem $item_number" Return the \f(CW$item\fR structure for the \f(CW$item_number\fR'th found file, or \&\f(CW\*(C`undef\*(C'\fR of no file with that number exists. .Sp The first file has number \f(CW0\fR, and the series has no holes, so you can iterate over all files by starting with zero and incrementing until you hit \f(CW\*(C`undef\*(C'\fR. .Sp This function has to walk the linear list of fils on each access, so if you want to iterate over all items, it is usually faster to use \&\f(CW\*(C`GetFileList\*(C'\fR. .ie n .IP "@items = GetFileList" 4 .el .IP "\f(CW@items\fR = GetFileList" 4 .IX Item "@items = GetFileList" Similar to \f(CW\*(C`GetFileListItem\*(C'\fR, but returns all files in one go, which is very much faster for large number of items, and has no drawbacks when used for a small number of items. .SS "Decoding files" .IX Subsection "Decoding files" .ie n .IP "$retval = $item\->rename ($newname)" 4 .el .IP "\f(CW$retval\fR = \f(CW$item\fR\->rename ($newname)" 4 .IX Item "$retval = $item->rename ($newname)" Change the ondisk filename where the decoded file will be saved. .ie n .IP "$retval = $item\->decode_temp" 4 .el .IP "\f(CW$retval\fR = \f(CW$item\fR\->decode_temp" 4 .IX Item "$retval = $item->decode_temp" Decode the file into a temporary location, use \f(CW\*(C`$item\->infile\*(C'\fR to retrieve the temporary filename. .ie n .IP "$retval = $item\->remove_temp" 4 .el .IP "\f(CW$retval\fR = \f(CW$item\fR\->remove_temp" 4 .IX Item "$retval = $item->remove_temp" Remove the temporarily decoded file again. .ie n .IP "$retval = $item\->decode ([$target_path])" 4 .el .IP "\f(CW$retval\fR = \f(CW$item\fR\->decode ([$target_path])" 4 .IX Item "$retval = $item->decode ([$target_path])" Decode the file to its destination, or the given target path. .ie n .IP "$retval = $item\->info (callback-function)" 4 .el .IP "\f(CW$retval\fR = \f(CW$item\fR\->info (callback-function)" 4 .IX Item "$retval = $item->info (callback-function)" .SS "Querying (and setting) item attributes" .IX Subsection "Querying (and setting) item attributes" .PD 0 .ie n .IP "$state = $item\->state" 4 .el .IP "\f(CW$state\fR = \f(CW$item\fR\->state" 4 .IX Item "$state = $item->state" .ie n .IP "$mode = $item\->mode ([newmode])" 4 .el .IP "\f(CW$mode\fR = \f(CW$item\fR\->mode ([newmode])" 4 .IX Item "$mode = $item->mode ([newmode])" .ie n .IP "$uudet = $item\->uudet" 4 .el .IP "\f(CW$uudet\fR = \f(CW$item\fR\->uudet" 4 .IX Item "$uudet = $item->uudet" .ie n .IP "$size = $item\->size" 4 .el .IP "\f(CW$size\fR = \f(CW$item\fR\->size" 4 .IX Item "$size = $item->size" .ie n .IP "$filename = $item\->filename ([newfilename})" 4 .el .IP "\f(CW$filename\fR = \f(CW$item\fR\->filename ([newfilename})" 4 .IX Item "$filename = $item->filename ([newfilename})" .ie n .IP "$subfname = $item\->subfname" 4 .el .IP "\f(CW$subfname\fR = \f(CW$item\fR\->subfname" 4 .IX Item "$subfname = $item->subfname" .ie n .IP "$mimeid = $item\->mimeid" 4 .el .IP "\f(CW$mimeid\fR = \f(CW$item\fR\->mimeid" 4 .IX Item "$mimeid = $item->mimeid" .ie n .IP "$mimetype = $item\->mimetype" 4 .el .IP "\f(CW$mimetype\fR = \f(CW$item\fR\->mimetype" 4 .IX Item "$mimetype = $item->mimetype" .ie n .IP "$binfile = $item\->binfile" 4 .el .IP "\f(CW$binfile\fR = \f(CW$item\fR\->binfile" 4 .IX Item "$binfile = $item->binfile" .PD .SS "Information about source parts" .IX Subsection "Information about source parts" .ie n .IP "$parts = $item\->parts" 4 .el .IP "\f(CW$parts\fR = \f(CW$item\fR\->parts" 4 .IX Item "$parts = $item->parts" Return information about all parts (source files) used to decode the file as a list of hashrefs with the following structure: .Sp .Vb 11 \& { \& partno => , \& # the following member sonly exist when they contain useful information \& sfname => , \& filename => , \& subfname => , \& subject => , \& origin => , \& mimetype => , \& mimeid => , \& } .Ve .Sp Usually you are interested mostly the \f(CW\*(C`sfname\*(C'\fR and possibly the \f(CW\*(C`partno\*(C'\fR and \f(CW\*(C`filename\*(C'\fR members. .SS "Functions below are not documented and not very well tested \- feedback welcome" .IX Subsection "Functions below are not documented and not very well tested - feedback welcome" .Vb 7 \& QuickDecode \& EncodeMulti \& EncodePartial \& EncodeToStream \& EncodeToFile \& E_PrepSingle \& E_PrepPartial .Ve .SS "EXTENSION FUNCTIONS" .IX Subsection "EXTENSION FUNCTIONS" Functions found in this module but not documented in the uulib documentation: .ie n .IP "$msg = straction ACT_xxx" 4 .el .IP "\f(CW$msg\fR = straction ACT_xxx" 4 .IX Item "$msg = straction ACT_xxx" Return a human readable string representing the given action code. .ie n .IP "$msg = strerror RET_xxx" 4 .el .IP "\f(CW$msg\fR = strerror RET_xxx" 4 .IX Item "$msg = strerror RET_xxx" Return a human readable string representing the given error code. .ie n .IP "$str = strencoding xxx_ENCODED" 4 .el .IP "\f(CW$str\fR = strencoding xxx_ENCODED" 4 .IX Item "$str = strencoding xxx_ENCODED" Return the name of the encoding type as a string. .ie n .IP "$str = strmsglevel MSG_xxx" 4 .el .IP "\f(CW$str\fR = strmsglevel MSG_xxx" 4 .IX Item "$str = strmsglevel MSG_xxx" Returns the message level as a string. .ie n .IP "SetFileNameCallback $cb" 4 .el .IP "SetFileNameCallback \f(CW$cb\fR" 4 .IX Item "SetFileNameCallback $cb" Sets (or queries) the FileNameCallback, which is called whenever the decoding library can't find a filename and wants to extract a filename from the subject line of a posting. The callback will be called with two arguments, the subject line and the current candidate for the filename. The latter argument can be \f(CW\*(C`undef\*(C'\fR, which means that no filename could be found (and likely no one exists, so it is safe to also return \f(CW\*(C`undef\*(C'\fR in this case). If it doesn't return anything (not even \&\f(CW\*(C`undef\*(C'\fR!), then nothing happens, so this is a no-op callback: .Sp .Vb 3 \& sub cb { \& return (); \& } .Ve .Sp If it returns \f(CW\*(C`undef\*(C'\fR, then this indicates that no filename could be found. In all other cases, the return value is taken to be the filename. .Sp This is a slightly more useful callback: .Sp .Vb 8 \& sub cb { \& return unless $_[1]; # skip "Re:"\-plies et al. \& my ($subject, $filename) = @_; \& # if we find some *.rar, take it \& return $1 if $subject =~ /(\ew+\e.rar)/; \& # otherwise just pass what we have \& return (); \& } .Ve .SH "LARGE EXAMPLE DECODER" .IX Header "LARGE EXAMPLE DECODER" The general workflow for decoding is like this: .ie n .IP "1. Configure options with ""SetOption"" or ""SetXXXCallback""." 4 .el .IP "1. Configure options with \f(CWSetOption\fR or \f(CWSetXXXCallback\fR." 4 .IX Item "1. Configure options with SetOption or SetXXXCallback." .PD 0 .ie n .IP "2. Load all source files with ""LoadFile""." 4 .el .IP "2. Load all source files with \f(CWLoadFile\fR." 4 .IX Item "2. Load all source files with LoadFile." .ie n .IP "3. Optionally ""Smerge""." 4 .el .IP "3. Optionally \f(CWSmerge\fR." 4 .IX Item "3. Optionally Smerge." .ie n .IP "4. Iterate over all ""GetFileList"" items (i.e. result files)." 4 .el .IP "4. Iterate over all \f(CWGetFileList\fR items (i.e. result files)." 4 .IX Item "4. Iterate over all GetFileList items (i.e. result files)." .ie n .IP "5. ""CleanUp"" to delete files and free items." 4 .el .IP "5. \f(CWCleanUp\fR to delete files and free items." 4 .IX Item "5. CleanUp to delete files and free items." .PD .PP What follows is the file \f(CW\*(C`example\-decoder\*(C'\fR from the distribution that illustrates the above worklfow in a non-trivial example. .PP .Vb 1 \& #!/usr/bin/perl \& \& # decode all the files in the directory uusrc/ and copy \& # the resulting files to uudst/ \& \& use Convert::UUlib \*(Aq:all\*(Aq; \& \& sub namefilter { \& my ($path) = @_; \& \& $path=~s/^.*[\e/\e\e]//; \& \& $path \& } \& \& sub busycb { \& my ($action, $curfile, $partno, $numparts, $percent, $fsize) = @_; \& $_[0]=straction($action); \& print "busy_callback(", (join ",",@_), ")\en"; \& 0 \& } \& \& SetOption OPT_RBUF, 128*1024; \& SetOption OPT_WBUF, 1024*1024; \& SetOption OPT_IGNMODE, 1; \& SetOption OPT_IGNMODE, 1; \& SetOption OPT_VERBOSE, 1; \& SetOption OPT_AUTOCHK, 0; \& \& # show the three ways you can set callback functions. I normally \& # prefer the one with the sub inplace. \& SetFNameFilter \e&namefilter; \& \& SetBusyCallback "busycb", 333; \& \& SetMsgCallback sub { \& my ($msg, $level) = @_; \& print uc strmsglevel $_[1], ": $msg\en"; \& }; \& \& # the following non\-trivial FileNameCallback takes care \& # of some subject lines not detected properly by uulib: \& SetFileNameCallback sub { \& return unless $_[1]; # skip "Re:"\-plies et al. \& local $_ = $_[0]; \& \& # the following rules are rather effective on some newsgroups, \& # like alt.binaries.games.anime, where non\-mime, uuencoded data \& # is very common \& \& # if we find some *.rar, take it as the filename \& return $1 if /(\eS{3,}\e.(?:[rstuvwxyz]\ed\ed|rar))\es/i; \& \& # one common subject format \& return $1 if /\- "(.{2,}?\e..+?)" (?:yenc )?\e(\ed+\e/\ed+\e)/i; \& \& # \- filename.par (04/55) \& return $1 if /\- "?(\eS{3,}\e.\eS+?)"? (?:yenc )?\e(\ed+\e/\ed+\e)/i; \& \& # \- (xxx) No. 1 sayuri81.jpg 756565 bytes \& # \- (20 files) No.17 Roseanne.jpg [2/2] \& return $1 if /No\e.[ 0\-9]+ (\eS+\e....) (?:\ed+ bytes )?\e[/; \& \& # try to detect some common forms of filenames \& return $1 if /([a\-z0\-9_\e\-+.]{3,}\e.[a\-z]{3,4}(?:.\ed+))/i; \& \& # otherwise just pass what we have \& () \& }; \& \& # now read all files in the directory uusrc/* \& for () { \& my ($retval, $count) = LoadFile ($_, $_, 1); \& print "file($_), status(", strerror $retval, ") parts($count)\en"; \& } \& \& Smerge \-1; \& \& SetOption OPT_SAVEPATH, "uudst/"; \& \& # now wade through all files and their source parts \& for my $uu (GetFileList) { \& print "file ", $uu\->filename, "\en"; \& print " state ", $uu\->state, "\en"; \& print " mode ", $uu\->mode, "\en"; \& print " uudet ", strencoding $uu\->uudet, "\en"; \& print " size ", $uu\->size, "\en"; \& print " subfname ", $uu\->subfname, "\en"; \& print " mimeid ", $uu\->mimeid, "\en"; \& print " mimetype ", $uu\->mimetype, "\en"; \& \& # print additional info about all parts \& print " parts"; \& for ($uu\->parts) { \& for my $k (sort keys %$_) { \& print " $k=$_\->{$k}"; \& } \& print "\en"; \& } \& \& $uu\->remove_temp; \& \& if (my $err = $uu\->decode) { \& print " ERROR ", strerror $err, "\en"; \& } else { \& print " successfully saved as uudst/", $uu\->filename, "\en"; \& } \& } \& \& print "cleanup...\en"; \& \& CleanUp; .Ve .SH "PERLMULTICORE SUPPORT" .IX Header "PERLMULTICORE SUPPORT" This module supports the perlmulticore standard (see for more info) for the following functions \- generally these are functions accessing the disk and/or using considerable CPU time: .PP .Vb 5 \& LoadFile \& $item\->decode \& $item\->decode_temp \& $item\->remove_temp \& $item\->info .Ve .PP The perl interpreter will be reacquired/released on every callback invocation, so for performance reasons, callbacks should be avoided if that is costly. .PP Future versions might enable multicore support for more functions. .SH "BUGS AND LIMITATIONS" .IX Header "BUGS AND LIMITATIONS" The original uulib library this module uses was written at a time where main memory of measured in megabytes and buffer overflows as a security thign didn't exist. While a lot of security fixes have been applied over the years (includign some defense in depth mechanism that can shield against a lot of as-of-yet undetected bugs), using this library for security purposes requires care. .PP Likewise, file sizes when the uulib library was written were tiny compared to today, so do not expect this library to handle files larger than 2GB. .PP Lastly, this module uses a very "C\-like" interface, which means it doesn't protect you from invalid points as you might expect from "more perlish" modules \- for example, accessing a file item object after callinbg \&\f(CW\*(C`CleanUp\*(C'\fR will likely result in crashes, memory corruption, or worse. .SH AUTHOR .IX Header "AUTHOR" Marc Lehmann , the original uulib library was written by Frank Pilhofer , and later heavily bugfixed by Marc Lehmann. .SH "SEE ALSO" .IX Header "SEE ALSO" \&\fBperl\fR\|(1), uudeview homepage at .