.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "PARALLEL_ALTERNATIVES 7" .TH PARALLEL_ALTERNATIVES 7 "2024-01-26" "20240122" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" parallel_alternatives \- Alternatives to GNU parallel .SH "DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES" .IX Header "DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES" There are a lot programs that share functionality with \s-1GNU\s0 \&\fBparallel\fR. Some of these are specialized tools, and while \s-1GNU\s0 \&\fBparallel\fR can emulate many of them, a specialized tool can be better at a given task. \s-1GNU\s0 \fBparallel\fR strives to include the best of the general functionality without sacrificing ease of use. .PP \&\fBparallel\fR has existed since 2002\-01\-06 and as \s-1GNU\s0 \fBparallel\fR since 2010. A lot of the alternatives have not had the vitality to survive that long, but have come and gone during that time. .PP \&\s-1GNU\s0 \fBparallel\fR is actively maintained with a new release every month since 2010. Most other alternatives are fleeting interests of the developers with irregular releases and only maintained for a few years. .SS "\s-1SUMMARY LEGEND\s0" .IX Subsection "SUMMARY LEGEND" The following features are in some of the comparable tools: .PP \fIInputs\fR .IX Subsection "Inputs" .IP "I1. Arguments can be read from stdin" 4 .IX Item "I1. Arguments can be read from stdin" .PD 0 .IP "I2. Arguments can be read from a file" 4 .IX Item "I2. Arguments can be read from a file" .IP "I3. Arguments can be read from multiple files" 4 .IX Item "I3. Arguments can be read from multiple files" .IP "I4. Arguments can be read from command line" 4 .IX Item "I4. Arguments can be read from command line" .IP "I5. Arguments can be read from a table" 4 .IX Item "I5. Arguments can be read from a table" .IP "I6. Arguments can be read from the same file using #! (shebang)" 4 .IX Item "I6. Arguments can be read from the same file using #! (shebang)" .IP "I7. Line oriented input as default (Quoting of special chars not needed)" 4 .IX Item "I7. Line oriented input as default (Quoting of special chars not needed)" .PD .PP \fIManipulation of input\fR .IX Subsection "Manipulation of input" .IP "M1. Composed command" 4 .IX Item "M1. Composed command" .PD 0 .IP "M2. Multiple arguments can fill up an execution line" 4 .IX Item "M2. Multiple arguments can fill up an execution line" .IP "M3. Arguments can be put anywhere in the execution line" 4 .IX Item "M3. Arguments can be put anywhere in the execution line" .IP "M4. Multiple arguments can be put anywhere in the execution line" 4 .IX Item "M4. Multiple arguments can be put anywhere in the execution line" .IP "M5. Arguments can be replaced with context" 4 .IX Item "M5. Arguments can be replaced with context" .IP "M6. Input can be treated as the complete command line" 4 .IX Item "M6. Input can be treated as the complete command line" .PD .PP \fIOutputs\fR .IX Subsection "Outputs" .IP "O1. Grouping output so output from different jobs do not mix" 4 .IX Item "O1. Grouping output so output from different jobs do not mix" .PD 0 .IP "O2. Send stderr (standard error) to stderr (standard error)" 4 .IX Item "O2. Send stderr (standard error) to stderr (standard error)" .IP "O3. Send stdout (standard output) to stdout (standard output)" 4 .IX Item "O3. Send stdout (standard output) to stdout (standard output)" .IP "O4. Order of output can be same as order of input" 4 .IX Item "O4. Order of output can be same as order of input" .IP "O5. Stdout only contains stdout (standard output) from the command" 4 .IX Item "O5. Stdout only contains stdout (standard output) from the command" .IP "O6. Stderr only contains stderr (standard error) from the command" 4 .IX Item "O6. Stderr only contains stderr (standard error) from the command" .IP "O7. Buffering on disk" 4 .IX Item "O7. Buffering on disk" .IP "O8. No temporary files left if killed" 4 .IX Item "O8. No temporary files left if killed" .IP "O9. Test if disk runs full during run" 4 .IX Item "O9. Test if disk runs full during run" .IP "O10. Output of a line bigger than 4 \s-1GB\s0" 4 .IX Item "O10. Output of a line bigger than 4 GB" .PD .PP \fIExecution\fR .IX Subsection "Execution" .IP "E1. Run jobs in parallel" 4 .IX Item "E1. Run jobs in parallel" .PD 0 .IP "E2. List running jobs" 4 .IX Item "E2. List running jobs" .IP "E3. Finish running jobs, but do not start new jobs" 4 .IX Item "E3. Finish running jobs, but do not start new jobs" .IP "E4. Number of running jobs can depend on number of cpus" 4 .IX Item "E4. Number of running jobs can depend on number of cpus" .IP "E5. Finish running jobs, but do not start new jobs after first failure" 4 .IX Item "E5. Finish running jobs, but do not start new jobs after first failure" .IP "E6. Number of running jobs can be adjusted while running" 4 .IX Item "E6. Number of running jobs can be adjusted while running" .IP "E7. Only spawn new jobs if load is less than a limit" 4 .IX Item "E7. Only spawn new jobs if load is less than a limit" .PD .PP \fIRemote execution\fR .IX Subsection "Remote execution" .IP "R1. Jobs can be run on remote computers" 4 .IX Item "R1. Jobs can be run on remote computers" .PD 0 .IP "R2. Basefiles can be transferred" 4 .IX Item "R2. Basefiles can be transferred" .IP "R3. Argument files can be transferred" 4 .IX Item "R3. Argument files can be transferred" .IP "R4. Result files can be transferred" 4 .IX Item "R4. Result files can be transferred" .IP "R5. Cleanup of transferred files" 4 .IX Item "R5. Cleanup of transferred files" .IP "R6. No config files needed" 4 .IX Item "R6. No config files needed" .IP "R7. Do not run more than \s-1SSHD\s0's MaxStartups can handle" 4 .IX Item "R7. Do not run more than SSHD's MaxStartups can handle" .IP "R8. Configurable \s-1SSH\s0 command" 4 .IX Item "R8. Configurable SSH command" .IP "R9. Retry if connection breaks occasionally" 4 .IX Item "R9. Retry if connection breaks occasionally" .PD .PP \fISemaphore\fR .IX Subsection "Semaphore" .IP "S1. Possibility to work as a mutex" 4 .IX Item "S1. Possibility to work as a mutex" .PD 0 .IP "S2. Possibility to work as a counting semaphore" 4 .IX Item "S2. Possibility to work as a counting semaphore" .PD .PP \fILegend\fR .IX Subsection "Legend" .IP "\- = no" 4 .IX Item "- = no" .PD 0 .IP "x = not applicable" 4 .IX Item "x = not applicable" .IP "\s-1ID\s0 = yes" 4 .IX Item "ID = yes" .PD .PP As every new version of the programs are not tested the table may be outdated. Please file a bug report if you find errors (See \s-1REPORTING BUGS\s0). .PP parallel: .IP "I1 I2 I3 I4 I5 I6 I7" 4 .IX Item "I1 I2 I3 I4 I5 I6 I7" .PD 0 .IP "M1 M2 M3 M4 M5 M6" 4 .IX Item "M1 M2 M3 M4 M5 M6" .IP "O1 O2 O3 O4 O5 O6 O7 O8 O9 O10" 4 .IX Item "O1 O2 O3 O4 O5 O6 O7 O8 O9 O10" .IP "E1 E2 E3 E4 E5 E6 E7" 4 .IX Item "E1 E2 E3 E4 E5 E6 E7" .IP "R1 R2 R3 R4 R5 R6 R7 R8 R9" 4 .IX Item "R1 R2 R3 R4 R5 R6 R7 R8 R9" .IP "S1 S2" 4 .IX Item "S1 S2" .PD .SS "\s-1DIFFERENCES BETWEEN\s0 xargs \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN xargs AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- \- \- \- \-" 4 .IX Item "I1 I2 - - - - -" .PD 0 .IP "\- M2 M3 \- \- \-" 4 .IX Item "- M2 M3 - - -" .IP "\- O2 O3 \- O5 O6" 4 .IX Item "- O2 O3 - O5 O6" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- x \- \- \-" 4 .IX Item "- - - - - x - - -" .IP "\- \-" 4 .PD .PP \&\fBxargs\fR offers some of the same possibilities as \s-1GNU\s0 \fBparallel\fR. .PP \&\fBxargs\fR deals badly with special characters (such as space, \e, ' and "). To see the problem try this: .PP .Vb 7 \& touch important_file \& touch \*(Aqnot important_file\*(Aq \& ls not* | xargs rm \& mkdir \-p "My brother\*(Aqs 12\e" records" \& ls | xargs rmdir \& touch \*(Aqc:\ewindows\esystem32\eclfs.sys\*(Aq \& echo \*(Aqc:\ewindows\esystem32\eclfs.sys\*(Aq | xargs ls \-l .Ve .PP You can specify \fB\-0\fR, but many input generators are not optimized for using \fB\s-1NUL\s0\fR as separator but are optimized for \fBnewline\fR as separator. E.g. \fBawk\fR, \fBls\fR, \fBecho\fR, \fBtar \-v\fR, \fBhead\fR (requires using \fB\-z\fR), \fBtail\fR (requires using \fB\-z\fR), \fBsed\fR (requires using \&\fB\-z\fR), \fBperl\fR (\fB\-0\fR and \e0 instead of \en), \fBlocate\fR (requires using \fB\-0\fR), \fBfind\fR (requires using \fB\-print0\fR), \fBgrep\fR (requires using \fB\-z\fR or \fB\-Z\fR), \fBsort\fR (requires using \fB\-z\fR). .PP \&\s-1GNU\s0 \fBparallel\fR's newline separation can be emulated with: .PP .Vb 1 \& cat | xargs \-d "\en" \-n1 command .Ve .PP \&\fBxargs\fR can run a given number of jobs in parallel, but has no support for running number-of-cpu-cores jobs in parallel. .PP \&\fBxargs\fR has no support for grouping the output, therefore output may run together, e.g. the first half of a line is from one process and the last half of the line is from another process. The example \&\fBParallel grep\fR cannot be done reliably with \fBxargs\fR because of this. To see this in action try: .PP .Vb 10 \& parallel perl \-e "\*(Aq"\*(Aq$a="1"."{}"x10000000;print $a,"\en"\*(Aq"\*(Aq" \e \& \*(Aq>\*(Aq {} ::: a b c d e f g h \& # Serial = no mixing = the wanted result \& # \*(Aqtr \-s a\-z\*(Aq squeezes repeating letters into a single letter \& echo a b c d e f g h | xargs \-P1 \-n1 grep 1 | tr \-s a\-z \& # Compare to 8 jobs in parallel \& parallel \-kP8 \-n1 grep 1 ::: a b c d e f g h | tr \-s a\-z \& echo a b c d e f g h | xargs \-P8 \-n1 grep 1 | tr \-s a\-z \& echo a b c d e f g h | xargs \-P8 \-n1 grep \-\-line\-buffered 1 | \e \& tr \-s a\-z .Ve .PP Or try this: .PP .Vb 11 \& slow_seq() { \& echo Count to "$@" \& seq "$@" | \& perl \-ne \*(Aq$|=1; for(split//){ print; select($a,$a,$a,0.100);}\*(Aq \& } \& export \-f slow_seq \& # Serial = no mixing = the wanted result \& seq 8 | xargs \-n1 \-P1 \-I {} bash \-c \*(Aqslow_seq {}\*(Aq \& # Compare to 8 jobs in parallel \& seq 8 | parallel \-P8 slow_seq {} \& seq 8 | xargs \-n1 \-P8 \-I {} bash \-c \*(Aqslow_seq {}\*(Aq .Ve .PP \&\fBxargs\fR has no support for keeping the order of the output, therefore if running jobs in parallel using \fBxargs\fR the output of the second job cannot be postponed till the first job is done. .PP \&\fBxargs\fR has no support for running jobs on remote computers. .PP \&\fBxargs\fR has no support for context replace, so you will have to create the arguments. .PP If you use a replace string in \fBxargs\fR (\fB\-I\fR) you can not force \&\fBxargs\fR to use more than one argument. .PP Quoting in \fBxargs\fR works like \fB\-q\fR in \s-1GNU\s0 \fBparallel\fR. This means composed commands and redirection require using \fBbash \-c\fR. .PP .Vb 2 \& ls | parallel "wc {} >{}.wc" \& ls | parallel "echo {}; ls {}|wc" .Ve .PP becomes (assuming you have 8 cores and that none of the filenames contain space, " or '). .PP .Vb 2 \& ls | xargs \-d "\en" \-P8 \-I {} bash \-c "wc {} >{}.wc" \& ls | xargs \-d "\en" \-P8 \-I {} bash \-c "echo {}; ls {}|wc" .Ve .PP A more extreme example can be found on: https://unix.stackexchange.com/q/405552/ .PP https://www.gnu.org/software/findutils/ .SS "\s-1DIFFERENCES BETWEEN\s0 find \-exec \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN find -exec AND GNU Parallel" Summary (see legend above): .IP "\- \- \- x \- x \-" 4 .IX Item "- - - x - x -" .PD 0 .IP "\- M2 M3 \- \- \- \-" 4 .IX Item "- M2 M3 - - - -" .IP "\- O2 O3 O4 O5 O6" 4 .IX Item "- O2 O3 O4 O5 O6" .IP "\- \- \- \- \- \- \-" 4 .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "x x" 4 .IX Item "x x" .PD .PP \&\fBfind \-exec\fR offers some of the same possibilities as \s-1GNU\s0 \fBparallel\fR. .PP \&\fBfind \-exec\fR only works on files. Processing other input (such as hosts or URLs) will require creating these inputs as files. \fBfind \&\-exec\fR has no support for running commands in parallel. .PP https://www.gnu.org/software/findutils/ (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 make \-j \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN make -j AND GNU Parallel" Summary (see legend above): .IP "\- \- \- \- \- \- \-" 4 .PD 0 .IP "\- \- \- \- \- \-" 4 .IP "O1 O2 O3 \- x O6" 4 .IX Item "O1 O2 O3 - x O6" .IP "E1 \- \- \- E5 \-" 4 .IX Item "E1 - - - E5 -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBmake \-j\fR can run jobs in parallel, but requires a crafted Makefile to do this. That results in extra quoting to get filenames containing newlines to work correctly. .PP \&\fBmake \-j\fR computes a dependency graph before running jobs. Jobs run by \s-1GNU\s0 \fBparallel\fR does not depend on each other. .PP (Very early versions of \s-1GNU\s0 \fBparallel\fR were coincidentally implemented using \fBmake \-j\fR). .PP https://www.gnu.org/software/make/ (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 ppss \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN ppss AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- \- \- \- I7" 4 .IX Item "I1 I2 - - - - I7" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "O1 \- \- x \- \-" 4 .IX Item "O1 - - x - -" .IP "E1 E2 ?E3 E4 \- \- \-" 4 .IX Item "E1 E2 ?E3 E4 - - -" .IP "R1 R2 R3 R4 \- \- ?R7 ? ?" 4 .IX Item "R1 R2 R3 R4 - - ?R7 ? ?" .IP "\- \-" 4 .PD .PP \&\fBppss\fR is also a tool for running jobs in parallel. .PP The output of \fBppss\fR is status information and thus not useful for using as input for another command. The output from the jobs are put into files. .PP The argument replace string ($ITEM) cannot be changed. Arguments must be quoted \- thus arguments containing special characters (space '"&!*) may cause problems. More than one argument is not supported. Filenames containing newlines are not processed correctly. When reading input from a file null cannot be used as a terminator. \fBppss\fR needs to read the whole input file before starting any jobs. .PP Output and status information is stored in ppss_dir and thus requires cleanup when completed. If the dir is not removed before running \&\fBppss\fR again it may cause nothing to happen as \fBppss\fR thinks the task is already done. \s-1GNU\s0 \fBparallel\fR will normally not need cleaning up if running locally and will only need cleaning up if stopped abnormally and running remote (\fB\-\-cleanup\fR may not complete if stopped abnormally). The example \fBParallel grep\fR would require extra postprocessing if written using \fBppss\fR. .PP For remote systems \s-1PPSS\s0 requires 3 steps: config, deploy, and start. \s-1GNU\s0 \fBparallel\fR only requires one step. .PP \fI\s-1EXAMPLES FROM\s0 ppss \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM ppss MANUAL" .PP Here are the examples from \fBppss\fR's manual page with the equivalent using \s-1GNU\s0 \fBparallel\fR: .PP .Vb 1 \& 1$ ./ppss.sh standalone \-d /path/to/files \-c \*(Aqgzip \*(Aq \& \& 1$ find /path/to/files \-type f | parallel gzip \& \& 2$ ./ppss.sh standalone \-d /path/to/files \e \& \-c \*(Aqcp "$ITEM" /destination/dir \*(Aq \& \& 2$ find /path/to/files \-type f | parallel cp {} /destination/dir \& \& 3$ ./ppss.sh standalone \-f list\-of\-urls.txt \-c \*(Aqwget \-q \*(Aq \& \& 3$ parallel \-a list\-of\-urls.txt wget \-q \& \& 4$ ./ppss.sh standalone \-f list\-of\-urls.txt \-c \*(Aqwget \-q "$ITEM"\*(Aq \& \& 4$ parallel \-a list\-of\-urls.txt wget \-q {} \& \& 5$ ./ppss config \-C config.cfg \-c \*(Aqencode.sh \*(Aq \-d /source/dir \e \& \-m 192.168.1.100 \-u ppss \-k ppss\-key.key \-S ./encode.sh \e \& \-n nodes.txt \-o /some/output/dir \-\-upload \-\-download; \& ./ppss deploy \-C config.cfg \& ./ppss start \-C config \& \& 5$ # parallel does not use configs. If you want \& # a different username put it in nodes.txt: user@hostname \& find source/dir \-type f | \& parallel \-\-sshloginfile nodes.txt \-\-trc {.}.mp3 \e \& lame \-a {} \-o {.}.mp3 \-\-preset standard \-\-quiet \& \& 6$ ./ppss stop \-C config.cfg \& \& 6$ killall \-TERM parallel \& \& 7$ ./ppss pause \-C config.cfg \& \& 7$ Press: CTRL\-Z or killall \-SIGTSTP parallel \& \& 8$ ./ppss continue \-C config.cfg \& \& 8$ Enter: fg or killall \-SIGCONT parallel \& \& 9$ ./ppss.sh status \-C config.cfg \& \& 9$ killall \-SIGUSR2 parallel .Ve .PP https://github.com/louwrentius/PPSS (Last checked: 2010\-12) .SS "\s-1DIFFERENCES BETWEEN\s0 pexec \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN pexec AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- I4 I5 \- \-" 4 .IX Item "I1 I2 - I4 I5 - -" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "O1 O2 O3 \- O5 O6" 4 .IX Item "O1 O2 O3 - O5 O6" .IP "E1 \- \- E4 \- E6 \-" 4 .IX Item "E1 - - E4 - E6 -" .IP "R1 \- \- \- \- R6 \- \- \-" 4 .IX Item "R1 - - - - R6 - - -" .IP "S1 \-" 4 .IX Item "S1 -" .PD .PP \&\fBpexec\fR is also a tool for running jobs in parallel. .PP \fI\s-1EXAMPLES FROM\s0 pexec \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM pexec MANUAL" .PP Here are the examples from \fBpexec\fR's info page with the equivalent using \s-1GNU\s0 \fBparallel\fR: .PP .Vb 2 \& 1$ pexec \-o sqrt\-%s.dat \-p "$(seq 10)" \-e NUM \-n 4 \-c \-\- \e \& \*(Aqecho "scale=10000;sqrt($NUM)" | bc\*(Aq \& \& 1$ seq 10 | parallel \-j4 \*(Aqecho "scale=10000;sqrt({})" | \e \& bc > sqrt\-{}.dat\*(Aq \& \& 2$ pexec \-p "$(ls myfiles*.ext)" \-i %s \-o %s.sort \-\- sort \& \& 2$ ls myfiles*.ext | parallel sort {} ">{}.sort" \& \& 3$ pexec \-f image.list \-n auto \-e B \-u star.log \-c \-\- \e \& \*(Aqfistar $B.fits \-f 100 \-F id,x,y,flux \-o $B.star\*(Aq \& \& 3$ parallel \-a image.list \e \& \*(Aqfistar {}.fits \-f 100 \-F id,x,y,flux \-o {}.star\*(Aq 2>star.log \& \& 4$ pexec \-r *.png \-e IMG \-c \-o \- \-\- \e \& \*(Aqconvert $IMG ${IMG%.png}.jpeg ; "echo $IMG: done"\*(Aq \& \& 4$ ls *.png | parallel \*(Aqconvert {} {.}.jpeg; echo {}: done\*(Aq \& \& 5$ pexec \-r *.png \-i %s \-o %s.jpg \-c \*(Aqpngtopnm | pnmtojpeg\*(Aq \& \& 5$ ls *.png | parallel \*(Aqpngtopnm < {} | pnmtojpeg > {}.jpg\*(Aq \& \& 6$ for p in *.png ; do echo ${p%.png} ; done | \e \& pexec \-f \- \-i %s.png \-o %s.jpg \-c \*(Aqpngtopnm | pnmtojpeg\*(Aq \& \& 6$ ls *.png | parallel \*(Aqpngtopnm < {} | pnmtojpeg > {.}.jpg\*(Aq \& \& 7$ LIST=$(for p in *.png ; do echo ${p%.png} ; done) \& pexec \-r $LIST \-i %s.png \-o %s.jpg \-c \*(Aqpngtopnm | pnmtojpeg\*(Aq \& \& 7$ ls *.png | parallel \*(Aqpngtopnm < {} | pnmtojpeg > {.}.jpg\*(Aq \& \& 8$ pexec \-n 8 \-r *.jpg \-y unix \-e IMG \-c \e \& \*(Aqpexec \-j \-m blockread \-d $IMG | \e \& jpegtopnm | pnmscale 0.5 | pnmtojpeg | \e \& pexec \-j \-m blockwrite \-s th_$IMG\*(Aq \& \& 8$ # Combining GNU B and GNU B. \& ls *jpg | parallel \-j8 \*(Aqsem \-\-id blockread cat {} | jpegtopnm |\*(Aq \e \& \*(Aqpnmscale 0.5 | pnmtojpeg | sem \-\-id blockwrite cat > th_{}\*(Aq \& \& # If reading and writing is done to the same disk, this may be \& # faster as only one process will be either reading or writing: \& ls *jpg | parallel \-j8 \*(Aqsem \-\-id diskio cat {} | jpegtopnm |\*(Aq \e \& \*(Aqpnmscale 0.5 | pnmtojpeg | sem \-\-id diskio cat > th_{}\*(Aq .Ve .PP https://www.gnu.org/software/pexec/ (Last checked: 2010\-12) .SS "\s-1DIFFERENCES BETWEEN\s0 xjobs \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN xjobs AND GNU Parallel" \&\fBxjobs\fR is also a tool for running jobs in parallel. It only supports running jobs on your local computer. .PP \&\fBxjobs\fR deals badly with special characters just like \fBxargs\fR. See the section \fB\s-1DIFFERENCES BETWEEN\s0 xargs \s-1AND GNU\s0 Parallel\fR. .PP \fI\s-1EXAMPLES FROM\s0 xjobs \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM xjobs MANUAL" .PP Here are the examples from \fBxjobs\fR's man page with the equivalent using \s-1GNU\s0 \fBparallel\fR: .PP .Vb 1 \& 1$ ls \-1 *.zip | xjobs unzip \& \& 1$ ls *.zip | parallel unzip \& \& 2$ ls \-1 *.zip | xjobs \-n unzip \& \& 2$ ls *.zip | parallel unzip >/dev/null \& \& 3$ find . \-name \*(Aq*.bak\*(Aq | xjobs gzip \& \& 3$ find . \-name \*(Aq*.bak\*(Aq | parallel gzip \& \& 4$ ls \-1 *.jar | sed \*(Aqs/\e(.*\e)/\e1 > \e1.idx/\*(Aq | xjobs jar tf \& \& 4$ ls *.jar | parallel jar tf {} \*(Aq>\*(Aq {}.idx \& \& 5$ xjobs \-s script \& \& 5$ cat script | parallel \& \& 6$ mkfifo /var/run/my_named_pipe; \& xjobs \-s /var/run/my_named_pipe & \& echo unzip 1.zip >> /var/run/my_named_pipe; \& echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe \& \& 6$ mkfifo /var/run/my_named_pipe; \& cat /var/run/my_named_pipe | parallel & \& echo unzip 1.zip >> /var/run/my_named_pipe; \& echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe .Ve .PP https://www.maier\-komor.de/xjobs.html (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 prll \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN prll AND GNU Parallel" \&\fBprll\fR is also a tool for running jobs in parallel. It does not support running jobs on remote computers. .PP \&\fBprll\fR encourages using \s-1BASH\s0 aliases and \s-1BASH\s0 functions instead of scripts. \s-1GNU\s0 \fBparallel\fR supports scripts directly, functions if they are exported using \fBexport \-f\fR, and aliases if using \fBenv_parallel\fR. .PP \&\fBprll\fR generates a lot of status information on stderr (standard error) which makes it harder to use the stderr (standard error) output of the job directly as input for another program. .PP \fI\s-1EXAMPLES FROM\s0 prll's \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM prll's MANUAL" .PP Here is the example from \fBprll\fR's man page with the equivalent using \s-1GNU\s0 \fBparallel\fR: .PP .Vb 1 \& 1$ prll \-s \*(Aqmogrify \-flip $1\*(Aq *.jpg \& \& 1$ parallel mogrify \-flip ::: *.jpg .Ve .PP https://github.com/exzombie/prll (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 dxargs \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN dxargs AND GNU Parallel" \&\fBdxargs\fR is also a tool for running jobs in parallel. .PP \&\fBdxargs\fR does not deal well with more simultaneous jobs than \s-1SSHD\s0's MaxStartups. \fBdxargs\fR is only built for remote run jobs, but does not support transferring of files. .PP https://web.archive.org/web/20120518070250/http://www. semicomplete.com/blog/geekery/distributed\-xargs.html (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 mdm/middleman \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN mdm/middleman AND GNU Parallel" middleman(mdm) is also a tool for running jobs in parallel. .PP \fI\s-1EXAMPLES FROM\s0 middleman's \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM middleman's WEBSITE" .PP Here are the shellscripts of https://web.archive.org/web/20110728064735/http://mdm. berlios.de/usage.html ported to \s-1GNU\s0 \fBparallel\fR: .PP .Vb 3 \& 1$ seq 19 | parallel buffon \-o \- | sort \-n > result \& cat files | parallel cmd \& find dir \-execdir sem cmd {} \e; .Ve .PP https://github.com/cklin/mdm (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 xapply \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN xapply AND GNU Parallel" \&\fBxapply\fR can run jobs in parallel on the local computer. .PP \fI\s-1EXAMPLES FROM\s0 xapply's \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM xapply's MANUAL" .PP Here are the examples from \fBxapply\fR's man page with the equivalent using \s-1GNU\s0 \fBparallel\fR: .PP .Vb 1 \& 1$ xapply \*(Aq(cd %1 && make all)\*(Aq */ \& \& 1$ parallel \*(Aqcd {} && make all\*(Aq ::: */ \& \& 2$ xapply \-f \*(Aqdiff %1 ../version5/%1\*(Aq manifest | more \& \& 2$ parallel diff {} ../version5/{} < manifest | more \& \& 3$ xapply \-p/dev/null \-f \*(Aqdiff %1 %2\*(Aq manifest1 checklist1 \& \& 3$ parallel \-\-link diff {1} {2} :::: manifest1 checklist1 \& \& 4$ xapply \*(Aqindent\*(Aq *.c \& \& 4$ parallel indent ::: *.c \& \& 5$ find ~ksb/bin \-type f ! \-perm \-111 \-print | \e \& xapply \-f \-v \*(Aqchmod a+x\*(Aq \- \& \& 5$ find ~ksb/bin \-type f ! \-perm \-111 \-print | \e \& parallel \-v chmod a+x \& \& 6$ find */ \-... | fmt 960 1024 | xapply \-f \-i /dev/tty \*(Aqvi\*(Aq \- \& \& 6$ sh <(find */ \-... | parallel \-s 1024 echo vi) \& \& 6$ find */ \-... | parallel \-s 1024 \-Xuj1 vi \& \& 7$ find ... | xapply \-f \-5 \-i /dev/tty \*(Aqvi\*(Aq \- \- \- \- \- \& \& 7$ sh <(find ... | parallel \-n5 echo vi) \& \& 7$ find ... | parallel \-n5 \-uj1 vi \& \& 8$ xapply \-fn "" /etc/passwd \& \& 8$ parallel \-k echo < /etc/passwd \& \& 9$ tr \*(Aq:\*(Aq \*(Aq\e012\*(Aq < /etc/passwd | \e \& xapply \-7 \-nf \*(Aqchown %1 %6\*(Aq \- \- \- \- \- \- \- \& \& 9$ tr \*(Aq:\*(Aq \*(Aq\e012\*(Aq < /etc/passwd | parallel \-N7 chown {1} {6} \& \& 10$ xapply \*(Aq[ \-d %1/RCS ] || echo %1\*(Aq */ \& \& 10$ parallel \*(Aq[ \-d {}/RCS ] || echo {}\*(Aq ::: */ \& \& 11$ xapply \-f \*(Aq[ \-f %1 ] && echo %1\*(Aq List | ... \& \& 11$ parallel \*(Aq[ \-f {} ] && echo {}\*(Aq < List | ... .Ve .PP https://www.databits.net/~ksb/msrc/local/bin/xapply/xapply.html (Last checked: 2010\-12) .SS "\s-1DIFFERENCES BETWEEN AIX\s0 apply \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN AIX apply AND GNU Parallel" \&\fBapply\fR can build command lines based on a template and arguments \- very much like \s-1GNU\s0 \fBparallel\fR. \fBapply\fR does not run jobs in parallel. \fBapply\fR does not use an argument separator (like \fB:::\fR); instead the template must be the first argument. .PP \fI\s-1EXAMPLES FROM IBM\s0's \s-1KNOWLEDGE CENTER\s0\fR .IX Subsection "EXAMPLES FROM IBM's KNOWLEDGE CENTER" .PP Here are the examples from \s-1IBM\s0's Knowledge Center and the corresponding command using \s-1GNU\s0 \fBparallel\fR: .PP To obtain results similar to those of the \fBls\fR command, enter: .IX Subsection "To obtain results similar to those of the ls command, enter:" .PP .Vb 2 \& 1$ apply echo * \& 1$ parallel echo ::: * .Ve .PP To compare the file named a1 to the file named b1, and the file named a2 to the file named b2, enter: .IX Subsection "To compare the file named a1 to the file named b1, and the file named a2 to the file named b2, enter:" .PP .Vb 2 \& 2$ apply \-2 cmp a1 b1 a2 b2 \& 2$ parallel \-N2 cmp ::: a1 b1 a2 b2 .Ve .PP To run the \fBwho\fR command five times, enter: .IX Subsection "To run the who command five times, enter:" .PP .Vb 2 \& 3$ apply \-0 who 1 2 3 4 5 \& 3$ parallel \-N0 who ::: 1 2 3 4 5 .Ve .PP To link all files in the current directory to the directory /usr/joe, enter: .IX Subsection "To link all files in the current directory to the directory /usr/joe, enter:" .PP .Vb 2 \& 4$ apply \*(Aqln %1 /usr/joe\*(Aq * \& 4$ parallel ln {} /usr/joe ::: * .Ve .PP https://www\-01.ibm.com/support/knowledgecenter/ ssw_aix_71/com.ibm.aix.cmds1/apply.htm (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 paexec \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN paexec AND GNU Parallel" \&\fBpaexec\fR can run jobs in parallel on both the local and remote computers. .PP \&\fBpaexec\fR requires commands to print a blank line as the last output. This means you will have to write a wrapper for most programs. .PP \&\fBpaexec\fR has a job dependency facility so a job can depend on another job to be executed successfully. Sort of a poor-man's \fBmake\fR. .PP \fI\s-1EXAMPLES FROM\s0 paexec's \s-1EXAMPLE CATALOG\s0\fR .IX Subsection "EXAMPLES FROM paexec's EXAMPLE CATALOG" .PP Here are the examples from \fBpaexec\fR's example catalog with the equivalent using \s-1GNU\s0 \fBparallel\fR: .PP 1_div_X_run .IX Subsection "1_div_X_run" .PP .Vb 1 \& 1$ ../../paexec \-s \-l \-c "\`pwd\`/1_div_X_cmd" \-n +1 < My\e brother\e\*(Aqs\e 12\e"\e records \& \& ls | map \*(Aqecho %; wc %\*(Aq .Ve .PP It works with \s-1GNU\s0 \fBparallel\fR: .PP .Vb 1 \& ls | parallel \*(Aqecho {}; wc {}\*(Aq .Ve .PP Under some circumstances it also works with \fBmap\fR: .PP .Vb 1 \& ls | map \*(Aqecho % works %\*(Aq .Ve .PP But tiny changes make it reject the input with special characters: .PP .Vb 1 \& ls | map \*(Aqecho % does not work "%"\*(Aq .Ve .PP This means that many \s-1UTF\-8\s0 characters will be rejected. This is by design. From the web page: "As such, programs that \fIquietly handle them, with no warnings at all,\fR are doing their users a disservice." .PP \&\fBmap\fR delays each job by 0.01 s. This can be emulated by using \&\fBparallel \-\-delay 0.01\fR. .PP \&\fBmap\fR prints '+' on stderr when a job starts, and '\-' when a job finishes. This cannot be disabled. \fBparallel\fR has \fB\-\-bar\fR if you need to see progress. .PP \&\fBmap\fR's replacement strings (% \f(CW%D\fR \f(CW%B\fR \f(CW%E\fR) can be simulated in \s-1GNU\s0 \&\fBparallel\fR by putting this in \fB~/.parallel/config\fR: .PP .Vb 4 \& \-\-rpl \*(Aq%\*(Aq \& \-\-rpl \*(Aq%D $_=Q(::dirname($_));\*(Aq \& \-\-rpl \*(Aq%B s:.*/::;s:\e.[^/.]+$::;\*(Aq \& \-\-rpl \*(Aq%E s:.*\e.::\*(Aq .Ve .PP \&\fBmap\fR does not have an argument separator on the command line, but uses the first argument as command. This makes quoting harder which again may affect readability. Compare: .PP .Vb 1 \& map \-p 2 \*(Aqperl \-ne \*(Aq"\*(Aq"\*(Aq/^\eS+\es+\eS+$/ and print $ARGV,"\en"\*(Aq"\*(Aq" * \& \& parallel \-q perl \-ne \*(Aq/^\eS+\es+\eS+$/ and print $ARGV,"\en"\*(Aq ::: * .Ve .PP \&\fBmap\fR can do multiple arguments with context replace, but not without context replace: .PP .Vb 1 \& parallel \-\-xargs echo \*(AqBEGIN{\*(Aq{}\*(Aq}END\*(Aq ::: 1 2 3 \& \& map "echo \*(AqBEGIN{\*(Aq%\*(Aq}END\*(Aq" 1 2 3 .Ve .PP \&\fBmap\fR has no support for grouping. So this gives the wrong results: .PP .Vb 9 \& parallel perl \-e \*(Aq\e$a=\e"1{}\e"x10000000\e;print\e \e$a,\e"\e\en\e"\*(Aq \*(Aq>\*(Aq {} \e \& ::: a b c d e f \& ls \-l a b c d e f \& parallel \-kP4 \-n1 grep 1 ::: a b c d e f > out.par \& map \-n1 \-p 4 \*(Aqgrep 1\*(Aq a b c d e f > out.map\-unbuf \& map \-n1 \-p 4 \*(Aqgrep \-\-line\-buffered 1\*(Aq a b c d e f > out.map\-linebuf \& map \-n1 \-p 1 \*(Aqgrep \-\-line\-buffered 1\*(Aq a b c d e f > out.map\-serial \& ls \-l out* \& md5sum out* .Ve .PP \fI\s-1EXAMPLES FROM\s0 map's \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM map's WEBSITE" .PP Here are the examples from \fBmap\fR's web page with the equivalent using \&\s-1GNU\s0 \fBparallel\fR: .PP .Vb 1 \& 1$ ls *.gif | map convert % %B.png # default max\-args: 1 \& \& 1$ ls *.gif | parallel convert {} {.}.png \& \& 2$ map "mkdir %B; tar \-C %B \-xf %" *.tgz # default max\-args: 1 \& \& 2$ parallel \*(Aqmkdir {.}; tar \-C {.} \-xf {}\*(Aq ::: *.tgz \& \& 3$ ls *.gif | map cp % /tmp # default max\-args: 100 \& \& 3$ ls *.gif | parallel \-X cp {} /tmp \& \& 4$ ls *.tar | map \-n 1 tar \-xf % \& \& 4$ ls *.tar | parallel tar \-xf \& \& 5$ map "cp % /tmp" *.tgz \& \& 5$ parallel cp {} /tmp ::: *.tgz \& \& 6$ map "du \-sm /home/%/mail" alice bob carol \& \& 6$ parallel "du \-sm /home/{}/mail" ::: alice bob carol \& or if you prefer running a single job with multiple args: \& 6$ parallel \-Xj1 "du \-sm /home/{}/mail" ::: alice bob carol \& \& 7$ cat /etc/passwd | map \-d: \*(Aqecho user %1 has shell %7\*(Aq \& \& 7$ cat /etc/passwd | parallel \-\-colsep : \*(Aqecho user {1} has shell {7}\*(Aq \& \& 8$ export MAP_MAX_PROCS=$(( \`nproc\` / 2 )) \& \& 8$ export PARALLEL=\-j50% .Ve .PP https://github.com/sitaramc/map (Last checked: 2020\-05) .SS "\s-1DIFFERENCES BETWEEN\s0 ladon \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN ladon AND GNU Parallel" \&\fBladon\fR can run multiple jobs on files in parallel. .PP \&\fBladon\fR only works on files and the only way to specify files is using a quoted glob string (such as \e*.jpg). It is not possible to list the files manually. .PP As replacement strings it uses \s-1FULLPATH DIRNAME BASENAME EXT RELDIR RELPATH\s0 .PP These can be simulated using \s-1GNU\s0 \fBparallel\fR by putting this in \&\fB~/.parallel/config\fR: .PP .Vb 8 \& \-\-rpl \*(AqFULLPATH $_=Q($_);chomp($_=qx{readlink \-f $_});\*(Aq \& \-\-rpl \*(AqDIRNAME $_=Q(::dirname($_));chomp($_=qx{readlink \-f $_});\*(Aq \& \-\-rpl \*(AqBASENAME s:.*/::;s:\e.[^/.]+$::;\*(Aq \& \-\-rpl \*(AqEXT s:.*\e.::\*(Aq \& \-\-rpl \*(AqRELDIR $_=Q($_);chomp(($_,$c)=qx{readlink \-f $_;pwd}); \& s:\eQ$c/\eE::;$_=::dirname($_);\*(Aq \& \-\-rpl \*(AqRELPATH $_=Q($_);chomp(($_,$c)=qx{readlink \-f $_;pwd}); \& s:\eQ$c/\eE::;\*(Aq .Ve .PP \&\fBladon\fR deals badly with filenames containing " and newline, and it fails for output larger than 200k: .PP .Vb 1 \& ladon \*(Aq*\*(Aq \-\- seq 36000 | wc .Ve .PP \fI\s-1EXAMPLES FROM\s0 ladon \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM ladon MANUAL" .PP It is assumed that the '\-\-rpl's above are put in \fB~/.parallel/config\fR and that it is run under a shell that supports '**' globbing (such as \fBzsh\fR): .PP .Vb 1 \& 1$ ladon "**/*.txt" \-\- echo RELPATH \& \& 1$ parallel echo RELPATH ::: **/*.txt \& \& 2$ ladon "~/Documents/**/*.pdf" \-\- shasum FULLPATH >hashes.txt \& \& 2$ parallel shasum FULLPATH ::: ~/Documents/**/*.pdf >hashes.txt \& \& 3$ ladon \-m thumbs/RELDIR "**/*.jpg" \-\- convert FULLPATH \e \& \-thumbnail 100x100^ \-gravity center \-extent 100x100 \e \& thumbs/RELPATH \& \& 3$ parallel mkdir \-p thumbs/RELDIR\e; convert FULLPATH \& \-thumbnail 100x100^ \-gravity center \-extent 100x100 \e \& thumbs/RELPATH ::: **/*.jpg \& \& 4$ ladon "~/Music/*.wav" \-\- lame \-V 2 FULLPATH DIRNAME/BASENAME.mp3 \& \& 4$ parallel lame \-V 2 FULLPATH DIRNAME/BASENAME.mp3 ::: ~/Music/*.wav .Ve .PP https://github.com/danielgtaylor/ladon (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 jobflow \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN jobflow AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- I7" 4 .IX Item "I1 - - - - - I7" .PD 0 .IP "\- \- M3 \- \- (M6)" 4 .IX Item "- - M3 - - (M6)" .IP "O1 O2 O3 \- O5 O6 (O7) \- \- O10" 4 .IX Item "O1 O2 O3 - O5 O6 (O7) - - O10" .IP "E1 \- \- \- \- E6 \-" 4 .IX Item "E1 - - - - E6 -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBjobflow\fR can run multiple jobs in parallel. .PP Just like \fBxargs\fR output from \fBjobflow\fR jobs running in parallel mix together by default. \fBjobflow\fR can buffer into files with \&\fB\-buffered\fR (placed in /run/shm), but these are not cleaned up if \&\fBjobflow\fR dies unexpectedly (e.g. by Ctrl-C). If the total output is big (in the order of RAM+swap) it can cause the system to slow to a crawl and eventually run out of memory. .PP Just like \fBxargs\fR redirection and composed commands require wrapping with \fBbash \-c\fR. .PP Input lines can at most be 4096 bytes. .PP \&\fBjobflow\fR is faster than \s-1GNU\s0 \fBparallel\fR but around 6 times slower than \fBparallel-bash\fR. .PP \&\fBjobflow\fR has no equivalent for \fB\-\-pipe\fR, or \fB\-\-sshlogin\fR. .PP \&\fBjobflow\fR makes it possible to set resource limits on the running jobs. This can be emulated by \s-1GNU\s0 \fBparallel\fR using \fBbash\fR's \fBulimit\fR: .PP .Vb 1 \& jobflow \-limits=mem=100M,cpu=3,fsize=20M,nofiles=300 myjob \& \& parallel \*(Aqulimit \-v 102400 \-t 3 \-f 204800 \-n 300 myjob\*(Aq .Ve .PP \fI\s-1EXAMPLES FROM\s0 jobflow \s-1README\s0\fR .IX Subsection "EXAMPLES FROM jobflow README" .PP .Vb 1 \& 1$ cat things.list | jobflow \-threads=8 \-exec ./mytask {} \& \& 1$ cat things.list | parallel \-j8 ./mytask {} \& \& 2$ seq 100 | jobflow \-threads=100 \-exec echo {} \& \& 2$ seq 100 | parallel \-j100 echo {} \& \& 3$ cat urls.txt | jobflow \-threads=32 \-exec wget {} \& \& 3$ cat urls.txt | parallel \-j32 wget {} \& \& 4$ find . \-name \*(Aq*.bmp\*(Aq | \e \& jobflow \-threads=8 \-exec bmp2jpeg {.}.bmp {.}.jpg \& \& 4$ find . \-name \*(Aq*.bmp\*(Aq | \e \& parallel \-j8 bmp2jpeg {.}.bmp {.}.jpg \& \& 5$ seq 100 | jobflow \-skip 10 \-count 10 \& \& 5$ seq 100 | parallel \-\-filter \*(Aq{1} > 10 and {1} <= 20\*(Aq echo \& \& 5$ seq 100 | parallel echo \*(Aq{= $_>10 and $_<=20 or skip() =}\*(Aq .Ve .PP https://github.com/rofl0r/jobflow (Last checked: 2022\-05) .SS "\s-1DIFFERENCES BETWEEN\s0 gargs \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN gargs AND GNU Parallel" \&\fBgargs\fR can run multiple jobs in parallel. .PP Older versions cache output in memory. This causes it to be extremely slow when the output is larger than the physical \s-1RAM,\s0 and can cause the system to run out of memory. .PP See more details on this in \fBman parallel_design\fR. .PP Newer versions cache output in files, but leave files in \f(CW$TMPDIR\fR if it is killed. .PP Output to stderr (standard error) is changed if the command fails. .PP \fI\s-1EXAMPLES FROM\s0 gargs \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM gargs WEBSITE" .PP .Vb 1 \& 1$ seq 12 \-1 1 | gargs \-p 4 \-n 3 "sleep {0}; echo {1} {2}" \& \& 1$ seq 12 \-1 1 | parallel \-P 4 \-n 3 "sleep {1}; echo {2} {3}" \& \& 2$ cat t.txt | gargs \-\-sep "\es+" \e \& \-p 2 "echo \*(Aq{0}:{1}\-{2}\*(Aq full\-line: \e\*(Aq{}\e\*(Aq" \& \& 2$ cat t.txt | parallel \-\-colsep "\e\es+" \e \& \-P 2 "echo \*(Aq{1}:{2}\-{3}\*(Aq full\-line: \e\*(Aq{}\e\*(Aq" .Ve .PP https://github.com/brentp/gargs (Last checked: 2016\-08) .SS "\s-1DIFFERENCES BETWEEN\s0 orgalorg \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN orgalorg AND GNU Parallel" \&\fBorgalorg\fR can run the same job on multiple machines. This is related to \fB\-\-onall\fR and \fB\-\-nonall\fR. .PP \&\fBorgalorg\fR supports entering the \s-1SSH\s0 password \- provided it is the same for all servers. \s-1GNU\s0 \fBparallel\fR advocates using \fBssh-agent\fR instead, but it is possible to emulate \fBorgalorg\fR's behavior by setting \s-1SSHPASS\s0 and by using \fB\-\-ssh \*(L"sshpass ssh\*(R"\fR. .PP To make the emulation easier, make a simple alias: .PP .Vb 1 \& alias par_emul="parallel \-j0 \-\-ssh \*(Aqsshpass ssh\*(Aq \-\-nonall \-\-tag \-\-lb" .Ve .PP If you want to supply a password run: .PP .Vb 1 \& SSHPASS=\`ssh\-askpass\` .Ve .PP or set the password directly: .PP .Vb 1 \& SSHPASS=P4$$w0rd! .Ve .PP If the above is set up you can then do: .PP .Vb 2 \& orgalorg \-o frontend1 \-o frontend2 \-p \-C uptime \& par_emul \-S frontend1 \-S frontend2 uptime \& \& orgalorg \-o frontend1 \-o frontend2 \-p \-C top \-bid 1 \& par_emul \-S frontend1 \-S frontend2 top \-bid 1 \& \& orgalorg \-o frontend1 \-o frontend2 \-p \-er /tmp \-n \e \& \*(Aqmd5sum /tmp/bigfile\*(Aq \-S bigfile \& par_emul \-S frontend1 \-S frontend2 \-\-basefile bigfile \e \& \-\-workdir /tmp md5sum /tmp/bigfile .Ve .PP \&\fBorgalorg\fR has a progress indicator for the transferring of a file. \s-1GNU\s0 \fBparallel\fR does not. .PP https://github.com/reconquest/orgalorg (Last checked: 2016\-08) .SS "\s-1DIFFERENCES BETWEEN\s0 Rust parallel(mmstick) \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN Rust parallel(mmstick) AND GNU Parallel" Rust parallel focuses on speed. It is almost as fast as \fBxargs\fR, but not as fast as \fBparallel-bash\fR. It implements a few features from \s-1GNU\s0 \&\fBparallel\fR, but lacks many functions. All these fail: .PP .Vb 4 \& # Read arguments from file \& parallel \-a file echo \& # Changing the delimiter \& parallel \-d _ echo ::: a_b_c_ .Ve .PP These do something different from \s-1GNU\s0 \fBparallel\fR .PP .Vb 10 \& # \-q to protect quoted $ and space \& parallel \-q perl \-e \*(Aq$a=shift; print "$a"x10000000\*(Aq ::: a b c \& # Generation of combination of inputs \& parallel echo {1} {2} ::: red green blue ::: S M L XL XXL \& # {= perl expression =} replacement string \& parallel echo \*(Aq{= s/new/old/ =}\*(Aq ::: my.new your.new \& # \-\-pipe \& seq 100000 | parallel \-\-pipe wc \& # linked arguments \& parallel echo ::: S M L :::+ sml med lrg ::: R G B :::+ red grn blu \& # Run different shell dialects \& zsh \-c \*(Aqparallel echo \e={} ::: zsh && true\*(Aq \& csh \-c \*(Aqparallel echo \e$\e{\e} ::: shell && true\*(Aq \& bash \-c \*(Aqparallel echo \e$\e({}\e) ::: pwd && true\*(Aq \& # Rust parallel does not start before the last argument is read \& (seq 10; sleep 5; echo 2) | time parallel \-j2 \*(Aqsleep 2; echo\*(Aq \& tail \-f /var/log/syslog | parallel echo .Ve .PP Most of the examples from the book \s-1GNU\s0 Parallel 2018 do not work, thus Rust parallel is not close to being a compatible replacement. .PP Rust parallel has no remote facilities. .PP It uses /tmp/parallel for tmp files and does not clean up if terminated abruptly. If another user on the system uses Rust parallel, then /tmp/parallel will have the wrong permissions and Rust parallel will fail. A malicious user can setup the right permissions and symlink the output file to one of the user's files and next time the user uses Rust parallel it will overwrite this file. .PP .Vb 7 \& attacker$ mkdir /tmp/parallel \& attacker$ chmod a+rwX /tmp/parallel \& # Symlink to the file the attacker wants to zero out \& attacker$ ln \-s ~victim/.important\-file /tmp/parallel/stderr_1 \& victim$ seq 1000 | parallel echo \& # This file is now overwritten with stderr from \*(Aqecho\*(Aq \& victim$ cat ~victim/.important\-file .Ve .PP If /tmp/parallel runs full during the run, Rust parallel does not report this, but finishes with success \- thereby risking data loss. .PP https://github.com/mmstick/parallel (Last checked: 2016\-08) .SS "\s-1DIFFERENCES BETWEEN\s0 Rush \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN Rush AND GNU Parallel" \&\fBrush\fR (https://github.com/shenwei356/rush) is written in Go and based on \fBgargs\fR. .PP Just like \s-1GNU\s0 \fBparallel\fR \fBrush\fR buffers in temporary files. But opposite \s-1GNU\s0 \fBparallel\fR \fBrush\fR does not clean up, if the process dies abnormally. .PP \&\fBrush\fR has some string manipulations that can be emulated by putting this into ~/.parallel/config (/ is used instead of %, and % is used instead of ^ as that is closer to bash's ${var%postfix}): .PP .Vb 5 \& \-\-rpl \*(Aq{:} s:(\e.[^/]+)*$::\*(Aq \& \-\-rpl \*(Aq{:%([^}]+?)} s:$$1(\e.[^/]+)*$::\*(Aq \& \-\-rpl \*(Aq{/:%([^}]*?)} s:.*/(.*)$$1(\e.[^/]+)*$:$1:\*(Aq \& \-\-rpl \*(Aq{/:} s:(.*/)?([^/.]+)(\e.[^/]+)*$:$2:\*(Aq \& \-\-rpl \*(Aq{@(.*?)} /$$1/ and $_=$1;\*(Aq .Ve .PP \fI\s-1EXAMPLES FROM\s0 rush's \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM rush's WEBSITE" .PP Here are the examples from \fBrush\fR's website with the equivalent command in \s-1GNU\s0 \fBparallel\fR. .PP \&\fB1. Simple run, quoting is not necessary\fR .PP .Vb 1 \& 1$ seq 1 3 | rush echo {} \& \& 1$ seq 1 3 | parallel echo {} .Ve .PP \&\fB2. Read data from file (`\-i`)\fR .PP .Vb 1 \& 2$ rush echo {} \-i data1.txt \-i data2.txt \& \& 2$ cat data1.txt data2.txt | parallel echo {} .Ve .PP \&\fB3. Keep output order (`\-k`)\fR .PP .Vb 1 \& 3$ seq 1 3 | rush \*(Aqecho {}\*(Aq \-k \& \& 3$ seq 1 3 | parallel \-k echo {} .Ve .PP \&\fB4. Timeout (`\-t`)\fR .PP .Vb 1 \& 4$ time seq 1 | rush \*(Aqsleep 2; echo {}\*(Aq \-t 1 \& \& 4$ time seq 1 | parallel \-\-timeout 1 \*(Aqsleep 2; echo {}\*(Aq .Ve .PP \&\fB5. Retry (`\-r`)\fR .PP .Vb 1 \& 5$ seq 1 | rush \*(Aqpython unexisted_script.py\*(Aq \-r 1 \& \& 5$ seq 1 | parallel \-\-retries 2 \*(Aqpython unexisted_script.py\*(Aq .Ve .PP Use \fB\-u\fR to see it is really run twice: .PP .Vb 1 \& 5$ seq 1 | parallel \-u \-\-retries 2 \*(Aqpython unexisted_script.py\*(Aq .Ve .PP \&\fB6. Dirname (`{/}`) and basename (`{%}`) and remove custom suffix (`{^suffix}`)\fR .PP .Vb 1 \& 6$ echo dir/file_1.txt.gz | rush \*(Aqecho {/} {%} {^_1.txt.gz}\*(Aq \& \& 6$ echo dir/file_1.txt.gz | \& parallel \-\-plus echo {//} {/} {%_1.txt.gz} .Ve .PP \&\fB7. Get basename, and remove last (`{.}`) or any (`{:}`) extension\fR .PP .Vb 1 \& 7$ echo dir.d/file.txt.gz | rush \*(Aqecho {.} {:} {%.} {%:}\*(Aq \& \& 7$ echo dir.d/file.txt.gz | parallel \*(Aqecho {.} {:} {/.} {/:}\*(Aq .Ve .PP \&\fB8. Job \s-1ID,\s0 combine fields index and other replacement strings\fR .PP .Vb 2 \& 8$ echo 12 file.txt dir/s_1.fq.gz | \& rush \*(Aqecho job {#}: {2} {2.} {3%:^_1}\*(Aq \& \& 8$ echo 12 file.txt dir/s_1.fq.gz | \& parallel \-\-colsep \*(Aq \*(Aq \*(Aqecho job {#}: {2} {2.} {3/:%_1}\*(Aq .Ve .PP \&\fB9. Capture submatch using regular expression (`{@regexp}`)\fR .PP .Vb 1 \& 9$ echo read_1.fq.gz | rush \*(Aqecho {@(.+)_\ed}\*(Aq \& \& 9$ echo read_1.fq.gz | parallel \*(Aqecho {@(.+)_\ed}\*(Aq .Ve .PP \&\fB10. Custom field delimiter (`\-d`)\fR .PP .Vb 1 \& 10$ echo a=b=c | rush \*(Aqecho {1} {2} {3}\*(Aq \-d = \& \& 10$ echo a=b=c | parallel \-d = echo {1} {2} {3} .Ve .PP \&\fB11. Send multi-lines to every command (`\-n`)\fR .PP .Vb 1 \& 11$ seq 5 | rush \-n 2 \-k \*(Aqecho "{}"; echo\*(Aq \& \& 11$ seq 5 | \& parallel \-n 2 \-k \e \& \*(Aqecho {=\-1 $_=join"\en",@arg[1..$#arg] =}; echo\*(Aq \& \& 11$ seq 5 | rush \-n 2 \-k \*(Aqecho "{}"; echo\*(Aq \-J \*(Aq \*(Aq \& \& 11$ seq 5 | parallel \-n 2 \-k \*(Aqecho {}; echo\*(Aq .Ve .PP \&\fB12. Custom record delimiter (`\-D`), note that empty records are not used.\fR .PP .Vb 1 \& 12$ echo a b c d | rush \-D " " \-k \*(Aqecho {}\*(Aq \& \& 12$ echo a b c d | parallel \-d " " \-k \*(Aqecho {}\*(Aq \& \& 12$ echo abcd | rush \-D "" \-k \*(Aqecho {}\*(Aq \& \& Cannot be done by GNU Parallel \& \& 12$ cat fasta.fa \& >seq1 \& tag \& >seq2 \& cat \& gat \& >seq3 \& attac \& a \& cat \& \& 12$ cat fasta.fa | rush \-D ">" \e \& \*(Aqecho FASTA record {#}: name: {1} sequence: {2}\*(Aq \-k \-d "\en" \& # rush fails to join the multiline sequences \& \& 12$ cat fasta.fa | (read \-n1 ignore_first_char; \& parallel \-d \*(Aq>\*(Aq \-\-colsep \*(Aq\en\*(Aq echo FASTA record {#}: \e \& name: {1} sequence: \*(Aq{=2 $_=join"",@arg[2..$#arg]=}\*(Aq \& ) .Ve .PP \&\fB13. Assign value to variable, like `awk \-v` (`\-v`)\fR .PP .Vb 2 \& 13$ seq 1 | \& rush \*(Aqecho Hello, {fname} {lname}!\*(Aq \-v fname=Wei \-v lname=Shen \& \& 13$ seq 1 | \& parallel \-N0 \e \& \*(Aqfname=Wei; lname=Shen; echo Hello, ${fname} ${lname}!\*(Aq \& \& 13$ for var in a b; do \e \& 13$ seq 1 3 | rush \-k \-v var=$var \*(Aqecho var: {var}, data: {}\*(Aq; \e \& 13$ done .Ve .PP In \s-1GNU\s0 \fBparallel\fR you would typically do: .PP .Vb 1 \& 13$ seq 1 3 | parallel \-k echo var: {1}, data: {2} ::: a b :::: \- .Ve .PP If you \fIreally\fR want the var: .PP .Vb 2 \& 13$ seq 1 3 | \& parallel \-k var={1} \*(Aq;echo var: $var, data: {}\*(Aq ::: a b :::: \- .Ve .PP If you \fIreally\fR want the \fBfor\fR\-loop: .PP .Vb 4 \& 13$ for var in a b; do \& export var; \& seq 1 3 | parallel \-k \*(Aqecho var: $var, data: {}\*(Aq; \& done .Ve .PP Contrary to \fBrush\fR this also works if the value is complex like: .PP .Vb 1 \& My brother\*(Aqs 12" records .Ve .PP \&\fB14. Preset variable (`\-v`), avoid repeatedly writing verbose replacement strings\fR .PP .Vb 2 \& 14$ # naive way \& echo read_1.fq.gz | rush \*(Aqecho {:^_1} {:^_1}_2.fq.gz\*(Aq \& \& 14$ echo read_1.fq.gz | parallel \*(Aqecho {:%_1} {:%_1}_2.fq.gz\*(Aq \& \& 14$ # macro + removing suffix \& echo read_1.fq.gz | \& rush \-v p=\*(Aq{:^_1}\*(Aq \*(Aqecho {p} {p}_2.fq.gz\*(Aq \& \& 14$ echo read_1.fq.gz | \& parallel \*(Aqp={:%_1}; echo $p ${p}_2.fq.gz\*(Aq \& \& 14$ # macro + regular expression \& echo read_1.fq.gz | rush \-v p=\*(Aq{@(.+?)_\ed}\*(Aq \*(Aqecho {p} {p}_2.fq.gz\*(Aq \& \& 14$ echo read_1.fq.gz | parallel \*(Aqp={@(.+?)_\ed}; echo $p ${p}_2.fq.gz\*(Aq .Ve .PP Contrary to \fBrush\fR \s-1GNU\s0 \fBparallel\fR works with complex values: .PP .Vb 2 \& 14$ echo "My brother\*(Aqs 12\e"read_1.fq.gz" | \& parallel \*(Aqp={@(.+?)_\ed}; echo $p ${p}_2.fq.gz\*(Aq .Ve .PP \&\fB15. Interrupt jobs by `Ctrl\-C`, rush will stop unfinished commands and exit.\fR .PP .Vb 2 \& 15$ seq 1 20 | rush \*(Aqsleep 1; echo {}\*(Aq \& ^C \& \& 15$ seq 1 20 | parallel \*(Aqsleep 1; echo {}\*(Aq \& ^C .Ve .PP \&\fB16. Continue/resume jobs (`\-c`). When some jobs failed (by execution failure, timeout, or canceling by user with `Ctrl + C`), please switch flag `\-c/\-\-continue` on and run again, so that `rush` can save successful commands and ignore them in \f(BI\s-1NEXT\s0\fB run.\fR .PP .Vb 3 \& 16$ seq 1 3 | rush \*(Aqsleep {}; echo {}\*(Aq \-t 3 \-c \& cat successful_cmds.rush \& seq 1 3 | rush \*(Aqsleep {}; echo {}\*(Aq \-t 3 \-c \& \& 16$ seq 1 3 | parallel \-\-joblog mylog \-\-timeout 2 \e \& \*(Aqsleep {}; echo {}\*(Aq \& cat mylog \& seq 1 3 | parallel \-\-joblog mylog \-\-retry\-failed \e \& \*(Aqsleep {}; echo {}\*(Aq .Ve .PP Multi-line jobs: .PP .Vb 5 \& 16$ seq 1 3 | rush \*(Aqsleep {}; echo {}; \e \& echo finish {}\*(Aq \-t 3 \-c \-C finished.rush \& cat finished.rush \& seq 1 3 | rush \*(Aqsleep {}; echo {}; \e \& echo finish {}\*(Aq \-t 3 \-c \-C finished.rush \& \& 16$ seq 1 3 | \& parallel \-\-joblog mylog \-\-timeout 2 \*(Aqsleep {}; echo {}; \e \& echo finish {}\*(Aq \& cat mylog \& seq 1 3 | \& parallel \-\-joblog mylog \-\-retry\-failed \*(Aqsleep {}; echo {}; \e \& echo finish {}\*(Aq .Ve .PP \&\fB17. A comprehensive example: downloading 1K+ pages given by three \s-1URL\s0 list files using `phantomjs save_page.js` (some page contents are dynamically generated by Javascript, so `wget` does not work). Here I set max jobs number (`\-j`) as `20`, each job has a max running time (`\-t`) of `60` seconds and `3` retry changes (`\-r`). Continue flag `\-c` is also switched on, so we can continue unfinished jobs. Luckily, it's accomplished in one run :)\fR .PP .Vb 6 \& 17$ for f in $(seq 2014 2016); do \e \& /bin/rm \-rf $f; mkdir \-p $f; \e \& cat $f.html.txt | rush \-v d=$f \-d = \e \& \*(Aqphantomjs save_page.js "{}" > {d}/{3}.html\*(Aq \e \& \-j 20 \-t 60 \-r 3 \-c; \e \& done .Ve .PP \&\s-1GNU\s0 \fBparallel\fR can append to an existing joblog with '+': .PP .Vb 8 \& 17$ rm mylog \& for f in $(seq 2014 2016); do \& /bin/rm \-rf $f; mkdir \-p $f; \& cat $f.html.txt | \& parallel \-j20 \-\-timeout 60 \-\-retries 4 \-\-joblog +mylog \e \& \-\-colsep = \e \& phantomjs save_page.js {1}={2}={3} \*(Aq>\*(Aq $f/{3}.html \& done .Ve .PP \&\fB18. A bioinformatics example: mapping with `bwa`, and processing result with `samtools`:\fR .PP .Vb 11 \& 18$ ref=ref/xxx.fa \& threads=25 \& ls \-d raw.cluster.clean.mapping/* \e \& | rush \-v ref=$ref \-v j=$threads \-v p=\*(Aq{}/{%}\*(Aq \e \& \*(Aqbwa mem \-t {j} \-M \-a {ref} {p}_1.fq.gz {p}_2.fq.gz >{p}.sam;\e \& samtools view \-bS {p}.sam > {p}.bam; \e \& samtools sort \-T {p}.tmp \-@ {j} {p}.bam \-o {p}.sorted.bam; \e \& samtools index {p}.sorted.bam; \e \& samtools flagstat {p}.sorted.bam > {p}.sorted.bam.flagstat; \e \& /bin/rm {p}.bam {p}.sam;\*(Aq \e \& \-j 2 \-\-verbose \-c \-C mapping.rush .Ve .PP \&\s-1GNU\s0 \fBparallel\fR would use a function: .PP .Vb 10 \& 18$ ref=ref/xxx.fa \& export ref \& thr=25 \& export thr \& bwa_sam() { \& p="$1" \& bam="$p".bam \& sam="$p".sam \& sortbam="$p".sorted.bam \& bwa mem \-t $thr \-M \-a $ref ${p}_1.fq.gz ${p}_2.fq.gz > "$sam" \& samtools view \-bS "$sam" > "$bam" \& samtools sort \-T ${p}.tmp \-@ $thr "$bam" \-o "$sortbam" \& samtools index "$sortbam" \& samtools flagstat "$sortbam" > "$sortbam".flagstat \& /bin/rm "$bam" "$sam" \& } \& export \-f bwa_sam \& ls \-d raw.cluster.clean.mapping/* | \& parallel \-j 2 \-\-verbose \-\-joblog mylog bwa_sam .Ve .PP \fIOther \f(BIrush\fI features\fR .IX Subsection "Other rush features" .PP \&\fBrush\fR has: .IP "\(bu" 4 \&\fBawk \-v\fR like custom defined variables (\fB\-v\fR) .Sp With \s-1GNU\s0 \fBparallel\fR you would simply set a shell variable: .Sp .Vb 2 \& parallel \*(Aqv={}; echo "$v"\*(Aq ::: foo \& echo foo | rush \-v v={} \*(Aqecho {v}\*(Aq .Ve .Sp Also \fBrush\fR does not like special chars. So these \fBdo not work\fR: .Sp .Vb 2 \& echo does not work | rush \-v v=\e" \*(Aqecho {v}\*(Aq \& echo "My brother\*(Aqs 12\e" records" | rush \-v v={} \*(Aqecho {v}\*(Aq .Ve .Sp Whereas the corresponding \s-1GNU\s0 \fBparallel\fR version works: .Sp .Vb 2 \& parallel \*(Aqv=\e"; echo "$v"\*(Aq ::: works \& parallel \*(Aqv={}; echo "$v"\*(Aq ::: "My brother\*(Aqs 12\e" records" .Ve .IP "\(bu" 4 Exit on first error(s) (\-e) .Sp This is called \fB\-\-halt now,fail=1\fR (or shorter: \fB\-\-halt 2\fR) when used with \s-1GNU\s0 \fBparallel\fR. .IP "\(bu" 4 Settable records sending to every command (\fB\-n\fR, default 1) .Sp This is also called \fB\-n\fR in \s-1GNU\s0 \fBparallel\fR. .IP "\(bu" 4 Practical replacement strings .RS 4 .IP "{:} remove any extension" 4 .IX Item "{:} remove any extension" With \s-1GNU\s0 \fBparallel\fR this can be emulated by: .Sp .Vb 1 \& parallel \-\-plus echo \*(Aq{/\e..*/}\*(Aq ::: foo.ext.bar.gz .Ve .IP "{^suffix}, remove suffix" 4 .IX Item "{^suffix}, remove suffix" With \s-1GNU\s0 \fBparallel\fR this can be emulated by: .Sp .Vb 1 \& parallel \-\-plus echo \*(Aq{%.bar.gz}\*(Aq ::: foo.ext.bar.gz .Ve .IP "{@regexp}, capture submatch using regular expression" 4 .IX Item "{@regexp}, capture submatch using regular expression" With \s-1GNU\s0 \fBparallel\fR this can be emulated by: .Sp .Vb 2 \& parallel \-\-rpl \*(Aq{@(.*?)} /$$1/ and $_=$1;\*(Aq \e \& echo \*(Aq{@\ed_(.*).gz}\*(Aq ::: 1_foo.gz .Ve .IP "{%.}, {%:}, basename without extension" 4 .IX Item "{%.}, {%:}, basename without extension" With \s-1GNU\s0 \fBparallel\fR this can be emulated by: .Sp .Vb 1 \& parallel echo \*(Aq{= s:.*/::;s/\e..*// =}\*(Aq ::: dir/foo.bar.gz .Ve .Sp And if you need it often, you define a \fB\-\-rpl\fR in \&\fB\f(CB$HOME\fB/.parallel/config\fR: .Sp .Vb 2 \& \-\-rpl \*(Aq{%.} s:.*/::;s/\e..*//\*(Aq \& \-\-rpl \*(Aq{%:} s:.*/::;s/\e..*//\*(Aq .Ve .Sp Then you can use them as: .Sp .Vb 1 \& parallel echo {%.} {%:} ::: dir/foo.bar.gz .Ve .RE .RS 4 .RE .IP "\(bu" 4 Preset variable (macro) .Sp E.g. .Sp .Vb 1 \& echo foosuffix | rush \-v p={^suffix} \*(Aqecho {p}_new_suffix\*(Aq .Ve .Sp With \s-1GNU\s0 \fBparallel\fR this can be emulated by: .Sp .Vb 2 \& echo foosuffix | \& parallel \-\-plus \*(Aqp={%suffix}; echo ${p}_new_suffix\*(Aq .Ve .Sp Opposite \fBrush\fR \s-1GNU\s0 \fBparallel\fR works fine if the input contains double space, ' and ": .Sp .Vb 2 \& echo "1\*(Aq6\e" foosuffix" | \& parallel \-\-plus \*(Aqp={%suffix}; echo "${p}"_new_suffix\*(Aq .Ve .IP "\(bu" 4 Commands of multi-lines .Sp While you \fIcan\fR use multi-lined commands in \s-1GNU\s0 \fBparallel\fR, to improve readability \s-1GNU\s0 \fBparallel\fR discourages the use of multi-line commands. In most cases it can be written as a function: .Sp .Vb 3 \& seq 1 3 | \& parallel \-\-timeout 2 \-\-joblog my.log \*(Aqsleep {}; echo {}; \e \& echo finish {}\*(Aq .Ve .Sp Could be written as: .Sp .Vb 7 \& doit() { \& sleep "$1" \& echo "$1" \& echo finish "$1" \& } \& export \-f doit \& seq 1 3 | parallel \-\-timeout 2 \-\-joblog my.log doit .Ve .Sp The failed commands can be resumed with: .Sp .Vb 3 \& seq 1 3 | \& parallel \-\-resume\-failed \-\-joblog my.log \*(Aqsleep {}; echo {};\e \& echo finish {}\*(Aq .Ve .PP https://github.com/shenwei356/rush (Last checked: 2017\-05) .SS "\s-1DIFFERENCES BETWEEN\s0 ClusterSSH \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel" ClusterSSH solves a different problem than \s-1GNU\s0 \fBparallel\fR. .PP ClusterSSH opens a terminal window for each computer and using a master window you can run the same command on all the computers. This is typically used for administrating several computers that are almost identical. .PP \&\s-1GNU\s0 \fBparallel\fR runs the same (or different) commands with different arguments in parallel possibly using remote computers to help computing. If more than one computer is listed in \fB\-S\fR \s-1GNU\s0 \fBparallel\fR may only use one of these (e.g. if there are 8 jobs to be run and one computer has 8 cores). .PP \&\s-1GNU\s0 \fBparallel\fR can be used as a poor-man's version of ClusterSSH: .PP \&\fBparallel \-\-nonall \-S server\-a,server\-b do_stuff foo bar\fR .PP https://github.com/duncs/clusterssh (Last checked: 2010\-12) .SS "\s-1DIFFERENCES BETWEEN\s0 coshell \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN coshell AND GNU Parallel" \&\fBcoshell\fR only accepts full commands on standard input. Any quoting needs to be done by the user. .PP Commands are run in \fBsh\fR so any \fBbash\fR/\fBtcsh\fR/\fBzsh\fR specific syntax will not work. .PP Output can be buffered by using \fB\-d\fR. Output is buffered in memory, so big output can cause swapping and therefore be terrible slow or even cause out of memory. .PP https://github.com/gdm85/coshell (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 spread \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN spread AND GNU Parallel" \&\fBspread\fR runs commands on all directories. .PP It can be emulated with \s-1GNU\s0 \fBparallel\fR using this Bash function: .PP .Vb 6 \& spread() { \& _cmds() { \& perl \-e \*(Aq$"=" && ";print "@ARGV"\*(Aq "cd {}" "$@" \& } \& parallel $(_cmds "$@")\*(Aq|| echo exit status $?\*(Aq ::: */ \& } .Ve .PP This works except for the \fB\-\-exclude\fR option. .PP (Last checked: 2017\-11) .SS "\s-1DIFFERENCES BETWEEN\s0 pyargs \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN pyargs AND GNU Parallel" \&\fBpyargs\fR deals badly with input containing spaces. It buffers stdout, but not stderr. It buffers in \s-1RAM.\s0 {} does not work as replacement string. It does not support running functions. .PP \&\fBpyargs\fR does not support composed commands if run with \fB\-\-lines\fR, and fails on \fBpyargs traceroute gnu.org fsf.org\fR. .PP \fIExamples\fR .IX Subsection "Examples" .PP .Vb 2 \& seq 5 | pyargs \-P50 \-L seq \& seq 5 | parallel \-P50 \-\-lb seq \& \& seq 5 | pyargs \-P50 \-\-mark \-L seq \& seq 5 | parallel \-P50 \-\-lb \e \& \-\-tagstring OUTPUT\*(Aq[{= $_=$job\->replaced() =}]\*(Aq seq \& # Similar, but not precisely the same \& seq 5 | parallel \-P50 \-\-lb \-\-tag seq \& \& seq 5 | pyargs \-P50 \-\-mark command \& # Somewhat longer with GNU Parallel due to the special \& # \-\-mark formatting \& cmd="$(echo "command" | parallel \-\-shellquote)" \& wrap_cmd() { \& echo "MARK $cmd $@================================" >&3 \& echo "OUTPUT START[$cmd $@]:" \& eval $cmd "$@" \& echo "OUTPUT END[$cmd $@]" \& } \& (seq 5 | env_parallel \-P2 wrap_cmd) 3>&1 \& # Similar, but not exactly the same \& seq 5 | parallel \-t \-\-tag command \& \& (echo \*(Aq1 2 3\*(Aq;echo 4 5 6) | pyargs \-\-stream seq \& (echo \*(Aq1 2 3\*(Aq;echo 4 5 6) | perl \-pe \*(Aqs/\en/ /\*(Aq | \& parallel \-r \-d\*(Aq \*(Aq seq \& # Similar, but not exactly the same \& parallel seq ::: 1 2 3 4 5 6 .Ve .PP https://github.com/robertblackwell/pyargs (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 concurrently \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN concurrently AND GNU Parallel" \&\fBconcurrently\fR runs jobs in parallel. .PP The output is prepended with the job number, and may be incomplete: .PP .Vb 2 \& $ concurrently \*(Aqseq 100000\*(Aq | (sleep 3;wc \-l) \& 7165 .Ve .PP When pretty printing it caches output in memory. Output mixes by using test \s-1MIX\s0 below whether or not output is cached. .PP There seems to be no way of making a template command and have \&\fBconcurrently\fR fill that with different args. The full commands must be given on the command line. .PP There is also no way of controlling how many jobs should be run in parallel at a time \- i.e. \*(L"number of jobslots\*(R". Instead all jobs are simply started in parallel. .PP https://github.com/kimmobrunfeldt/concurrently (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 map(soveran) \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN map(soveran) AND GNU Parallel" \&\fBmap\fR does not run jobs in parallel by default. The \s-1README\s0 suggests using: .PP .Vb 1 \& ... | map t \*(Aqsleep $t && say done &\*(Aq .Ve .PP But this fails if more jobs are run in parallel than the number of available processes. Since there is no support for parallelization in \&\fBmap\fR itself, the output also mixes: .PP .Vb 1 \& seq 10 | map i \*(Aqecho start\-$i && sleep 0.$i && echo end\-$i &\*(Aq .Ve .PP The major difference is that \s-1GNU\s0 \fBparallel\fR is built for parallelization and \fBmap\fR is not. So \s-1GNU\s0 \fBparallel\fR has lots of ways of dealing with the issues that parallelization raises: .IP "\(bu" 4 Keep the number of processes manageable .IP "\(bu" 4 Make sure output does not mix .IP "\(bu" 4 Make Ctrl-C kill all running processes .PP \fI\s-1EXAMPLES FROM\s0 maps \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM maps WEBSITE" .PP Here are the 5 examples converted to \s-1GNU\s0 Parallel: .PP .Vb 2 \& 1$ ls *.c | map f \*(Aqfoo $f\*(Aq \& 1$ ls *.c | parallel foo \& \& 2$ ls *.c | map f \*(Aqfoo $f; bar $f\*(Aq \& 2$ ls *.c | parallel \*(Aqfoo {}; bar {}\*(Aq \& \& 3$ cat urls | map u \*(Aqcurl \-O $u\*(Aq \& 3$ cat urls | parallel curl \-O \& \& 4$ printf "1\en1\en1\en" | map t \*(Aqsleep $t && say done\*(Aq \& 4$ printf "1\en1\en1\en" | parallel \*(Aqsleep {} && say done\*(Aq \& 4$ parallel \*(Aqsleep {} && say done\*(Aq ::: 1 1 1 \& \& 5$ printf "1\en1\en1\en" | map t \*(Aqsleep $t && say done &\*(Aq \& 5$ printf "1\en1\en1\en" | parallel \-j0 \*(Aqsleep {} && say done\*(Aq \& 5$ parallel \-j0 \*(Aqsleep {} && say done\*(Aq ::: 1 1 1 .Ve .PP https://github.com/soveran/map (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 loop \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN loop AND GNU Parallel" \&\fBloop\fR mixes stdout and stderr: .PP .Vb 1 \& loop \*(Aqls /no\-such\-file\*(Aq >/dev/null .Ve .PP \&\fBloop\fR's replacement string \fB\f(CB$ITEM\fB\fR does not quote strings: .PP .Vb 1 \& echo \*(Aqtwo spaces\*(Aq | loop \*(Aqecho $ITEM\*(Aq .Ve .PP \&\fBloop\fR cannot run functions: .PP .Vb 3 \& myfunc() { echo joe; } \& export \-f myfunc \& loop \*(Aqmyfunc this fails\*(Aq .Ve .PP \fI\s-1EXAMPLES FROM\s0 loop's \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM loop's WEBSITE" .PP Some of the examples from https://github.com/Miserlou/Loop/ can be emulated with \s-1GNU\s0 \fBparallel\fR: .PP .Vb 12 \& # A couple of functions will make the code easier to read \& $ loopy() { \& yes | parallel \-uN0 \-j1 "$@" \& } \& $ export \-f loopy \& $ time_out() { \& parallel \-uN0 \-q \-\-timeout "$@" ::: 1 \& } \& $ match() { \& perl \-0777 \-ne \*(Aqgrep /\*(Aq"$1"\*(Aq/,$_ and print or exit 1\*(Aq \& } \& $ export \-f match \& \& $ loop \*(Aqls\*(Aq \-\-every 10s \& $ loopy \-\-delay 10s ls \& \& $ loop \*(Aqtouch $COUNT.txt\*(Aq \-\-count\-by 5 \& $ loopy touch \*(Aq{= $_=seq()*5 =}\*(Aq.txt \& \& $ loop \-\-until\-contains 200 \-\- \e \& ./get_response_code.sh \-\-site mysite.biz\` \& $ loopy \-\-halt now,success=1 \e \& \*(Aq./get_response_code.sh \-\-site mysite.biz | match 200\*(Aq \& \& $ loop \*(Aq./poke_server\*(Aq \-\-for\-duration 8h \& $ time_out 8h loopy ./poke_server \& \& $ loop \*(Aq./poke_server\*(Aq \-\-until\-success \& $ loopy \-\-halt now,success=1 ./poke_server \& \& $ cat files_to_create.txt | loop \*(Aqtouch $ITEM\*(Aq \& $ cat files_to_create.txt | parallel touch {} \& \& $ loop \*(Aqls\*(Aq \-\-for\-duration 10min \-\-summary \& # \-\-joblog is somewhat more verbose than \-\-summary \& $ time_out 10m loopy \-\-joblog my.log ./poke_server; cat my.log \& \& $ loop \*(Aqecho hello\*(Aq \& $ loopy echo hello \& \& $ loop \*(Aqecho $COUNT\*(Aq \& # GNU Parallel counts from 1 \& $ loopy echo {#} \& # Counting from 0 can be forced \& $ loopy echo \*(Aq{= $_=seq()\-1 =}\*(Aq \& \& $ loop \*(Aqecho $COUNT\*(Aq \-\-count\-by 2 \& $ loopy echo \*(Aq{= $_=2*(seq()\-1) =}\*(Aq \& \& $ loop \*(Aqecho $COUNT\*(Aq \-\-count\-by 2 \-\-offset 10 \& $ loopy echo \*(Aq{= $_=10+2*(seq()\-1) =}\*(Aq \& \& $ loop \*(Aqecho $COUNT\*(Aq \-\-count\-by 1.1 \& # GNU Parallel rounds 3.3000000000000003 to 3.3 \& $ loopy echo \*(Aq{= $_=1.1*(seq()\-1) =}\*(Aq \& \& $ loop \*(Aqecho $COUNT $ACTUALCOUNT\*(Aq \-\-count\-by 2 \& $ loopy echo \*(Aq{= $_=2*(seq()\-1) =} {#}\*(Aq \& \& $ loop \*(Aqecho $COUNT\*(Aq \-\-num 3 \-\-summary \& # \-\-joblog is somewhat more verbose than \-\-summary \& $ seq 3 | parallel \-\-joblog my.log echo; cat my.log \& \& $ loop \*(Aqls \-foobarbatz\*(Aq \-\-num 3 \-\-summary \& # \-\-joblog is somewhat more verbose than \-\-summary \& $ seq 3 | parallel \-\-joblog my.log \-N0 ls \-foobarbatz; cat my.log \& \& $ loop \*(Aqecho $COUNT\*(Aq \-\-count\-by 2 \-\-num 50 \-\-only\-last \& # Can be emulated by running 2 jobs \& $ seq 49 | parallel echo \*(Aq{= $_=2*(seq()\-1) =}\*(Aq >/dev/null \& $ echo 50| parallel echo \*(Aq{= $_=2*(seq()\-1) =}\*(Aq \& \& $ loop \*(Aqdate\*(Aq \-\-every 5s \& $ loopy \-\-delay 5s date \& \& $ loop \*(Aqdate\*(Aq \-\-for\-duration 8s \-\-every 2s \& $ time_out 8s loopy \-\-delay 2s date \& \& $ loop \*(Aqdate \-u\*(Aq \-\-until\-time \*(Aq2018\-05\-25 20:50:00\*(Aq \-\-every 5s \& $ seconds=$((\`date \-d 2019\-05\-25T20:50:00 +%s\` \- \`date +%s\`))s \& $ time_out $seconds loopy \-\-delay 5s date \-u \& \& $ loop \*(Aqecho $RANDOM\*(Aq \-\-until\-contains "666" \& $ loopy \-\-halt now,success=1 \*(Aqecho $RANDOM | match 666\*(Aq \& \& $ loop \*(Aqif (( RANDOM % 2 )); then \& (echo "TRUE"; true); \& else \& (echo "FALSE"; false); \& fi\*(Aq \-\-until\-success \& $ loopy \-\-halt now,success=1 \*(Aqif (( $RANDOM % 2 )); then \& (echo "TRUE"; true); \& else \& (echo "FALSE"; false); \& fi\*(Aq \& \& $ loop \*(Aqif (( RANDOM % 2 )); then \& (echo "TRUE"; true); \& else \& (echo "FALSE"; false); \& fi\*(Aq \-\-until\-error \& $ loopy \-\-halt now,fail=1 \*(Aqif (( $RANDOM % 2 )); then \& (echo "TRUE"; true); \& else \& (echo "FALSE"; false); \& fi\*(Aq \& \& $ loop \*(Aqdate\*(Aq \-\-until\-match "(\ed{4})" \& $ loopy \-\-halt now,success=1 \*(Aqdate | match [0\-9][0\-9][0\-9][0\-9]\*(Aq \& \& $ loop \*(Aqecho $ITEM\*(Aq \-\-for red,green,blue \& $ parallel echo ::: red green blue \& \& $ cat /tmp/my\-list\-of\-files\-to\-create.txt | loop \*(Aqtouch $ITEM\*(Aq \& $ cat /tmp/my\-list\-of\-files\-to\-create.txt | parallel touch \& \& $ ls | loop \*(Aqcp $ITEM $ITEM.bak\*(Aq; ls \& $ ls | parallel cp {} {}.bak; ls \& \& $ loop \*(Aqecho $ITEM | tr a\-z A\-Z\*(Aq \-i \& $ parallel \*(Aqecho {} | tr a\-z A\-Z\*(Aq \& # Or more efficiently: \& $ parallel \-\-pipe tr a\-z A\-Z \& \& $ loop \*(Aqecho $ITEM\*(Aq \-\-for "\`ls\`" \& $ parallel echo {} ::: "\`ls\`" \& \& $ ls | loop \*(Aq./my_program $ITEM\*(Aq \-\-until\-success; \& $ ls | parallel \-\-halt now,success=1 ./my_program {} \& \& $ ls | loop \*(Aq./my_program $ITEM\*(Aq \-\-until\-fail; \& $ ls | parallel \-\-halt now,fail=1 ./my_program {} \& \& $ ./deploy.sh; \& loop \*(Aqcurl \-sw "%{http_code}" http://coolwebsite.biz\*(Aq \e \& \-\-every 5s \-\-until\-contains 200; \& ./announce_to_slack.sh \& $ ./deploy.sh; \& loopy \-\-delay 5s \-\-halt now,success=1 \e \& \*(Aqcurl \-sw "%{http_code}" http://coolwebsite.biz | match 200\*(Aq; \& ./announce_to_slack.sh \& \& $ loop "ping \-c 1 mysite.com" \-\-until\-success; ./do_next_thing \& $ loopy \-\-halt now,success=1 ping \-c 1 mysite.com; ./do_next_thing \& \& $ ./create_big_file \-o my_big_file.bin; \& loop \*(Aqls\*(Aq \-\-until\-contains \*(Aqmy_big_file.bin\*(Aq; \& ./upload_big_file my_big_file.bin \& # inotifywait is a better tool to detect file system changes. \& # It can even make sure the file is complete \& # so you are not uploading an incomplete file \& $ inotifywait \-qmre MOVED_TO \-e CLOSE_WRITE \-\-format %w%f . | \& grep my_big_file.bin \& \& $ ls | loop \*(Aqcp $ITEM $ITEM.bak\*(Aq \& $ ls | parallel cp {} {}.bak \& \& $ loop \*(Aq./do_thing.sh\*(Aq \-\-every 15s \-\-until\-success \-\-num 5 \& $ parallel \-\-retries 5 \-\-delay 15s ::: ./do_thing.sh .Ve .PP https://github.com/Miserlou/Loop/ (Last checked: 2018\-10) .SS "\s-1DIFFERENCES BETWEEN\s0 lorikeet \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN lorikeet AND GNU Parallel" \&\fBlorikeet\fR can run jobs in parallel. It does this based on a dependency graph described in a file, so this is similar to \fBmake\fR. .PP https://github.com/cetra3/lorikeet (Last checked: 2018\-10) .SS "\s-1DIFFERENCES BETWEEN\s0 spp \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN spp AND GNU Parallel" \&\fBspp\fR can run jobs in parallel. \fBspp\fR does not use a command template to generate the jobs, but requires jobs to be in a file. Output from the jobs mix. .PP https://github.com/john01dav/spp (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 paral \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN paral AND GNU Parallel" \&\fBparal\fR prints a lot of status information and stores the output from the commands run into files. This means it cannot be used the middle of a pipe like this .PP .Vb 1 \& paral "echo this" "echo does not" "echo work" | wc .Ve .PP Instead it puts the output into files named like \&\fBout_#_\f(BIcommand\fB.out.log\fR. To get a very similar behaviour with \s-1GNU\s0 \&\fBparallel\fR use \fB\-\-results \&'out_{#}_{=s/[^\esa\-z_0\-9]//g;s/\es+/_/g=}.log' \-\-eta\fR .PP \&\fBparal\fR only takes arguments on the command line and each argument should be a full command. Thus it does not use command templates. .PP This limits how many jobs it can run in total, because they all need to fit on a single command line. .PP \&\fBparal\fR has no support for running jobs remotely. .PP \fI\s-1EXAMPLES FROM README\s0.markdown\fR .IX Subsection "EXAMPLES FROM README.markdown" .PP The examples from \fB\s-1README\s0.markdown\fR and the corresponding command run with \s-1GNU\s0 \fBparallel\fR (\fB\-\-results \&'out_{#}_{=s/[^\esa\-z_0\-9]//g;s/\es+/_/g=}.log' \-\-eta\fR is omitted from the \s-1GNU\s0 \fBparallel\fR command): .PP .Vb 2 \& 1$ paral "command 1" "command 2 \-\-flag" "command arg1 arg2" \& 1$ parallel ::: "command 1" "command 2 \-\-flag" "command arg1 arg2" \& \& 2$ paral "sleep 1 && echo c1" "sleep 2 && echo c2" \e \& "sleep 3 && echo c3" "sleep 4 && echo c4" "sleep 5 && echo c5" \& 2$ parallel ::: "sleep 1 && echo c1" "sleep 2 && echo c2" \e \& "sleep 3 && echo c3" "sleep 4 && echo c4" "sleep 5 && echo c5" \& # Or shorter: \& parallel "sleep {} && echo c{}" ::: {1..5} \& \& 3$ paral \-n=0 "sleep 5 && echo c5" "sleep 4 && echo c4" \e \& "sleep 3 && echo c3" "sleep 2 && echo c2" "sleep 1 && echo c1" \& 3$ parallel ::: "sleep 5 && echo c5" "sleep 4 && echo c4" \e \& "sleep 3 && echo c3" "sleep 2 && echo c2" "sleep 1 && echo c1" \& # Or shorter: \& parallel \-j0 "sleep {} && echo c{}" ::: 5 4 3 2 1 \& \& 4$ paral \-n=1 "sleep 5 && echo c5" "sleep 4 && echo c4" \e \& "sleep 3 && echo c3" "sleep 2 && echo c2" "sleep 1 && echo c1" \& 4$ parallel \-j1 "sleep {} && echo c{}" ::: 5 4 3 2 1 \& \& 5$ paral \-n=2 "sleep 5 && echo c5" "sleep 4 && echo c4" \e \& "sleep 3 && echo c3" "sleep 2 && echo c2" "sleep 1 && echo c1" \& 5$ parallel \-j2 "sleep {} && echo c{}" ::: 5 4 3 2 1 \& \& 6$ paral \-n=5 "sleep 5 && echo c5" "sleep 4 && echo c4" \e \& "sleep 3 && echo c3" "sleep 2 && echo c2" "sleep 1 && echo c1" \& 6$ parallel \-j5 "sleep {} && echo c{}" ::: 5 4 3 2 1 \& \& 7$ paral \-n=1 "echo a && sleep 0.5 && echo b && sleep 0.5 && \e \& echo c && sleep 0.5 && echo d && sleep 0.5 && \e \& echo e && sleep 0.5 && echo f && sleep 0.5 && \e \& echo g && sleep 0.5 && echo h" \& 7$ parallel ::: "echo a && sleep 0.5 && echo b && sleep 0.5 && \e \& echo c && sleep 0.5 && echo d && sleep 0.5 && \e \& echo e && sleep 0.5 && echo f && sleep 0.5 && \e \& echo g && sleep 0.5 && echo h" .Ve .PP https://github.com/amattn/paral (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 concurr \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN concurr AND GNU Parallel" \&\fBconcurr\fR is built to run jobs in parallel using a client/server model. .PP \fI\s-1EXAMPLES FROM README\s0.md\fR .IX Subsection "EXAMPLES FROM README.md" .PP The examples from \fB\s-1README\s0.md\fR: .PP .Vb 2 \& 1$ concurr \*(Aqecho job {#} on slot {%}: {}\*(Aq : arg1 arg2 arg3 arg4 \& 1$ parallel \*(Aqecho job {#} on slot {%}: {}\*(Aq ::: arg1 arg2 arg3 arg4 \& \& 2$ concurr \*(Aqecho job {#} on slot {%}: {}\*(Aq :: file1 file2 file3 \& 2$ parallel \*(Aqecho job {#} on slot {%}: {}\*(Aq :::: file1 file2 file3 \& \& 3$ concurr \*(Aqecho {}\*(Aq < input_file \& 3$ parallel \*(Aqecho {}\*(Aq < input_file \& \& 4$ cat file | concurr \*(Aqecho {}\*(Aq \& 4$ cat file | parallel \*(Aqecho {}\*(Aq .Ve .PP \&\fBconcurr\fR deals badly empty input files and with output larger than 64 \s-1KB.\s0 .PP https://github.com/mmstick/concurr (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 lesser-parallel \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN lesser-parallel AND GNU Parallel" \&\fBlesser-parallel\fR is the inspiration for \fBparallel \-\-embed\fR. Both \&\fBlesser-parallel\fR and \fBparallel \-\-embed\fR define bash functions that can be included as part of a bash script to run jobs in parallel. .PP \&\fBlesser-parallel\fR implements a few of the replacement strings, but hardly any options, whereas \fBparallel \-\-embed\fR gives you the full \&\s-1GNU\s0 \fBparallel\fR experience. .PP https://github.com/kou1okada/lesser\-parallel (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 npm-parallel \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN npm-parallel AND GNU Parallel" \&\fBnpm-parallel\fR can run npm tasks in parallel. .PP There are no examples and very little documentation, so it is hard to compare to \s-1GNU\s0 \fBparallel\fR. .PP https://github.com/spion/npm\-parallel (Last checked: 2019\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 machma \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN machma AND GNU Parallel" \&\fBmachma\fR runs tasks in parallel. It gives time stamped output. It buffers in \s-1RAM.\s0 .PP \fI\s-1EXAMPLES FROM README\s0.md\fR .IX Subsection "EXAMPLES FROM README.md" .PP The examples from \s-1README\s0.md: .PP .Vb 6 \& 1$ # Put shorthand for timestamp in config for the examples \& echo \*(Aq\-\-rpl \*(Aq\e \& \e\*(Aq\*(Aq{time} $_=::strftime("%Y\-%m\-%d %H:%M:%S",localtime())\*(Aq\e\*(Aq \e \& > ~/.parallel/machma \& echo \*(Aq\-\-line\-buffer \-\-tagstring "{#} {time} {}"\*(Aq \e \& >> ~/.parallel/machma \& \& 2$ find . \-iname \*(Aq*.jpg\*(Aq | \& machma \-\- mogrify \-resize 1200x1200 \-filter Lanczos {} \& find . \-iname \*(Aq*.jpg\*(Aq | \& parallel \-\-bar \-Jmachma mogrify \-resize 1200x1200 \e \& \-filter Lanczos {} \& \& 3$ cat /tmp/ips | machma \-p 2 \-\- ping \-c 2 \-q {} \& 3$ cat /tmp/ips | parallel \-j2 \-Jmachma ping \-c 2 \-q {} \& \& 4$ cat /tmp/ips | \& machma \-\- sh \-c \*(Aqping \-c 2 \-q $0 > /dev/null && echo alive\*(Aq {} \& 4$ cat /tmp/ips | \& parallel \-Jmachma \*(Aqping \-c 2 \-q {} > /dev/null && echo alive\*(Aq \& \& 5$ find . \-iname \*(Aq*.jpg\*(Aq | \& machma \-\-timeout 5s \-\- mogrify \-resize 1200x1200 \e \& \-filter Lanczos {} \& 5$ find . \-iname \*(Aq*.jpg\*(Aq | \& parallel \-\-timeout 5s \-\-bar mogrify \-resize 1200x1200 \e \& \-filter Lanczos {} \& \& 6$ find . \-iname \*(Aq*.jpg\*(Aq \-print0 | \& machma \-\-null \-\- mogrify \-resize 1200x1200 \-filter Lanczos {} \& 6$ find . \-iname \*(Aq*.jpg\*(Aq \-print0 | \& parallel \-\-null \-\-bar mogrify \-resize 1200x1200 \e \& \-filter Lanczos {} .Ve .PP https://github.com/fd0/machma (Last checked: 2019\-06) .SS "\s-1DIFFERENCES BETWEEN\s0 interlace \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN interlace AND GNU Parallel" Summary (see legend above): .IP "\- I2 I3 I4 \- \- \-" 4 .IX Item "- I2 I3 I4 - - -" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "\- O2 O3 \- \- \- \- x x" 4 .IX Item "- O2 O3 - - - - x x" .IP "E1 E2 \- \- \- \- \-" 4 .IX Item "E1 E2 - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBinterlace\fR is built for network analysis to run network tools in parallel. .PP \&\fBinterface\fR does not buffer output, so output from different jobs mixes. .PP The overhead for each target is O(n*n), so with 1000 targets it becomes very slow with an overhead in the order of 500ms/target. .PP \fI\s-1EXAMPLES FROM\s0 interlace's \s-1WEBSITE\s0\fR .IX Subsection "EXAMPLES FROM interlace's WEBSITE" .PP Using \fBprips\fR most of the examples from https://github.com/codingo/Interlace can be run with \s-1GNU\s0 \fBparallel\fR: .PP Blocker .PP .Vb 5 \& commands.txt: \& mkdir \-p _output_/_target_/scans/ \& _blocker_ \& nmap _target_ \-oA _output_/_target_/scans/_target_\-nmap \& interlace \-tL ./targets.txt \-cL commands.txt \-o $output \& \& parallel \-a targets.txt \e \& mkdir \-p $output/{}/scans/\e; nmap {} \-oA $output/{}/scans/{}\-nmap .Ve .PP Blocks .PP .Vb 7 \& commands.txt: \& _block:nmap_ \& mkdir \-p _target_/output/scans/ \& nmap _target_ \-oN _target_/output/scans/_target_\-nmap \& _block:nmap_ \& nikto \-\-host _target_ \& interlace \-tL ./targets.txt \-cL commands.txt \& \& _nmap() { \& mkdir \-p $1/output/scans/ \& nmap $1 \-oN $1/output/scans/$1\-nmap \& } \& export \-f _nmap \& parallel ::: _nmap "nikto \-\-host" :::: targets.txt .Ve .PP Run Nikto Over Multiple Sites .PP .Vb 2 \& interlace \-tL ./targets.txt \-threads 5 \e \& \-c "nikto \-\-host _target_ > ./_target_\-nikto.txt" \-v \& \& parallel \-a targets.txt \-P5 nikto \-\-host {} \e> ./{}_\-nikto.txt .Ve .PP Run Nikto Over Multiple Sites and Ports .PP .Vb 3 \& interlace \-tL ./targets.txt \-threads 5 \-c \e \& "nikto \-\-host _target_:_port_ > ./_target_\-_port_\-nikto.txt" \e \& \-p 80,443 \-v \& \& parallel \-P5 nikto \-\-host {1}:{2} \e> ./{1}\-{2}\-nikto.txt \e \& :::: targets.txt ::: 80 443 .Ve .PP Run a List of Commands against Target Hosts .PP .Vb 6 \& commands.txt: \& nikto \-\-host _target_:_port_ > _output_/_target_\-nikto.txt \& sslscan _target_:_port_ > _output_/_target_\-sslscan.txt \& testssl.sh _target_:_port_ > _output_/_target_\-testssl.txt \& interlace \-t example.com \-o ~/Engagements/example/ \e \& \-cL ./commands.txt \-p 80,443 \& \& parallel \-\-results ~/Engagements/example/{2}:{3}{1} {1} {2}:{3} \e \& ::: "nikto \-\-host" sslscan testssl.sh ::: example.com ::: 80 443 .Ve .PP \&\s-1CIDR\s0 notation with an application that doesn't support it .PP .Vb 2 \& interlace \-t 192.168.12.0/24 \-c "vhostscan _target_ \e \& \-oN _output_/_target_\-vhosts.txt" \-o ~/scans/ \-threads 50 \& \& prips 192.168.12.0/24 | \& parallel \-P50 vhostscan {} \-oN ~/scans/{}\-vhosts.txt .Ve .PP Glob notation with an application that doesn't support it .PP .Vb 2 \& interlace \-t 192.168.12.* \-c "vhostscan _target_ \e \& \-oN _output_/_target_\-vhosts.txt" \-o ~/scans/ \-threads 50 \& \& # Glob is not supported in prips \& prips 192.168.12.0/24 | \& parallel \-P50 vhostscan {} \-oN ~/scans/{}\-vhosts.txt .Ve .PP Dash (\-) notation with an application that doesn't support it .PP .Vb 3 \& interlace \-t 192.168.12.1\-15 \-c \e \& "vhostscan _target_ \-oN _output_/_target_\-vhosts.txt" \e \& \-o ~/scans/ \-threads 50 \& \& # Dash notation is not supported in prips \& prips 192.168.12.1 192.168.12.15 | \& parallel \-P50 vhostscan {} \-oN ~/scans/{}\-vhosts.txt .Ve .PP Threading Support for an application that doesn't support it .PP .Vb 3 \& interlace \-tL ./target\-list.txt \-c \e \& "vhostscan \-t _target_ \-oN _output_/_target_\-vhosts.txt" \e \& \-o ~/scans/ \-threads 50 \& \& cat ./target\-list.txt | \& parallel \-P50 vhostscan \-t {} \-oN ~/scans/{}\-vhosts.txt .Ve .PP alternatively .PP .Vb 4 \& ./vhosts\-commands.txt: \& vhostscan \-t $target \-oN _output_/_target_\-vhosts.txt \& interlace \-cL ./vhosts\-commands.txt \-tL ./target\-list.txt \e \& \-threads 50 \-o ~/scans \& \& ./vhosts\-commands.txt: \& vhostscan \-t "$1" \-oN "$2" \& parallel \-P50 ./vhosts\-commands.txt {} ~/scans/{}\-vhosts.txt \e \& :::: ./target\-list.txt .Ve .PP Exclusions .PP .Vb 3 \& interlace \-t 192.168.12.0/24 \-e 192.168.12.0/26 \-c \e \& "vhostscan _target_ \-oN _output_/_target_\-vhosts.txt" \e \& \-o ~/scans/ \-threads 50 \& \& prips 192.168.12.0/24 | grep \-xv \-Ff <(prips 192.168.12.0/26) | \& parallel \-P50 vhostscan {} \-oN ~/scans/{}\-vhosts.txt .Ve .PP Run Nikto Using Multiple Proxies .PP .Vb 3 \& interlace \-tL ./targets.txt \-pL ./proxies.txt \-threads 5 \-c \e \& "nikto \-\-host _target_:_port_ \-useproxy _proxy_ > \e \& ./_target_\-_port_\-nikto.txt" \-p 80,443 \-v \& \& parallel \-j5 \e \& "nikto \-\-host {1}:{2} \-useproxy {3} > ./{1}\-{2}\-nikto.txt" \e \& :::: ./targets.txt ::: 80 443 :::: ./proxies.txt .Ve .PP https://github.com/codingo/Interlace (Last checked: 2019\-09) .SS "\s-1DIFFERENCES BETWEEN\s0 otonvm Parallel \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN otonvm Parallel AND GNU Parallel" I have been unable to get the code to run at all. It seems unfinished. .PP https://github.com/otonvm/Parallel (Last checked: 2019\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 k\-bx par \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN k-bx par AND GNU Parallel" \&\fBpar\fR requires Haskell to work. This limits the number of platforms this can work on. .PP \&\fBpar\fR does line buffering in memory. The memory usage is 3x the longest line (compared to 1x for \fBparallel \-\-lb\fR). Commands must be given as arguments. There is no template. .PP These are the examples from https://github.com/k\-bx/par with the corresponding \s-1GNU\s0 \fBparallel\fR command. .PP .Vb 4 \& par "echo foo; sleep 1; echo foo; sleep 1; echo foo" \e \& "echo bar; sleep 1; echo bar; sleep 1; echo bar" && echo "success" \& parallel \-\-lb ::: "echo foo; sleep 1; echo foo; sleep 1; echo foo" \e \& "echo bar; sleep 1; echo bar; sleep 1; echo bar" && echo "success" \& \& par "echo foo; sleep 1; foofoo" \e \& "echo bar; sleep 1; echo bar; sleep 1; echo bar" && echo "success" \& parallel \-\-lb \-\-halt 1 ::: "echo foo; sleep 1; foofoo" \e \& "echo bar; sleep 1; echo bar; sleep 1; echo bar" && echo "success" \& \& par "PARPREFIX=[fooechoer] echo foo" "PARPREFIX=[bar] echo bar" \& parallel \-\-lb \-\-colsep , \-\-tagstring {1} {2} \e \& ::: "[fooechoer],echo foo" "[bar],echo bar" \& \& par \-\-succeed "foo" "bar" && echo \*(Aqwow\*(Aq \& parallel "foo" "bar"; true && echo \*(Aqwow\*(Aq .Ve .PP https://github.com/k\-bx/par (Last checked: 2019\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 parallelshell \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN parallelshell AND GNU Parallel" \&\fBparallelshell\fR does not allow for composed commands: .PP .Vb 2 \& # This does not work \& parallelshell \*(Aqecho foo;echo bar\*(Aq \*(Aqecho baz;echo quuz\*(Aq .Ve .PP Instead you have to wrap that in a shell: .PP .Vb 1 \& parallelshell \*(Aqsh \-c "echo foo;echo bar"\*(Aq \*(Aqsh \-c "echo baz;echo quuz"\*(Aq .Ve .PP It buffers output in \s-1RAM.\s0 All commands must be given on the command line and all commands are started in parallel at the same time. This will cause the system to freeze if there are so many jobs that there is not enough memory to run them all at the same time. .PP https://github.com/keithamus/parallelshell (Last checked: 2019\-02) .PP https://github.com/darkguy2008/parallelshell (Last checked: 2019\-03) .SS "\s-1DIFFERENCES BETWEEN\s0 shell-executor \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN shell-executor AND GNU Parallel" \&\fBshell-executor\fR does not allow for composed commands: .PP .Vb 2 \& # This does not work \& sx \*(Aqecho foo;echo bar\*(Aq \*(Aqecho baz;echo quuz\*(Aq .Ve .PP Instead you have to wrap that in a shell: .PP .Vb 1 \& sx \*(Aqsh \-c "echo foo;echo bar"\*(Aq \*(Aqsh \-c "echo baz;echo quuz"\*(Aq .Ve .PP It buffers output in \s-1RAM.\s0 All commands must be given on the command line and all commands are started in parallel at the same time. This will cause the system to freeze if there are so many jobs that there is not enough memory to run them all at the same time. .PP https://github.com/royriojas/shell\-executor (Last checked: 2019\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 non-GNU par \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN non-GNU par AND GNU Parallel" \&\fBpar\fR buffers in memory to avoid mixing of jobs. It takes 1s per 1 million output lines. .PP \&\fBpar\fR needs to have all commands before starting the first job. The jobs are read from stdin (standard input) so any quoting will have to be done by the user. .PP Stdout (standard output) is prepended with o:. Stderr (standard error) is sendt to stdout (standard output) and prepended with e:. .PP For short jobs with little output \fBpar\fR is 20% faster than \s-1GNU\s0 \&\fBparallel\fR and 60% slower than \fBxargs\fR. .PP https://github.com/UnixJunkie/PAR .PP https://savannah.nongnu.org/projects/par (Last checked: 2019\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 fd \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN fd AND GNU Parallel" \&\fBfd\fR does not support composed commands, so commands must be wrapped in \fBsh \-c\fR. .PP It buffers output in \s-1RAM.\s0 .PP It only takes file names from the filesystem as input (similar to \fBfind\fR). .PP https://github.com/sharkdp/fd (Last checked: 2019\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 lateral \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN lateral AND GNU Parallel" \&\fBlateral\fR is very similar to \fBsem\fR: It takes a single command and runs it in the background. The design means that output from parallel running jobs may mix. If it dies unexpectly it leaves a socket in ~/.lateral/socket.PID. .PP \&\fBlateral\fR deals badly with too long command lines. This makes the \&\fBlateral\fR server crash: .PP .Vb 1 \& lateral run echo \`seq 100000| head \-c 1000k\` .Ve .PP Any options will be read by \fBlateral\fR so this does not work (\fBlateral\fR interprets the \fB\-l\fR): .PP .Vb 1 \& lateral run ls \-l .Ve .PP Composed commands do not work: .PP .Vb 1 \& lateral run pwd \*(Aq;\*(Aq ls .Ve .PP Functions do not work: .PP .Vb 3 \& myfunc() { echo a; } \& export \-f myfunc \& lateral run myfunc .Ve .PP Running \fBemacs\fR in the terminal causes the parent shell to die: .PP .Vb 5 \& echo \*(Aq#!/bin/bash\*(Aq > mycmd \& echo emacs \-nw >> mycmd \& chmod +x mycmd \& lateral start \& lateral run ./mycmd .Ve .PP Here are the examples from https://github.com/akramer/lateral with the corresponding \s-1GNU\s0 \fBsem\fR and \s-1GNU\s0 \fBparallel\fR commands: .PP .Vb 5 \& 1$ lateral start \& for i in $(cat /tmp/names); do \& lateral run \-\- some_command $i \& done \& lateral wait \& \& 1$ for i in $(cat /tmp/names); do \& sem some_command $i \& done \& sem \-\-wait \& \& 1$ parallel some_command :::: /tmp/names \& \& 2$ lateral start \& for i in $(seq 1 100); do \& lateral run \-\- my_slow_command < workfile$i > /tmp/logfile$i \& done \& lateral wait \& \& 2$ for i in $(seq 1 100); do \& sem my_slow_command < workfile$i > /tmp/logfile$i \& done \& sem \-\-wait \& \& 2$ parallel \*(Aqmy_slow_command < workfile{} > /tmp/logfile{}\*(Aq \e \& ::: {1..100} \& \& 3$ lateral start \-p 0 # yup, it will just queue tasks \& for i in $(seq 1 100); do \& lateral run \-\- command_still_outputs_but_wont_spam inputfile$i \& done \& # command output spam can commence \& lateral config \-p 10; lateral wait \& \& 3$ for i in $(seq 1 100); do \& echo "command inputfile$i" >> joblist \& done \& parallel \-j 10 :::: joblist \& \& 3$ echo 1 > /tmp/njobs \& parallel \-j /tmp/njobs command inputfile{} \e \& ::: {1..100} & \& echo 10 >/tmp/njobs \& wait .Ve .PP https://github.com/akramer/lateral (Last checked: 2019\-03) .SS "\s-1DIFFERENCES BETWEEN\s0 with-this \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN with-this AND GNU Parallel" The examples from https://github.com/amritb/with\-this.git and the corresponding \s-1GNU\s0 \fBparallel\fR command: .PP .Vb 2 \& with \-v "$(cat myurls.txt)" "curl \-L this" \& parallel curl \-L ::: myurls.txt \& \& with \-v "$(cat myregions.txt)" \e \& "aws \-\-region=this ec2 describe\-instance\-status" \& parallel aws \-\-region={} ec2 describe\-instance\-status \e \& :::: myregions.txt \& \& with \-v "$(ls)" "kubectl \-\-kubeconfig=this get pods" \& ls | parallel kubectl \-\-kubeconfig={} get pods \& \& with \-v "$(ls | grep config)" "kubectl \-\-kubeconfig=this get pods" \& ls | grep config | parallel kubectl \-\-kubeconfig={} get pods \& \& with \-v "$(echo {1..10})" "echo 123" \& parallel \-N0 echo 123 ::: {1..10} .Ve .PP Stderr is merged with stdout. \fBwith-this\fR buffers in \s-1RAM.\s0 It uses 3x the output size, so you cannot have output larger than 1/3rd the amount of \s-1RAM.\s0 The input values cannot contain spaces. Composed commands do not work. .PP \&\fBwith-this\fR gives some additional information, so the output has to be cleaned before piping it to the next command. .PP https://github.com/amritb/with\-this.git (Last checked: 2019\-03) .SS "\s-1DIFFERENCES BETWEEN\s0 Tollef's parallel (moreutils) \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN Tollef's parallel (moreutils) AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- I7" 4 .IX Item "- - - I4 - - I7" .PD 0 .IP "\- \- M3 \- \- M6" 4 .IX Item "- - M3 - - M6" .IP "\- O2 O3 \- O5 O6 \- x x" 4 .IX Item "- O2 O3 - O5 O6 - x x" .IP "E1 \- \- \- \- \- E7" 4 .IX Item "E1 - - - - - E7" .IP "\- x x x x x x x x" 4 .IX Item "- x x x x x x x x" .IP "\- \-" 4 .PD .PP \fI\s-1EXAMPLES FROM\s0 Tollef's parallel \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM Tollef's parallel MANUAL" .PP \&\fBTollef\fR parallel sh \-c \*(L"echo hi; sleep 2; echo bye\*(R" \*(-- 1 2 3 .PP \&\fB\s-1GNU\s0\fR parallel \*(L"echo hi; sleep 2; echo bye\*(R" ::: 1 2 3 .PP \&\fBTollef\fR parallel \-j 3 ufraw \-o processed \*(-- *.NEF .PP \&\fB\s-1GNU\s0\fR parallel \-j 3 ufraw \-o processed ::: *.NEF .PP \&\fBTollef\fR parallel \-j 3 \*(-- ls df \*(L"echo hi\*(R" .PP \&\fB\s-1GNU\s0\fR parallel \-j 3 ::: ls df \*(L"echo hi\*(R" .PP (Last checked: 2019\-08) .SS "\s-1DIFFERENCES BETWEEN\s0 rargs \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN rargs AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- I7" 4 .IX Item "I1 - - - - - I7" .PD 0 .IP "\- \- M3 M4 \- \-" 4 .IX Item "- - M3 M4 - -" .IP "\- O2 O3 \- O5 O6 \- O8 \-" 4 .IX Item "- O2 O3 - O5 O6 - O8 -" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBrargs\fR has elegant ways of doing named regexp capture and field ranges. .PP With \s-1GNU\s0 \fBparallel\fR you can use \fB\-\-rpl\fR to get a similar functionality as regexp capture gives, and use \fBjoin\fR and \fB\f(CB@arg\fB\fR to get the field ranges. But the syntax is longer. This: .PP .Vb 1 \& \-\-rpl \*(Aq{r(\ed+)\e.\e.(\ed+)} $_=join"$opt::colsep",@arg[$$1..$$2]\*(Aq .Ve .PP would make it possible to use: .PP .Vb 1 \& {1r3..6} .Ve .PP for field 3..6. .PP For full support of {n..m:s} including negative numbers use a dynamic replacement string like this: .PP .Vb 6 \& PARALLEL=\-\-rpl\e \e\*(Aq\*(Aq{r((\-?\ed+)?)\e.\e.((\-?\ed+)?)((:([^}]*))?)} \& $a = defined $$2 ? $$2 < 0 ? 1+$#arg+$$2 : $$2 : 1; \& $b = defined $$4 ? $$4 < 0 ? 1+$#arg+$$4 : $$4 : $#arg+1; \& $s = defined $$6 ? $$7 : " "; \& $_ = join $s,@arg[$a..$b]\*(Aq\e\*(Aq \& export PARALLEL .Ve .PP You can then do: .PP .Vb 3 \& head /etc/passwd | parallel \-\-colsep : echo ..={1r..} ..3={1r..3} \e \& 4..={1r4..} 2..4={1r2..4} 3..3={1r3..3} ..3:\-={1r..3:\-} \e \& ..3:/={1r..3:/} \-1={\-1} \-5={\-5} \-6={\-6} \-3..={1r\-3..} .Ve .PP \fI\s-1EXAMPLES FROM\s0 rargs \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM rargs MANUAL" .PP .Vb 1 \& 1$ ls *.bak | rargs \-p \*(Aq(.*)\e.bak\*(Aq mv {0} {1} \& \& 1$ ls *.bak | parallel mv {} {.} \& \& 2$ cat download\-list.csv | \& rargs \-p \*(Aq(?P.*),(?P.*)\*(Aq wget {url} \-O {filename} \& \& 2$ cat download\-list.csv | \& parallel \-\-csv wget {1} \-O {2} \& # or use regexps: \& 2$ cat download\-list.csv | \& parallel \-\-rpl \*(Aq{url} s/,.*//\*(Aq \-\-rpl \*(Aq{filename} s/.*?,//\*(Aq \e \& wget {url} \-O {filename} \& \& 3$ cat /etc/passwd | \& rargs \-d: echo \-e \*(Aqid: "{1}"\et name: "{5}"\et rest: "{6..::}"\*(Aq \& \& 3$ cat /etc/passwd | \& parallel \-q \-\-colsep : \e \& echo \-e \*(Aqid: "{1}"\et name: "{5}"\et rest: "{=6 $_=join":",@arg[6..$#arg]=}"\*(Aq .Ve .PP https://github.com/lotabout/rargs (Last checked: 2020\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 threader \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN threader AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- \-" 4 .IX Item "I1 - - - - - -" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "O1 \- O3 \- O5 \- \- x x" 4 .IX Item "O1 - O3 - O5 - - x x" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP Newline separates arguments, but newline at the end of file is treated as an empty argument. So this runs 2 jobs: .PP .Vb 1 \& echo two_jobs | threader \-run \*(Aqecho "$THREADID"\*(Aq .Ve .PP \&\fBthreader\fR ignores stderr, so any output to stderr is lost. \fBthreader\fR buffers in \s-1RAM,\s0 so output bigger than the machine's virtual memory will cause the machine to crash. .PP https://github.com/voodooEntity/threader (Last checked: 2020\-04) .SS "\s-1DIFFERENCES BETWEEN\s0 runp \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN runp AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- \- \- \- \-" 4 .IX Item "I1 I2 - - - - -" .PD 0 .IP "M1 \- (M3) \- \- M6" 4 .IX Item "M1 - (M3) - - M6" .IP "O1 O2 O3 \- O5 O6 \- x x \-" 4 .IX Item "O1 O2 O3 - O5 O6 - x x -" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP (M3): You can add a prefix and a postfix to the input, so it means you can only insert the argument on the command line once. .PP \&\fBrunp\fR runs 10 jobs in parallel by default. \fBrunp\fR blocks if output of a command is > 64 Kbytes. Quoting of input is needed. It adds output to stderr (this can be prevented with \-q) .PP \fIExamples as \s-1GNU\s0 Parallel\fR .IX Subsection "Examples as GNU Parallel" .PP .Vb 5 \& base=\*(Aqhttps://images\-api.nasa.gov/search\*(Aq \& query=\*(Aqjupiter\*(Aq \& desc=\*(Aqplanet\*(Aq \& type=\*(Aqimage\*(Aq \& url="$base?q=$query&description=$desc&media_type=$type" \& \& # Download the images in parallel using runp \& curl \-s $url | jq \-r .collection.items[].href | \e \& runp \-p \*(Aqcurl \-s\*(Aq | jq \-r .[] | grep large | \e \& runp \-p \*(Aqcurl \-s \-L \-O\*(Aq \& \& time curl \-s $url | jq \-r .collection.items[].href | \e \& runp \-g 1 \-q \-p \*(Aqcurl \-s\*(Aq | jq \-r .[] | grep large | \e \& runp \-g 1 \-q \-p \*(Aqcurl \-s \-L \-O\*(Aq \& \& # Download the images in parallel \& curl \-s $url | jq \-r .collection.items[].href | \e \& parallel curl \-s | jq \-r .[] | grep large | \e \& parallel curl \-s \-L \-O \& \& time curl \-s $url | jq \-r .collection.items[].href | \e \& parallel \-j 1 curl \-s | jq \-r .[] | grep large | \e \& parallel \-j 1 curl \-s \-L \-O .Ve .PP Run some test commands (read from file) .IX Subsection "Run some test commands (read from file)" .PP .Vb 7 \& # Create a file containing commands to run in parallel. \& cat << EOF > /tmp/test\-commands.txt \& sleep 5 \& sleep 3 \& blah # this will fail \& ls $PWD # PWD shell variable is used here \& EOF \& \& # Run commands from the file. \& runp /tmp/test\-commands.txt > /dev/null \& \& parallel \-a /tmp/test\-commands.txt > /dev/null .Ve .PP Ping several hosts and see packet loss (read from stdin) .IX Subsection "Ping several hosts and see packet loss (read from stdin)" .PP .Vb 6 \& # First copy this line and press Enter \& runp \-p \*(Aqping \-c 5 \-W 2\*(Aq \-s \*(Aq| grep loss\*(Aq \& localhost \& 1.1.1.1 \& 8.8.8.8 \& # Press Enter and Ctrl\-D when done entering the hosts \& \& # First copy this line and press Enter \& parallel ping \-c 5 \-W 2 {} \*(Aq| grep loss\*(Aq \& localhost \& 1.1.1.1 \& 8.8.8.8 \& # Press Enter and Ctrl\-D when done entering the hosts .Ve .PP Get directories' sizes (read from stdin) .IX Subsection "Get directories' sizes (read from stdin)" .PP .Vb 1 \& echo \-e "$HOME\en/etc\en/tmp" | runp \-q \-p \*(Aqsudo du \-sh\*(Aq \& \& echo \-e "$HOME\en/etc\en/tmp" | parallel sudo du \-sh \& # or: \& parallel sudo du \-sh ::: "$HOME" /etc /tmp .Ve .PP Compress files .IX Subsection "Compress files" .PP .Vb 1 \& find . \-iname \*(Aq*.txt\*(Aq | runp \-p \*(Aqgzip \-\-best\*(Aq \& \& find . \-iname \*(Aq*.txt\*(Aq | parallel gzip \-\-best .Ve .PP Measure \s-1HTTP\s0 request + response time .IX Subsection "Measure HTTP request + response time" .PP .Vb 4 \& export CURL="curl \-w \*(Aqtime_total: %{time_total}\en\*(Aq" \& CURL="$CURL \-o /dev/null \-s https://golang.org/" \& perl \-wE \*(Aqfor (1..10) { say $ENV{CURL} }\*(Aq | \& runp \-q # Make 10 requests \& \& perl \-wE \*(Aqfor (1..10) { say $ENV{CURL} }\*(Aq | parallel \& # or: \& parallel \-N0 "$CURL" ::: {1..10} .Ve .PP Find open \s-1TCP\s0 ports .IX Subsection "Find open TCP ports" .PP .Vb 10 \& cat << EOF > /tmp/host\-port.txt \& localhost 22 \& localhost 80 \& localhost 81 \& 127.0.0.1 443 \& 127.0.0.1 444 \& scanme.nmap.org 22 \& scanme.nmap.org 23 \& scanme.nmap.org 443 \& EOF \& \& 1$ cat /tmp/host\-port.txt | \& runp \-q \-p \*(Aqnetcat \-v \-w2 \-z\*(Aq 2>&1 | egrep \*(Aq(succeeded!|open)$\*(Aq \& \& # \-\-colsep is needed to split the line \& 1$ cat /tmp/host\-port.txt | \& parallel \-\-colsep \*(Aq \*(Aq netcat \-v \-w2 \-z 2>&1 | \& egrep \*(Aq(succeeded!|open)$\*(Aq \& # or use uq for unquoted: \& 1$ cat /tmp/host\-port.txt | \& parallel netcat \-v \-w2 \-z {=uq=} 2>&1 | \& egrep \*(Aq(succeeded!|open)$\*(Aq .Ve .PP https://github.com/jreisinger/runp (Last checked: 2020\-04) .SS "\s-1DIFFERENCES BETWEEN\s0 papply \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN papply AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- \-" 4 .IX Item "- - - I4 - - -" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "\- \- O3 \- O5 \- \- x x O10" 4 .IX Item "- - O3 - O5 - - x x O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBpapply\fR does not print the output if the command fails: .PP .Vb 2 \& $ papply \*(Aqecho %F; false\*(Aq foo \& "echo foo; false" did not succeed .Ve .PP \&\fBpapply\fR's replacement strings (%F \f(CW%d\fR \f(CW%f\fR \f(CW%n\fR \f(CW%e\fR \f(CW%z\fR) can be simulated in \s-1GNU\s0 \&\fBparallel\fR by putting this in \fB~/.parallel/config\fR: .PP .Vb 6 \& \-\-rpl \*(Aq%F\*(Aq \& \-\-rpl \*(Aq%d $_=Q(::dirname($_));\*(Aq \& \-\-rpl \*(Aq%f s:.*/::;\*(Aq \& \-\-rpl \*(Aq%n s:.*/::;s:\e.[^/.]+$::;\*(Aq \& \-\-rpl \*(Aq%e s:.*\e.:.:\*(Aq \& \-\-rpl \*(Aq%z $_=""\*(Aq .Ve .PP \&\fBpapply\fR buffers in \s-1RAM,\s0 and uses twice the amount of output. So output of 5 \s-1GB\s0 takes 10 \s-1GB RAM.\s0 .PP The buffering is very \s-1CPU\s0 intensive: Buffering a line of 5 \s-1GB\s0 takes 40 seconds (compared to 10 seconds with \s-1GNU\s0 \fBparallel\fR). .PP \fIExamples as \s-1GNU\s0 Parallel\fR .IX Subsection "Examples as GNU Parallel" .PP .Vb 1 \& 1$ papply gzip *.txt \& \& 1$ parallel gzip ::: *.txt \& \& 2$ papply "convert %F %n.jpg" *.png \& \& 2$ parallel convert {} {.}.jpg ::: *.png .Ve .PP https://pypi.org/project/papply/ (Last checked: 2020\-04) .SS "\s-1DIFFERENCES BETWEEN\s0 async \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN async AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- I7" 4 .IX Item "- - - I4 - - I7" .PD 0 .IP "\- \- \- \- \- M6" 4 .IX Item "- - - - - M6" .IP "\- O2 O3 \- O5 O6 \- x x O10" 4 .IX Item "- O2 O3 - O5 O6 - x x O10" .IP "E1 \- \- E4 \- E6 \-" 4 .IX Item "E1 - - E4 - E6 -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "S1 S2" 4 .IX Item "S1 S2" .PD .PP \&\fBasync\fR is very similary to \s-1GNU\s0 \fBparallel\fR's \fB\-\-semaphore\fR mode (aka \fBsem\fR). \fBasync\fR requires the user to start a server process. .PP The input is quoted like \fB\-q\fR so you need \fBbash \-c \*(L"...;...\*(R"\fR to run composed commands. .PP \fIExamples as \s-1GNU\s0 Parallel\fR .IX Subsection "Examples as GNU Parallel" .PP .Vb 1 \& 1$ S="/tmp/example_socket" \& \& 1$ ID=myid \& \& 2$ async \-s="$S" server \-\-start \& \& 2$ # GNU Parallel does not need a server to run \& \& 3$ for i in {1..20}; do \& # prints command output to stdout \& async \-s="$S" cmd \-\- bash \-c "sleep 1 && echo test $i" \& done \& \& 3$ for i in {1..20}; do \& # prints command output to stdout \& sem \-\-id "$ID" \-j100% "sleep 1 && echo test $i" \& # GNU Parallel will only print job when it is done \& # If you need output from different jobs to mix \& # use \-u or \-\-line\-buffer \& sem \-\-id "$ID" \-j100% \-\-line\-buffer "sleep 1 && echo test $i" \& done \& \& 4$ # wait until all commands are finished \& async \-s="$S" wait \& \& 4$ sem \-\-id "$ID" \-\-wait \& \& 5$ # configure the server to run four commands in parallel \& async \-s="$S" server \-j4 \& \& 5$ export PARALLEL=\-j4 \& \& 6$ mkdir "/tmp/ex_dir" \& for i in {21..40}; do \& # redirects command output to /tmp/ex_dir/file* \& async \-s="$S" cmd \-o "/tmp/ex_dir/file$i" \-\- \e \& bash \-c "sleep 1 && echo test $i" \& done \& \& 6$ mkdir "/tmp/ex_dir" \& for i in {21..40}; do \& # redirects command output to /tmp/ex_dir/file* \& sem \-\-id "$ID" \-\-result \*(Aq/tmp/my\-ex/file\-{=$_=""=}\*(Aq"$i" \e \& "sleep 1 && echo test $i" \& done \& \& 7$ sem \-\-id "$ID" \-\-wait \& \& 7$ async \-s="$S" wait \& \& 8$ # stops server \& async \-s="$S" server \-\-stop \& \& 8$ # GNU Parallel does not need to stop a server .Ve .PP https://github.com/ctbur/async/ (Last checked: 2023\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 pardi \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN pardi AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- \- \- \- I7" 4 .IX Item "I1 I2 - - - - I7" .PD 0 .IP "M1 \- \- \- \- M6" 4 .IX Item "M1 - - - - M6" .IP "O1 O2 O3 O4 O5 \- O7 \- \- O10" 4 .IX Item "O1 O2 O3 O4 O5 - O7 - - O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBpardi\fR is very similar to \fBparallel \-\-pipe \-\-cat\fR: It reads blocks of data and not arguments. So it cannot insert an argument in the command line. It puts the block into a temporary file, and this file name (%IN) can be put in the command line. You can only use \f(CW%IN\fR once. .PP It can also run full command lines in parallel (like: \fBcat file | parallel\fR). .PP \fI\s-1EXAMPLES FROM\s0 pardi test.sh\fR .IX Subsection "EXAMPLES FROM pardi test.sh" .PP .Vb 3 \& 1$ time pardi \-v \-c 100 \-i data/decoys.smi \-ie .smi \-oe .smi \e \& \-o data/decoys_std_pardi.smi \e \& \-w \*(Aq(standardiser \-i %IN \-o %OUT 2>&1) > /dev/null\*(Aq \& \& 1$ cat data/decoys.smi | \& time parallel \-N 100 \-\-pipe \-\-cat \e \& \*(Aq(standardiser \-i {} \-o {#} 2>&1) > /dev/null; cat {#}; rm {#}\*(Aq \e \& > data/decoys_std_pardi.smi \& \& 2$ pardi \-n 1 \-i data/test_in.types \-o data/test_out.types \e \& \-d \*(Aqr:^#atoms:\*(Aq \-w \*(Aqcat %IN > %OUT\*(Aq \& \& 2$ cat data/test_in.types | \& parallel \-n 1 \-k \-\-pipe \-\-cat \-\-regexp \-\-recstart \*(Aq^#atoms\*(Aq \e \& \*(Aqcat {}\*(Aq > data/test_out.types \& \& 3$ pardi \-c 6 \-i data/test_in.types \-o data/test_out.types \e \& \-d \*(Aqr:^#atoms:\*(Aq \-w \*(Aqcat %IN > %OUT\*(Aq \& \& 3$ cat data/test_in.types | \& parallel \-n 6 \-k \-\-pipe \-\-cat \-\-regexp \-\-recstart \*(Aq^#atoms\*(Aq \e \& \*(Aqcat {}\*(Aq > data/test_out.types \& \& 4$ pardi \-i data/decoys.mol2 \-o data/still_decoys.mol2 \e \& \-d \*(Aqs:@MOLECULE\*(Aq \-w \*(Aqcp %IN %OUT\*(Aq \& \& 4$ cat data/decoys.mol2 | \& parallel \-n 1 \-\-pipe \-\-cat \-\-recstart \*(Aq@MOLECULE\*(Aq \e \& \*(Aqcp {} {#}; cat {#}; rm {#}\*(Aq > data/still_decoys.mol2 \& \& 5$ pardi \-i data/decoys.mol2 \-o data/decoys2.mol2 \e \& \-d b:10000 \-w \*(Aqcp %IN %OUT\*(Aq \-\-preserve \& \& 5$ cat data/decoys.mol2 | \& parallel \-k \-\-pipe \-\-block 10k \-\-recend \*(Aq\*(Aq \-\-cat \e \& \*(Aqcat {} > {#}; cat {#}; rm {#}\*(Aq > data/decoys2.mol2 .Ve .PP https://github.com/UnixJunkie/pardi (Last checked: 2021\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 bthread \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN bthread AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- \-" 4 .IX Item "- - - I4 - - -" .PD 0 .IP "\- \- \- \- \- M6" 4 .IX Item "- - - - - M6" .IP "O1 \- O3 \- \- \- O7 O8 \- \-" 4 .IX Item "O1 - O3 - - - O7 O8 - -" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBbthread\fR takes around 1 sec per \s-1MB\s0 of output. The maximal output line length is 1073741759. .PP You cannot quote space in the command, so you cannot run composed commands like \fBsh \-c \*(L"echo a; echo b\*(R"\fR. .PP https://gitlab.com/netikras/bthread (Last checked: 2021\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 simple_gpu_scheduler \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- I7" 4 .IX Item "I1 - - - - - I7" .PD 0 .IP "M1 \- \- \- \- M6" 4 .IX Item "M1 - - - - M6" .IP "\- O2 O3 \- \- O6 \- x x O10" 4 .IX Item "- O2 O3 - - O6 - x x O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \fI\s-1EXAMPLES FROM\s0 simple_gpu_scheduler \s-1MANUAL\s0\fR .IX Subsection "EXAMPLES FROM simple_gpu_scheduler MANUAL" .PP .Vb 1 \& 1$ simple_gpu_scheduler \-\-gpus 0 1 2 < gpu_commands.txt \& \& 1$ parallel \-j3 \-\-shuf \e \& CUDA_VISIBLE_DEVICES=\*(Aq{=1 $_=slot()\-1 =} {=uq;=}\*(Aq \e \& < gpu_commands.txt \& \& 2$ simple_hypersearch \e \& "python3 train_dnn.py \-\-lr {lr} \-\-batch_size {bs}" \e \& \-p lr 0.001 0.0005 0.0001 \-p bs 32 64 128 | \& simple_gpu_scheduler \-\-gpus 0,1,2 \& \& 2$ parallel \-\-header : \-\-shuf \-j3 \-v \e \& CUDA_VISIBLE_DEVICES=\*(Aq{=1 $_=slot()\-1 =}\*(Aq \e \& python3 train_dnn.py \-\-lr {lr} \-\-batch_size {bs} \e \& ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128 \& \& 3$ simple_hypersearch \e \& "python3 train_dnn.py \-\-lr {lr} \-\-batch_size {bs}" \e \& \-\-n\-samples 5 \-p lr 0.001 0.0005 0.0001 \-p bs 32 64 128 | \& simple_gpu_scheduler \-\-gpus 0,1,2 \& \& 3$ parallel \-\-header : \-\-shuf \e \& CUDA_VISIBLE_DEVICES=\*(Aq{=1 $_=slot()\-1; seq()>5 and skip() =}\*(Aq \e \& python3 train_dnn.py \-\-lr {lr} \-\-batch_size {bs} \e \& ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128 \& \& 4$ touch gpu.queue \& tail \-f \-n 0 gpu.queue | simple_gpu_scheduler \-\-gpus 0,1,2 & \& echo "my_command_with | and stuff > logfile" >> gpu.queue \& \& 4$ touch gpu.queue \& tail \-f \-n 0 gpu.queue | \& parallel \-j3 CUDA_VISIBLE_DEVICES=\*(Aq{=1 $_=slot()\-1 =} {=uq;=}\*(Aq & \& # Needed to fill job slots once \& seq 3 | parallel echo true >> gpu.queue \& # Add jobs \& echo "my_command_with | and stuff > logfile" >> gpu.queue \& # Needed to flush output from completed jobs \& seq 3 | parallel echo true >> gpu.queue .Ve .PP https://github.com/ExpectationMax/simple_gpu_scheduler (Last checked: 2021\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 parasweep \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN parasweep AND GNU Parallel" \&\fBparasweep\fR is a Python module for facilitating parallel parameter sweeps. .PP A \fBparasweep\fR job will normally take a text file as input. The text file contains arguments for the job. Some of these arguments will be fixed and some of them will be changed by \fBparasweep\fR. .PP It does this by having a template file such as template.txt: .PP .Vb 7 \& Xval: {x} \& Yval: {y} \& FixedValue: 9 \& # x with 2 decimals \& DecimalX: {x:.2f} \& TenX: ${x*10} \& RandomVal: {r} .Ve .PP and from this template it generates the file to be used by the job by replacing the replacement strings. .PP Being a Python module \fBparasweep\fR integrates tighter with Python than \&\s-1GNU\s0 \fBparallel\fR. You get the parameters directly in a Python data structure. With \s-1GNU\s0 \fBparallel\fR you can use the \s-1JSON\s0 or \s-1CSV\s0 output format to get something similar, but you would have to read the output. .PP \&\fBparasweep\fR has a filtering method to ignore parameter combinations you do not need. .PP Instead of calling the jobs directly, \fBparasweep\fR can use Python's Distributed Resource Management Application \s-1API\s0 to make jobs run with different cluster software. .PP \&\s-1GNU\s0 \fBparallel\fR \fB\-\-tmpl\fR supports templates with replacement strings. Such as: .PP .Vb 7 \& Xval: {x} \& Yval: {y} \& FixedValue: 9 \& # x with 2 decimals \& DecimalX: {=x $_=sprintf("%.2f",$_) =} \& TenX: {=x $_=$_*10 =} \& RandomVal: {=1 $_=rand() =} .Ve .PP that can be used like: .PP .Vb 2 \& parallel \-\-header : \-\-tmpl my.tmpl={#}.t myprog {#}.t \e \& ::: x 1 2 3 ::: y 1 2 3 .Ve .PP Filtering is supported as: .PP .Vb 1 \& parallel \-\-filter \*(Aq{1} > {2}\*(Aq echo ::: 1 2 3 ::: 1 2 3 .Ve .PP https://github.com/eviatarbach/parasweep (Last checked: 2021\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 parallel-bash \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN parallel-bash AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- \- \- \- \-" 4 .IX Item "I1 I2 - - - - -" .PD 0 .IP "\- \- M3 \- \- M6" 4 .IX Item "- - M3 - - M6" .IP "\- O2 O3 \- O5 O6 \- O8 x O10" 4 .IX Item "- O2 O3 - O5 O6 - O8 x O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBparallel-bash\fR is written in pure bash. It is really fast (overhead of ~0.05 ms/job compared to \s-1GNU\s0 \fBparallel\fR's 3\-10 ms/job). So if your jobs are extremely short lived, and you can live with the quite limited command, this may be useful. .PP It works by making a queue for each process. Then the jobs are distributed to the queues in a round robin fashion. Finally the queues are started in parallel. This works fine, if you are lucky, but if not, all the long jobs may end up in the same queue, so you may see: .PP .Vb 6 \& $ printf "%b\en" 1 1 1 4 1 1 1 4 1 1 1 4 | \& time parallel \-P4 sleep {} \& (7 seconds) \& $ printf "%b\en" 1 1 1 4 1 1 1 4 1 1 1 4 | \& time ./parallel\-bash.bash \-p 4 \-c sleep {} \& (12 seconds) .Ve .PP Because it uses bash lists, the total number of jobs is limited to 167000..265000 depending on your environment. You get a segmentation fault, when you reach the limit. .PP Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend running jobs. .PP \fI\s-1EXAMPLES FROM\s0 parallel-bash\fR .IX Subsection "EXAMPLES FROM parallel-bash" .PP .Vb 1 \& 1$ some_input | parallel\-bash \-p 5 \-c echo \& \& 1$ some_input | parallel \-j 5 echo \& \& 2$ parallel\-bash \-p 5 \-c echo < some_file \& \& 2$ parallel \-j 5 echo < some_file \& \& 3$ parallel\-bash \-p 5 \-c echo <<< \*(Aqsome string\*(Aq \& \& 3$ parallel \-j 5 \-c echo <<< \*(Aqsome string\*(Aq \& \& 4$ something | parallel\-bash \-p 5 \-c echo {} {} \& \& 4$ something | parallel \-j 5 echo {} {} .Ve .PP https://reposhub.com/python/command\-line\-tools/Akianonymus\-parallel\-bash.html (Last checked: 2021\-06) .SS "\s-1DIFFERENCES BETWEEN\s0 bash-concurrent \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel" \&\fBbash-concurrent\fR is more an alternative to \fBmake\fR than to \s-1GNU\s0 \&\fBparallel\fR. Its input is very similar to a Makefile, where jobs depend on other jobs. .PP It has a nice progress indicator where you can see which jobs completed successfully, which jobs are currently running, which jobs failed, and which jobs were skipped due to a depending job failed. The indicator does not deal well with resizing the window. .PP Output is cached in tempfiles on disk, but is only shown if there is an error, so it is not meant to be part of a \s-1UNIX\s0 pipeline. If \&\fBbash-concurrent\fR crashes these tempfiles are not removed. .PP It uses an O(n*n) algorithm, so if you have 1000 independent jobs it takes 22 seconds to start it. .PP https://github.com/themattrix/bash\-concurrent (Last checked: 2021\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 spawntool \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN spawntool AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- \-" 4 .IX Item "I1 - - - - - -" .PD 0 .IP "M1 \- \- \- \- M6" 4 .IX Item "M1 - - - - M6" .IP "\- O2 O3 \- O5 O6 \- x x O10" 4 .IX Item "- O2 O3 - O5 O6 - x x O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBspawn\fR reads a full command line from stdin which it executes in parallel. .PP http://code.google.com/p/spawntool/ (Last checked: 2021\-07) .SS "\s-1DIFFERENCES BETWEEN\s0 go-pssh \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN go-pssh AND GNU Parallel" Summary (see legend above): .IP "\- \- \- \- \- \- \-" 4 .PD 0 .IP "M1 \- \- \- \- \-" 4 .IX Item "M1 - - - - -" .IP "O1 \- \- \- \- \- \- x x O10" 4 .IX Item "O1 - - - - - - x x O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "R1 R2 \- \- \- R6 \- \- \-" 4 .IX Item "R1 R2 - - - R6 - - -" .IP "\- \-" 4 .PD .PP \&\fBgo-pssh\fR does \fBssh\fR in parallel to multiple machines. It runs the same command on multiple machines similar to \fB\-\-nonall\fR. .PP The hostnames must be given as IP-addresses (not as hostnames). .PP Output is sent to stdout (standard output) if command is successful, and to stderr (standard error) if the command fails. .PP \fI\s-1EXAMPLES FROM\s0 go-pssh\fR .IX Subsection "EXAMPLES FROM go-pssh" .PP .Vb 1 \& 1$ go\-pssh \-l , \-u \-p \-P \-c "" \& \& 1$ parallel \-S \*(Aqsshpass \-p ssh \-p @\*(Aq \e \& \-\-nonall "" \& \& 2$ go\-pssh scp \-f host.txt \-u \-p \-P \e \& \-s /local/file_or_directory \-d /remote/directory \& \& 2$ parallel \-\-nonall \-\-slf host.txt \e \& \-\-basefile /local/file_or_directory/./ \-\-wd /remote/directory \& \-\-ssh \*(Aqsshpass \-p ssh \-p \-l \*(Aq true \& \& 3$ go\-pssh scp \-l , \-u \-p \-P \e \& \-s /local/file_or_directory \-d /remote/directory \& \& 3$ parallel \-\-nonall \-S , \e \& \-\-basefile /local/file_or_directory/./ \-\-wd /remote/directory \& \-\-ssh \*(Aqsshpass \-p ssh \-p \-l \*(Aq true .Ve .PP https://github.com/xuchenCN/go\-pssh (Last checked: 2021\-07) .SS "\s-1DIFFERENCES BETWEEN\s0 go-parallel \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN go-parallel AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- \- \- \- I7" 4 .IX Item "I1 I2 - - - - I7" .PD 0 .IP "\- \- M3 \- \- M6" 4 .IX Item "- - M3 - - M6" .IP "\- O2 O3 \- O5 \- \- x x \- O10" 4 .IX Item "- O2 O3 - O5 - - x x - O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBgo-parallel\fR uses Go templates for replacement strings. Quite similar to the \fI{= perl expr =}\fR replacement string. .PP \fI\s-1EXAMPLES FROM\s0 go-parallel\fR .IX Subsection "EXAMPLES FROM go-parallel" .PP .Vb 1 \& 1$ go\-parallel \-a ./files.txt \-t \*(Aqcp {{.Input}} {{.Input | dirname | dirname}}\*(Aq \& \& 1$ parallel \-a ./files.txt cp {} \*(Aq{= $_=::dirname(::dirname($_)) =}\*(Aq \& \& 2$ go\-parallel \-a ./files.txt \-t \*(Aqmkdir \-p {{.Input}} {{noExt .Input}}\*(Aq \& \& 2$ parallel \-a ./files.txt echo mkdir \-p {} {.} \& \& 3$ go\-parallel \-a ./files.txt \-t \*(Aqmkdir \-p {{.Input}} {{.Input | basename | noExt}}\*(Aq \& \& 3$ parallel \-a ./files.txt echo mkdir \-p {} {/.} .Ve .PP https://github.com/mylanconnolly/parallel (Last checked: 2021\-07) .SS "\s-1DIFFERENCES BETWEEN\s0 p \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN p AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- x" 4 .IX Item "- - - I4 - - x" .PD 0 .IP "\- \- \- \- \- M6" 4 .IX Item "- - - - - M6" .IP "\- O2 O3 \- O5 O6 \- x x \- O10" 4 .IX Item "- O2 O3 - O5 O6 - x x - O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBp\fR is a tiny shell script. It can color output with some predefined colors, but is otherwise quite limited. .PP It maxes out at around 116000 jobs (probably due to limitations in Bash). .PP \fI\s-1EXAMPLES FROM\s0 p\fR .IX Subsection "EXAMPLES FROM p" .PP Some of the examples from \fBp\fR cannot be implemented 100% by \s-1GNU\s0 \&\fBparallel\fR: The coloring is a bit different, and \s-1GNU\s0 \fBparallel\fR cannot have \fB\-\-tag\fR for some inputs and not for others. .PP The coloring done by \s-1GNU\s0 \fBparallel\fR is not exactly the same as \fBp\fR. .PP .Vb 2 \& 1$ p \-bc blue "ping 127.0.0.1" \-uc red "ping 192.168.0.1" \e \& \-rc yellow "ping 192.168.1.1" \-t example "ping example.com" \& \& 1$ parallel \-\-lb \-j0 \-\-color \-\-tag ping \e \& ::: 127.0.0.1 192.168.0.1 192.168.1.1 example.com \& \& 2$ p "tail \-f /var/log/httpd/access_log" \e \& \-bc red "tail \-f /var/log/httpd/error_log" \& \& 2$ cd /var/log/httpd; \& parallel \-\-lb \-\-color \-\-tag tail \-f ::: access_log error_log \& \& 3$ p tail \-f "some file" \e& p tail \-f "other file with space.txt" \& \& 3$ parallel \-\-lb tail \-f ::: \*(Aqsome file\*(Aq "other file with space.txt" \& \& 4$ p \-t project1 "hg pull project1" \-t project2 \e \& "hg pull project2" \-t project3 "hg pull project3" \& \& 4$ parallel \-\-lb hg pull ::: project{1..3} .Ve .PP https://github.com/rudymatela/evenmoreutils/blob/master/man/p.1.adoc (Last checked: 2022\-04) .SS "\s-1DIFFERENCES BETWEEN\s0 senechal \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN senechal AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- \-" 4 .IX Item "I1 - - - - - -" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "O1 \- O3 O4 \- \- \- x x \-" 4 .IX Item "O1 - O3 O4 - - - x x -" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBseneschal\fR only starts the first job after reading the last job, and output from the first job is only printed after the last job finishes. .PP 1 byte of output requites 3.5 bytes of \s-1RAM.\s0 .PP This makes it impossible to have a total output bigger than the virtual memory. .PP Even though output is kept in \s-1RAM\s0 outputing is quite slow: 30 MB/s. .PP Output larger than 4 \s-1GB\s0 causes random problems \- it looks like a race condition. .PP This: .PP .Vb 1 \& echo 1 | seneschal \-\-prefix=\*(Aqyes \`seq 1000\`|head \-c 1G\*(Aq >/dev/null .Ve .PP takes 4100(!) \s-1CPU\s0 seconds to run on a 64C64T server, but only 140 \s-1CPU\s0 seconds on a 4C8T laptop. So it looks like \fBseneschal\fR wastes a lot of \s-1CPU\s0 time coordinating the CPUs. .PP Compare this to: .PP .Vb 1 \& echo 1 | time \-v parallel \-N0 \*(Aqyes \`seq 1000\`|head \-c 1G\*(Aq >/dev/null .Ve .PP which takes 3\-8 \s-1CPU\s0 seconds. .PP \fI\s-1EXAMPLES FROM\s0 seneschal \s-1README\s0.md\fR .IX Subsection "EXAMPLES FROM seneschal README.md" .PP .Vb 1 \& 1$ echo $REPOS | seneschal \-\-prefix="cd {} && git pull" \& \& # If $REPOS is newline separated \& 1$ echo "$REPOS" | parallel \-k "cd {} && git pull" \& # If $REPOS is space separated \& 1$ echo \-n "$REPOS" | parallel \-d\*(Aq \*(Aq \-k "cd {} && git pull" \& \& COMMANDS="pwd \& sleep 5 && echo boom \& echo Howdy \& whoami" \& \& 2$ echo "$COMMANDS" | seneschal \-\-debug \& \& 2$ echo "$COMMANDS" | parallel \-k \-v \& \& 3$ ls \-1 | seneschal \-\-prefix="pushd {}; git pull; popd;" \& \& 3$ ls \-1 | parallel \-k "pushd {}; git pull; popd;" \& # Or if current dir also contains files: \& 3$ parallel \-k "pushd {}; git pull; popd;" ::: */ .Ve .PP https://github.com/TheWizardTower/seneschal (Last checked: 2022\-06) .SS "\s-1DIFFERENCES BETWEEN\s0 async \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN async AND GNU Parallel" Summary (see legend above): .IP "x x x x x x x" 4 .IX Item "x x x x x x x" .PD 0 .IP "\- x x x x x" 4 .IX Item "- x x x x x" .IP "x O2 O3 O4 O5 O6 \- x x O10" 4 .IX Item "x O2 O3 O4 O5 O6 - x x O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "S1 S2" 4 .IX Item "S1 S2" .PD .PP \&\fBasync\fR works like \fBsem\fR. .PP \fI\s-1EXAMPLES FROM\s0 async\fR .IX Subsection "EXAMPLES FROM async" .PP .Vb 1 \& 1$ S="/tmp/example_socket" \& \& async \-s="$S" server \-\-start \& \& for i in {1..20}; do \& # prints command output to stdout \& async \-s="$S" cmd \-\- bash \-c "sleep 1 && echo test $i" \& done \& \& # wait until all commands are finished \& async \-s="$S" wait \& \& 1$ S="example_id" \& \& # server not needed \& \& for i in {1..20}; do \& # prints command output to stdout \& sem \-\-bg \-\-id "$S" \-j100% "sleep 1 && echo test $i" \& done \& \& # wait until all commands are finished \& sem \-\-fg \-\-id "$S" \-\-wait \& \& 2$ # configure the server to run four commands in parallel \& async \-s="$S" server \-j4 \& \& mkdir "/tmp/ex_dir" \& for i in {21..40}; do \& # redirects command output to /tmp/ex_dir/file* \& async \-s="$S" cmd \-o "/tmp/ex_dir/file$i" \-\- \e \& bash \-c "sleep 1 && echo test $i" \& done \& \& async \-s="$S" wait \& \& # stops server \& async \-s="$S" server \-\-stop \& \& 2$ # starting server not needed \& \& mkdir "/tmp/ex_dir" \& for i in {21..40}; do \& # redirects command output to /tmp/ex_dir/file* \& sem \-\-bg \-\-id "$S" \-\-results "/tmp/ex_dir/file$i{}" \e \& "sleep 1 && echo test $i" \& done \& \& sem \-\-fg \-\-id "$S" \-\-wait \& \& # there is no server to stop .Ve .PP https://github.com/ctbur/async (Last checked: 2023\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 tandem \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN tandem AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- x" 4 .IX Item "- - - I4 - - x" .PD 0 .IP "M1 \- \- \- \- M6" 4 .IX Item "M1 - - - - M6" .IP "\- \- O3 \- \- \- \- x \- \-" 4 .IX Item "- - O3 - - - - x - -" .IP "E1 \- E3 \- E5 \- \-" 4 .IX Item "E1 - E3 - E5 - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBtandem\fR runs full commands in parallel. It is made for starting a \&\*(L"server\*(R", running a job against the server, and when the job is done, the server is killed. .PP More generally: it kills all jobs when the first job completes \- similar to '\-\-halt now,done=1'. .PP \&\fBtandem\fR silently discards some output. It is unclear exactly when this happens. It looks like a race condition, because it varies for each run. .PP .Vb 2 \& $ tandem "seq 10000" | wc \-l \& 6731 <\- This should always be 10002 .Ve .PP \fI\s-1EXAMPLES FROM\s0 Demo\fR .IX Subsection "EXAMPLES FROM Demo" .PP .Vb 4 \& tandem \e \& \*(Aqphp \-S localhost:8000\*(Aq \e \& \*(Aqesbuild src/*.ts \-\-bundle \-\-outdir=dist \-\-watch\*(Aq \e \& \*(Aqtailwind \-i src/index.css \-o dist/index.css \-\-watch\*(Aq \& \& # Emulate tandem\*(Aqs behaviour \& PARALLEL=\*(Aq\-\-color \-\-lb \-\-halt now,done=1 \-\-tagstring \*(Aq \& PARALLEL="$PARALLEL\*(Aq"\*(Aq{=s/ .*//; $_.=".".$app{$_}++;=}\*(Aq"\*(Aq" \& export PARALLEL \& \& parallel ::: \e \& \*(Aqphp \-S localhost:8000\*(Aq \e \& \*(Aqesbuild src/*.ts \-\-bundle \-\-outdir=dist \-\-watch\*(Aq \e \& \*(Aqtailwind \-i src/index.css \-o dist/index.css \-\-watch\*(Aq .Ve .PP \fI\s-1EXAMPLES FROM\s0 tandem \-h\fR .IX Subsection "EXAMPLES FROM tandem -h" .PP .Vb 4 \& # Emulate tandem\*(Aqs behaviour \& PARALLEL=\*(Aq\-\-color \-\-lb \-\-halt now,done=1 \-\-tagstring \*(Aq \& PARALLEL="$PARALLEL\*(Aq"\*(Aq{=s/ .*//; $_.=".".$app{$_}++;=}\*(Aq"\*(Aq" \& export PARALLEL \& \& 1$ tandem \*(Aqsleep 5 && echo "hello"\*(Aq \*(Aqsleep 2 && echo "world"\*(Aq \& \& 1$ parallel ::: \*(Aqsleep 5 && echo "hello"\*(Aq \*(Aqsleep 2 && echo "world"\*(Aq \& \& # \*(Aq\-t 0\*(Aq fails. But \*(Aq\-\-timeout 0 works\*(Aq \& 2$ tandem \-\-timeout 0 \*(Aqsleep 5 && echo "hello"\*(Aq \e \& \*(Aqsleep 2 && echo "world"\*(Aq \& \& 2$ parallel \-\-timeout 0 ::: \*(Aqsleep 5 && echo "hello"\*(Aq \e \& \*(Aqsleep 2 && echo "world"\*(Aq .Ve .PP \fI\s-1EXAMPLES FROM\s0 tandem's readme.md\fR .IX Subsection "EXAMPLES FROM tandem's readme.md" .PP .Vb 4 \& # Emulate tandem\*(Aqs behaviour \& PARALLEL=\*(Aq\-\-color \-\-lb \-\-halt now,done=1 \-\-tagstring \*(Aq \& PARALLEL="$PARALLEL\*(Aq"\*(Aq{=s/ .*//; $_.=".".$app{$_}++;=}\*(Aq"\*(Aq" \& export PARALLEL \& \& 1$ tandem \*(Aqnext dev\*(Aq \*(Aqnodemon \-\-quiet ./server.js\*(Aq \& \& 1$ parallel ::: \*(Aqnext dev\*(Aq \*(Aqnodemon \-\-quiet ./server.js\*(Aq \& \& 2$ cat package.json \& { \& "scripts": { \& "dev:php": "...", \& "dev:js": "...", \& "dev:css": "..." \& } \& } \& \& tandem \*(Aqnpm:dev:php\*(Aq \*(Aqnpm:dev:js\*(Aq \*(Aqnpm:dev:css\*(Aq \& \& # GNU Parallel uses bash functions instead \& 2$ cat package.sh \& dev:php() { ... ; } \& dev:js() { ... ; } \& dev:css() { ... ; } \& export \-f dev:php dev:js dev:css \& \& . package.sh \& parallel ::: dev:php dev:js dev:css \& \& 3$ tandem \*(Aqnpm:dev:*\*(Aq \& \& 3$ compgen \-A function | grep ^dev: | parallel .Ve .PP For usage in Makefiles, include a copy of \s-1GNU\s0 Parallel with your source using `parallel \-\-embed`. This has the added benefit of also working if access to the internet is down or restricted. .PP https://github.com/rosszurowski/tandem (Last checked: 2023\-01) .SS "\s-1DIFFERENCES BETWEEN\s0 rust\-parallel(aaronriekenberg) \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN rust-parallel(aaronriekenberg) AND GNU Parallel" Summary (see legend above): .IP "I1 I2 I3 \- \- \- \-" 4 .IX Item "I1 I2 I3 - - - -" .PD 0 .IP "\- \- \- \- \- M6" 4 .IX Item "- - - - - M6" .IP "O1 O2 O3 \- O5 O6 \- x \- O10" 4 .IX Item "O1 O2 O3 - O5 O6 - x - O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBrust-parallel\fR has a goal of only using Rust. It seems it is impossible to call bash functions from the command line. You would need to put these in a script. .PP Calling a script that misses the shebang line (#! as first line) fails. .PP \fI\s-1EXAMPLES FROM\s0 rust-parallel's \s-1README\s0.md\fR .IX Subsection "EXAMPLES FROM rust-parallel's README.md" .PP .Vb 7 \& $ cat >./test </dev/null | wc \-l \& 1$ time find ./ \-type f | \& parallel \-j28 \-m \-\- sha256sum 2>/dev/null | wc \-l \& \& 2$ time find ./ \-type f | \& forkrun \-l512 \-k \-\- sha256sum 2>/dev/null | wc \-l \& 2$ time find ./ \-type f | \& parallel \-j28 \-k \-m \-\- sha256sum 2>/dev/null | wc \-l .Ve .PP https://github.com/jkool702/forkrun (Last checked: 2023\-02) .SS "\s-1DIFFERENCES BETWEEN\s0 parallel-sh \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN parallel-sh AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- I4 \- \- \-" 4 .IX Item "I1 I2 - I4 - - -" .PD 0 .IP "M1 \- \- \- \- M6" 4 .IX Item "M1 - - - - M6" .IP "O1 O2 O3 \- O5 O6 \- \- \- O10" 4 .IX Item "O1 O2 O3 - O5 O6 - - - O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBparallel-sh\fR buffers in \s-1RAM.\s0 The buffering data takes O(n^1.5) time: .PP 2MB=0.107s 4MB=0.175s 8MB=0.342s 16MB=0.766s 32MB=2.2s 64MB=6.7s 128MB=20s 256MB=64s 512MB=248s 1024MB=998s 2048MB=3756s .PP It limits the practical usability to jobs outputting < 256 \s-1MB. GNU\s0 \&\fBparallel\fR buffers on disk, yet is faster for jobs with outputs > 16 \&\s-1MB\s0 and is only limited by the free space in \f(CW$TMPDIR\fR. .PP \&\fBparallel-sh\fR can kill running jobs if a job fails (Similar to \&\fB\-\-halt now,fail=1\fR). .PP \fI\s-1EXAMPLES\s0\fR .IX Subsection "EXAMPLES" .PP .Vb 1 \& 1$ parallel\-sh "sleep 2 && echo first" "sleep 1 && echo second" \& \& 1$ parallel ::: "sleep 2 && echo first" "sleep 1 && echo second" \& \& 2$ cat /tmp/commands \& sleep 2 && echo first \& sleep 1 && echo second \& \& 2$ parallel\-sh \-f /tmp/commands \& \& 2$ parallel \-a /tmp/commands \& \& 3$ echo \-e \*(Aqsleep 2 && echo first\ensleep 1 && echo second\*(Aq | \& parallel\-sh \& \& 3$ echo \-e \*(Aqsleep 2 && echo first\ensleep 1 && echo second\*(Aq | \& parallel .Ve .PP https://github.com/thyrc/parallel\-sh (Last checked: 2023\-04) .SS "\s-1DIFFERENCES BETWEEN\s0 bash-parallel \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN bash-parallel AND GNU Parallel" Summary (see legend above): .IP "\- I2 \- \- \- \- I7" 4 .IX Item "- I2 - - - - I7" .PD 0 .IP "M1 \- M3 \- M5 M6" 4 .IX Item "M1 - M3 - M5 M6" .IP "\- O2 O3 \- \- O6 \- O8 \- O10" 4 .IX Item "- O2 O3 - - O6 - O8 - O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBbash-parallel\fR is not as much a command as it is a shell script that you have to alter. It requires you to change the shell function process_job that runs the job, and set \f(CW$MAX_POOL_SIZE\fR to the number of jobs to run in parallel. .PP It is half as fast as \s-1GNU\s0 \fBparallel\fR for short jobs. .PP https://github.com/thilinaba/bash\-parallel (Last checked: 2023\-05) .SS "\s-1DIFFERENCES BETWEEN\s0 PaSH \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN PaSH AND GNU Parallel" Summary (see legend above): N/A .PP \&\fBpash\fR is quite different from \s-1GNU\s0 \fBparallel\fR. It is not a general parallelizer. It takes a shell script and analyses it and parallelizes parts of it by replacing the parts with commands that will give the same result. .PP This will replace \fBsort\fR with a command that does pretty much the same as \fBparsort \-\-parallel=8\fR (except somewhat slower): .PP .Vb 1 \& pa.sh \-\-width 8 \-c \*(Aqcat bigfile | sort\*(Aq .Ve .PP However, even a simple change will confuse \fBpash\fR and you will get no parallelization: .PP .Vb 2 \& pa.sh \-\-width 8 \-c \*(Aqmysort() { sort; }; cat bigfile | mysort\*(Aq \& pa.sh \-\-width 8 \-c \*(Aqcat bigfile | sort | md5sum\*(Aq .Ve .PP From the source it seems \fBpash\fR only looks at: awk cat col comm cut diff grep head mkfifo mv rm sed seq sort tail tee tr uniq wc xargs .PP For pipelines where these commands are bottlenecks, it might be worth testing if \fBpash\fR is faster than \s-1GNU\s0 \fBparallel\fR. .PP \&\fBpash\fR does not respect \f(CW$TMPDIR\fR but always uses /tmp. If \fBpash\fR dies unexpectantly it does not clean up. .PP https://github.com/binpash/pash (Last checked: 2023\-05) .SS "\s-1DIFFERENCES BETWEEN\s0 korovkin-parallel \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN korovkin-parallel AND GNU Parallel" Summary (see legend above): .IP "I1 \- \- \- \- \- \-" 4 .IX Item "I1 - - - - - -" .PD 0 .IP "M1 \- \- \- \- M6" 4 .IX Item "M1 - - - - M6" .IP "\- \- O3 \- \- \- \- x x \-" 4 .IX Item "- - O3 - - - - x x -" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "R1 \- \- \- \- R6 x x \-" 4 .IX Item "R1 - - - - R6 x x -" .IP "\- \-" 4 .PD .PP \&\fBkorovkin-parallel\fR prepends all lines with some info. .PP The output is colored with 6 color combinations, so job 1 and 7 will get the same color. .PP You can get similar output with: .PP .Vb 3 \& (echo ...) | \& parallel \-\-color \-j 10 \-\-lb \-\-tagstring \e \& \*(Aq[l:{#}:{=$_=sprintf("%7.03f",::now()\-$^T)=} {=$_=hh_mm_ss($^T)=} {%}]\*(Aq .Ve .PP Lines longer than 8192 chars are broken into lines shorter than 8192. \fBkorovkin-parallel\fR loses the last char for lines exactly 8193 chars long. .PP Short lines from different jobs do not mix, but long lines do: .PP .Vb 8 \& fun() { \& perl \-e \*(Aq$a="\*(Aq$1\*(Aq"x1000000; for(1..\*(Aq$2\*(Aq) { print $a };\*(Aq; \& echo; \& } \& export \-f fun \& (echo fun a 100;echo fun b 100) | korovkin\-parallel | tr \-s abcdef \& # Compare to: \& (echo fun a 100;echo fun b 100) | parallel | tr \-s abcdef .Ve .PP There should be only one line of a's and one line of b's. .PP Just like \s-1GNU\s0 \fBparallel\fR \fBkorovkin-parallel\fR offers a master/slave model, so workers on other servers can do some of the tasks. But contrary to \s-1GNU\s0 \fBparallel\fR you must manually start workers on these servers. The communication is neither authenticated nor encrypted. .PP It caches output in \s-1RAM:\s0 a 1GB line uses ~2.5GB \s-1RAM\s0 .PP https://github.com/korovkin/parallel (Last checked: 2023\-07) .SS "\s-1DIFFERENCES BETWEEN\s0 xe \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN xe AND GNU Parallel" Summary (see legend above): .IP "I1 I2 \- I4 \- \- I7" 4 .IX Item "I1 I2 - I4 - - I7" .PD 0 .IP "M1 \- M3 M4 \- M6" 4 .IX Item "M1 - M3 M4 - M6" .IP "\- O2 O3 \- O5 O6 \- O8 \- O10" 4 .IX Item "- O2 O3 - O5 O6 - O8 - O10" .IP "E1 \- \- E4 \- \- \-" 4 .IX Item "E1 - - E4 - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBxe\fR has a peculiar limitation: .PP .Vb 2 \& echo /bin/echo | xe {} OK \& echo echo | xe /bin/{} fails .Ve .PP \fI\s-1EXAMPLES\s0\fR .IX Subsection "EXAMPLES" .PP Compress all .c files in the current directory, using all \s-1CPU\s0 cores: .PP .Vb 1 \& 1$ xe \-a \-j0 gzip \-\- *.c \& \& 1$ parallel gzip ::: *.c .Ve .PP Remove all empty files, using \fBlr\fR\|(1): .PP .Vb 1 \& 2$ lr \-U \-t \*(Aqsize == 0\*(Aq | xe \-N0 rm \& \& 2$ lr \-U \-t \*(Aqsize == 0\*(Aq | parallel \-X rm .Ve .PP Convert .mp3 to .ogg, using all \s-1CPU\s0 cores: .PP .Vb 1 \& 3$ xe \-a \-j0 \-s \*(Aqffmpeg \-i "${1}" "${1%.mp3}.ogg"\*(Aq \-\- *.mp3 \& \& 3$ parallel ffmpeg \-i {} {.}.ogg ::: *.mp3 .Ve .PP Same, using percent rules: .PP .Vb 1 \& 4$ xe \-a \-j0 \-p %.mp3 ffmpeg \-i %.mp3 %.ogg \-\- *.mp3 \& \& 4$ parallel \-\-rpl \*(Aq% s/\e.mp3// or skip\*(Aq ffmpeg \-i %.mp3 %.ogg ::: *.mp3 .Ve .PP Similar, but hiding output of ffmpeg, instead showing spawned jobs: .PP .Vb 1 \& 5$ xe \-ap \-j0 \-vvq \*(Aq%.{m4a,ogg,opus}\*(Aq ffmpeg \-y \-i {} out/%.mp3 \-\- * \& \& 5$ parallel \-v \-\-rpl \*(Aq% s/\e.(m4a|ogg|opus)// or skip\*(Aq \e \& ffmpeg \-y \-i {} out/%.mp3 \*(Aq2>/dev/null\*(Aq ::: * \& \& 5$ parallel \-v ffmpeg \-y \-i {} out/{.}.mp3 \*(Aq2>/dev/null\*(Aq ::: * .Ve .PP https://github.com/leahneukirchen/xe (Last checked: 2023\-08) .SS "\s-1DIFFERENCES BETWEEN\s0 sp \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN sp AND GNU Parallel" Summary (see legend above): .IP "\- \- \- I4 \- \- \-" 4 .IX Item "- - - I4 - - -" .PD 0 .IP "M1 \- M3 \- \- M6" 4 .IX Item "M1 - M3 - - M6" .IP "\- O2 O3 \- O5 (O6) \- x x O10" 4 .IX Item "- O2 O3 - O5 (O6) - x x O10" .IP "E1 \- \- \- \- \- \-" 4 .IX Item "E1 - - - - - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBsp\fR has very few options. .PP It can either be used like: .PP .Vb 1 \& sp command {} option :: arg1 arg2 arg3 .Ve .PP which is similar to: .PP .Vb 1 \& parallel command {} option ::: arg1 arg2 arg3 .Ve .PP Or: .PP .Vb 1 \& sp command1 :: "command2 \-option" :: "command3 foo bar" .Ve .PP which is similar to: .PP .Vb 1 \& parallel ::: command1 "command2 \-option" "command3 foo bar" .Ve .PP \&\fBsp\fR deals badly with too many commands: This causes \fBsp\fR to run out of file handles and gives data loss. .PP For each command that fails, \fBsp\fR will print an error message on stderr (standard error). .PP You cannot used exported shell functions as commands. .PP \fI\s-1EXAMPLES\s0\fR .IX Subsection "EXAMPLES" .PP .Vb 1 \& 1$ sp echo {} :: 1 2 3 \& \& 1$ parallel echo {} ::: 1 2 3 \& \& 2$ sp echo {} {} :: 1 2 3 \& \& 2$ parallel echo {} {} :: 1 2 3 \& \& 3$ sp echo 1 :: echo 2 :: echo 3 \& \& 3$ parallel ::: \*(Aqecho 1\*(Aq \*(Aqecho 2\*(Aq \*(Aqecho 3\*(Aq \& \& 4$ sp a foo bar :: "b \*(Aqbaz bar\*(Aq" :: c \& \& 4$ parallel ::: \*(Aqa foo bar\*(Aq "b \*(Aqbaz bar\*(Aq" :: c .Ve .PP https://github.com/SergioBenitez/sp (Last checked: 2023\-10) .SS "\s-1DIFFERENCES BETWEEN\s0 repeater \s-1AND GNU\s0 Parallel" .IX Subsection "DIFFERENCES BETWEEN repeater AND GNU Parallel" Summary (see legend above): .IP "\- \- \- \- \- \- \-" 4 .PD 0 .IP "\- \- \- \- \- \-" 4 .IP "\- O2 O3 N/A \- O6 \- x x ?O10" 4 .IX Item "- O2 O3 N/A - O6 - x x ?O10" .IP "E1 \- \- \- E5 \- \-" 4 .IX Item "E1 - - - E5 - -" .IP "\- \- \- \- \- \- \- \- \-" 4 .IP "\- \-" 4 .PD .PP \&\fBrepeater\fR runs the same job repeatedly. In other words: It does not read arguments, thus is it an alternative for \s-1GNU\s0 \fBparallel\fR for only quite limited applications. .PP \&\fBrepeater\fR has an overhead of around 0.23 ms/job. Compared to \s-1GNU\s0 \&\fBparallel\fR's 2\-3 ms this is fast. Compared to \fBbash-parallel\fR's 0.05 ms/job it is slow. .PP \fIMemory use and run time for large output\fR .IX Subsection "Memory use and run time for large output" .PP Output takes O(n^2) time for output of size n. 10 \s-1MB\s0 takes ~1 second, 30 \s-1MB\s0 takes ~7 seconds, 100 \s-1MB\s0 takes ~60 seconds, 300 \s-1MB\s0 takes ~480 seconds, 1000 \s-1GB\s0 takes .PP 100 \s-1MB\s0 of output takes around 1 \s-1GB\s0 of \s-1RAM.\s0 .PP .Vb 4 \& # Run time = 15 sec \& # Memory use = 20 MB \& # Output = 1 GB per job \& \etime \-v parallel \-j1 seq ::: 120000000 120000000 >/dev/null \& \& # Run time = 4.7 sec \& # Memory use = 95 MB \& # Output = 8 MB per job \& \etime \-v repeater \-w 1 \-n 2 \-reportFile ./run_output seq 1200000 >/dev/null \& \& # Run time = 42 sec \& # Memory use = 277 MB \& # Output = 27 MB per job \& \etime \-v repeater \-w 1 \-n 2 \-reportFile ./run_output seq 3600000 >/dev/null \& \& # Run time = 530 sec \& # Memory use = 1000 MB \& # Output = 97 MB per job \& \etime \-v repeater \-w 1 \-n 2 \-reportFile ./run_output seq 12000000 >/dev/null \& \& # Run time = 2h41m \& # Memory use = 8.6 GB \& # Output = 1 GB per job \& \etime \-v repeater \-w 1 \-n 2 \-reportFile ./run_output seq 120000000 >/dev/null .Ve .PP For even just moderate sized outputs \s-1GNU\s0 \fBparallel\fR will be faster and use less memory. .PP \fI\s-1EXAMPLES\s0\fR .IX Subsection "EXAMPLES" .PP .Vb 2 \& 1$ repeater \-n 100 \-w 10 \-reportFile ./run_output \& \-output REPORT_FILE \-progress BOTH curl example.com \& \& 1$ seq 100 | parallel \-\-joblog run.log \-\-eta curl example.com > output \& \& 2$ repeater \-n 100 \-increment \-progress HIDDEN \-reportFile foo \& echo "this is increment: " INC \& 2$ seq 100 | parallel echo {} \& 2$ seq 100 | parallel echo \*(Aq{= $_ = ++$myvar =}\*(Aq .Ve .PP https://github.com/baalimago/repeater (Last checked: 2023\-12) .SS "Todo" .IX Subsection "Todo" https://github.com/justanhduc/task\-spooler .PP https://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html .PP https://www.npmjs.com/package/concurrently .PP http://code.google.com/p/push/ (cannot compile) .PP https://github.com/krashanoff/parallel .PP https://github.com/Nukesor/pueue .PP https://arxiv.org/pdf/2012.15443.pdf KumQuat .PP https://github.com/JeiKeiLim/simple_distribute_job .PP https://github.com/reggi/pkgrun \- not obvious how to use .PP https://github.com/benoror/better\-npm\-run \- not obvious how to use .PP https://github.com/bahmutov/with\-package .PP https://github.com/flesler/parallel .PP https://github.com/Julian/Verge .PP https://vicerveza.homeunix.net/~viric/soft/ts/ .PP https://github.com/chapmanjacobd/que .SH "TESTING OTHER TOOLS" .IX Header "TESTING OTHER TOOLS" There are certain issues that are very common on parallelizing tools. Here are a few stress tests. Be warned: If the tool is badly coded it may overload your machine. .SS "\s-1MIX:\s0 Output mixes" .IX Subsection "MIX: Output mixes" Output from 2 jobs should not mix. If the output is not used, this does not matter; but if the output \fIis\fR used then it is important that you do not get half a line from one job followed by half a line from another job. .PP If the tool does not buffer, output will most likely mix now and then. .PP This test stresses whether output mixes. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool="parallel \-j 30" \& \& cat <<\-EOF > mycommand \& #!/bin/bash \& \& # If a, b, c, d, e, and f mix: Very bad \& perl \-e \*(Aqprint STDOUT "a"x3000_000," "\*(Aq \& perl \-e \*(Aqprint STDERR "b"x3000_000," "\*(Aq \& perl \-e \*(Aqprint STDOUT "c"x3000_000," "\*(Aq \& perl \-e \*(Aqprint STDERR "d"x3000_000," "\*(Aq \& perl \-e \*(Aqprint STDOUT "e"x3000_000," "\*(Aq \& perl \-e \*(Aqprint STDERR "f"x3000_000," "\*(Aq \& echo \& echo >&2 \& EOF \& chmod +x mycommand \& \& # Run 30 jobs in parallel \& seq 30 | \& $paralleltool ./mycommand > >(tr \-s abcdef) 2> >(tr \-s abcdef >&2) \& \& # \*(Aqa c e\*(Aq and \*(Aqb d f\*(Aq should always stay together \& # and there should only be a single line per job .Ve .SS "\s-1STDERRMERGE:\s0 Stderr is merged with stdout" .IX Subsection "STDERRMERGE: Stderr is merged with stdout" Output from stdout and stderr should not be merged, but kept separated. .PP This test shows whether stdout is mixed with stderr. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool="parallel \-j0" \& \& cat <<\-EOF > mycommand \& #!/bin/bash \& \& echo stdout \& echo stderr >&2 \& echo stdout \& echo stderr >&2 \& EOF \& chmod +x mycommand \& \& # Run one job \& echo | \& $paralleltool ./mycommand > stdout 2> stderr \& cat stdout \& cat stderr .Ve .SS "\s-1RAM:\s0 Output limited by \s-1RAM\s0" .IX Subsection "RAM: Output limited by RAM" Some tools cache output in \s-1RAM.\s0 This makes them extremely slow if the output is bigger than physical memory and crash if the output is bigger than the virtual memory. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool="parallel \-j0" \& \& cat <<\*(AqEOF\*(Aq > mycommand \& #!/bin/bash \& \& # Generate 1 GB output \& yes "\`perl \-e \*(Aqprint \e"c\e"x30_000\*(Aq\`" | head \-c 1G \& EOF \& chmod +x mycommand \& \& # Run 20 jobs in parallel \& # Adjust 20 to be > physical RAM and < free space on /tmp \& seq 20 | time $paralleltool ./mycommand | wc \-c .Ve .SS "\s-1DISKFULL:\s0 Incomplete data if /tmp runs full" .IX Subsection "DISKFULL: Incomplete data if /tmp runs full" If caching is done on disk, the disk can run full during the run. Not all programs discover this. \s-1GNU\s0 Parallel discovers it, if it stays full for at least 2 seconds. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool="parallel \-j0" \& \& # This should be a dir with less than 100 GB free space \& smalldisk=/tmp/shm/parallel \& \& TMPDIR="$smalldisk" \& export TMPDIR \& \& max_output() { \& # Force worst case scenario: \& # Make GNU Parallel only check once per second \& sleep 10 \& # Generate 100 GB to fill $TMPDIR \& # Adjust if /tmp is bigger than 100 GB \& yes | head \-c 100G >$TMPDIR/$$ \& # Generate 10 MB output that will not be buffered \& # due to full disk \& perl \-e \*(Aqprint "X"x10_000_000\*(Aq | head \-c 10M \& echo This part is missing from incomplete output \& sleep 2 \& rm $TMPDIR/$$ \& echo Final output \& } \& \& export \-f max_output \& seq 10 | $paralleltool max_output | tr \-s X .Ve .SS "\s-1CLEANUP:\s0 Leaving tmp files at unexpected death" .IX Subsection "CLEANUP: Leaving tmp files at unexpected death" Some tools do not clean up tmp files if they are killed. If the tool buffers on disk, they may not clean up, if they are killed. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool=parallel \& \& ls /tmp >/tmp/before \& seq 10 | $paralleltool sleep & \& pid=$! \& # Give the tool time to start up \& sleep 1 \& # Kill it without giving it a chance to cleanup \& kill \-9 $! \& # Should be empty: No files should be left behind \& diff <(ls /tmp) /tmp/before .Ve .SS "\s-1SPCCHAR:\s0 Dealing badly with special file names." .IX Subsection "SPCCHAR: Dealing badly with special file names." It is not uncommon for users to create files like: .PP .Vb 1 \& My brother\*(Aqs 12" *** record (costs $$$).jpg .Ve .PP Some tools break on this. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool=parallel \& \& touch "My brother\*(Aqs 12\e" *** record (costs \e$\e$\e$).jpg" \& ls My*jpg | $paralleltool ls \-l .Ve .SS "\s-1COMPOSED:\s0 Composed commands do not work" .IX Subsection "COMPOSED: Composed commands do not work" Some tools require you to wrap composed commands into \fBbash \-c\fR. .PP .Vb 1 \& echo bar | $paralleltool echo foo\*(Aq;\*(Aq echo {} .Ve .SS "\s-1ONEREP:\s0 Only one replacement string allowed" .IX Subsection "ONEREP: Only one replacement string allowed" Some tools can only insert the argument once. .PP .Vb 1 \& echo bar | $paralleltool echo {} foo {} .Ve .SS "\s-1INPUTSIZE:\s0 Length of input should not be limited" .IX Subsection "INPUTSIZE: Length of input should not be limited" Some tools limit the length of the input lines artificially with no good reason. \s-1GNU\s0 \fBparallel\fR does not: .PP .Vb 1 \& perl \-e \*(Aqprint "foo."."x"x100_000_000\*(Aq | parallel echo {.} .Ve .PP \&\s-1GNU\s0 \fBparallel\fR limits the command to run to 128 \s-1KB\s0 due to \fBexecve\fR\|(1): .PP .Vb 1 \& perl \-e \*(Aqprint "x"x131_000\*(Aq | parallel echo {} | wc .Ve .SS "\s-1NUMWORDS:\s0 Speed depends on number of words" .IX Subsection "NUMWORDS: Speed depends on number of words" Some tools become very slow if output lines have many words. .PP .Vb 1 \& #!/bin/bash \& \& paralleltool=parallel \& \& cat <<\-EOF > mycommand \& #!/bin/bash \& \& # 10 MB of lines with 1000 words \& yes "\`seq 1000\`" | head \-c 10M \& EOF \& chmod +x mycommand \& \& # Run 30 jobs in parallel \& seq 30 | time $paralleltool \-j0 ./mycommand > /dev/null .Ve .SS "4GB: Output with a line > 4GB should be \s-1OK\s0" .IX Subsection "4GB: Output with a line > 4GB should be OK" .Vb 1 \& #!/bin/bash \& \& paralleltool="parallel \-j0" \& \& cat <<\-EOF > mycommand \& #!/bin/bash \& \& perl \-e \*(Aq\e$a="a"x1000_000; for(1..5000) { print \e$a }\*(Aq \& EOF \& chmod +x mycommand \& \& # Run 1 job \& seq 1 | $paralleltool ./mycommand | LC_ALL=C wc .Ve .SH "AUTHOR" .IX Header "AUTHOR" When using \s-1GNU\s0 \fBparallel\fR for a publication please cite: .PP O. Tange (2011): \s-1GNU\s0 Parallel \- The Command-Line Power Tool, ;login: The \s-1USENIX\s0 Magazine, February 2011:42\-47. .PP This helps funding further development; and it won't cost you a cent. If you pay 10000 \s-1EUR\s0 you should feel free to use \s-1GNU\s0 Parallel without citing. .PP Copyright (C) 2007\-10\-18 Ole Tange, http://ole.tange.dk .PP Copyright (C) 2008\-2010 Ole Tange, http://ole.tange.dk .PP Copyright (C) 2010\-2024 Ole Tange, http://ole.tange.dk and Free Software Foundation, Inc. .PP Parts of the manual concerning \fBxargs\fR compatibility is inspired by the manual of \fBxargs\fR from \s-1GNU\s0 findutils 4.4.2. .SH "LICENSE" .IX Header "LICENSE" This program is free software; you can redistribute it and/or modify it under the terms of the \s-1GNU\s0 General Public License as published by the Free Software Foundation; either version 3 of the License, or at your option any later version. .PP This program is distributed in the hope that it will be useful, but \s-1WITHOUT ANY WARRANTY\s0; without even the implied warranty of \&\s-1MERCHANTABILITY\s0 or \s-1FITNESS FOR A PARTICULAR PURPOSE.\s0 See the \&\s-1GNU\s0 General Public License for more details. .PP You should have received a copy of the \s-1GNU\s0 General Public License along with this program. If not, see . .SS "Documentation license I" .IX Subsection "Documentation license I" Permission is granted to copy, distribute and/or modify this documentation under the terms of the \s-1GNU\s0 Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the file \s-1LICENSES/GFDL\-1\s0.3\-or\-later.txt. .SS "Documentation license \s-1II\s0" .IX Subsection "Documentation license II" You are free: .IP "\fBto Share\fR" 9 .IX Item "to Share" to copy, distribute and transmit the work .IP "\fBto Remix\fR" 9 .IX Item "to Remix" to adapt the work .PP Under the following conditions: .IP "\fBAttribution\fR" 9 .IX Item "Attribution" You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). .IP "\fBShare Alike\fR" 9 .IX Item "Share Alike" If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. .PP With the understanding that: .IP "\fBWaiver\fR" 9 .IX Item "Waiver" Any of the above conditions can be waived if you get permission from the copyright holder. .IP "\fBPublic Domain\fR" 9 .IX Item "Public Domain" Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license. .IP "\fBOther Rights\fR" 9 .IX Item "Other Rights" In no way are any of the following rights affected by the license: .RS 9 .IP "\(bu" 2 Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations; .IP "\(bu" 2 The author's moral rights; .IP "\(bu" 2 Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. .RE .RS 9 .RE .IP "\fBNotice\fR" 9 .IX Item "Notice" For any reuse or distribution, you must make clear to others the license terms of this work. .PP A copy of the full license is included in the file as \&\s-1LICENCES/CC\-BY\-SA\-4.0\s0.txt .SH "DEPENDENCIES" .IX Header "DEPENDENCIES" \&\s-1GNU\s0 \fBparallel\fR uses Perl, and the Perl modules Getopt::Long, IPC::Open3, Symbol, IO::File, \s-1POSIX,\s0 and File::Temp. For remote usage it also uses rsync with ssh. .SH "SEE ALSO" .IX Header "SEE ALSO" \&\fBfind\fR(1), \fBxargs\fR(1), \fBmake\fR(1), \fBpexec\fR(1), \fBppss\fR(1), \&\fBxjobs\fR(1), \fBprll\fR(1), \fBdxargs\fR(1), \fBmdm\fR(1)