Upstream changes:
20160822
- Bug fixes and man page updates.
20160722
- env_parallel is now ready for wider testing. It is still beta
quality.
- env_parallel is heavily modified for all shells and testing has been
increased.
- Selectively choosing what to export using --env now works for
env_parallel (bash, csh, fish, ksh, pdksh, tcsh, zsh).
- --round-robin now gives more work to a job that processes faster
instead of same amount to all jobs.
- --pipepart works on block devices on GNU/Linux.
- Bug fixes and man page updates.
20160622
- $PATH can now be exported using --env PATH. Useful if GNU Parallel
is not in your path on remote machines.
- If --block is left out, --pipepart will use a block size that will
result in 10 jobs per jobslot.
- Bug fixes and man page updates.
20160522
- niceload --net pauses the program if the internet connection is
overloaded.
- Bug fixes and man page updates.
20160422
- :::+ and ::::+ work like ::: and :::: but links this input source to
the previous input source in a --xapply fashion. Contrary to
--xapply values do not wrap: The shortest input source determines
the length.
- --line-buffer --keep-order now outputs continously from the oldest
job still running. This is more what you would expect than the
earlier behaviour where --keep-order had no effect with
--line-buffer.
- env_parallel supports tcsh, csh, pdksh. In fish it now supports
arrays. In csh/tcsh it now supports variables, aliases, and arrays
with no special chars. In pdksh it supports aliases, functions,
variables, and arrays.
- Function exporting on Mac OS X works around old Bash version.
- Better CPU detection on OpenIndiana.
- Bug fixes and man page updates.
20160322
- env_parallel is a function that exports the environment (functions,
aliases, variables, and arrays) to GNU Parallel. Run 'man
env_parallel' for details.
- niceload --prg now searches for substrings if no process with the
name is found.
- Bug fixes and man page updates.
20160222
- Bug fixes and man page updates.
20160122
- --sql DBURL uses DBURL as storage for jobs and output. It does not
run any jobs so it requires at least one --sqlworker. DBURL must
point to a table.
- --sqlworker DBURL gets jobs from DBURL and stores the result back to
DBURL.
- --sqlandworker is a shorthand for --sql and --sqlworker.
- --sqlworker requires the output of a single job to fit in memory.
- --results now also saves a file called 'seq' containing the sequence
number.
- If $PARALLEL_ENV is a file, then that file will be read into
$PARALLEL_ENV.
- man parallel_tutorial has been given an overhaul.
- --workdir now accepts replacementstrings.
- Bug fixes and man page updates.
20151222
- --transfer is now an alias for --transferfile {}.
- --transferfile works like --transfer, but takes an argument like
--return. This makes it possible to combine transferring files
with multiple input sources: parallel -S server --tf {1} wc {2}
{1} ::: * ::: -l -w -c
- total_jobs() can now be used in {= =}: parallel echo job {#} of '{=
$_=total_jobs() =}' ::: {1..50}
- Bug fixes and man page updates.
20151122
- Bug fixes and man page updates.
20151022
- --plus makes it possible to use {##} as a shorthand for
{= $_=$Global::JobQueue->total_jobs() =} which gives the the number
of jobs to run in total.
- {= $_=$Global::JobQueue->total_jobs() =} is incompatible with -X,
-m, and --xargs.
- GNU Parallel is now mostly compatible with lsh
(http://www.lysator.liu.se/~nisse/lsh/) and somewhat compatible with
autossh (http://www.harding.motd.ca/autossh/).
- --workdir ... now also works when run locally.
- Bug fixes and man page updates.
20150922
- Bug fixes and man page updates.
20150822
- If $job->skip() is called in {= =} the job will not be run.
- @arg can now be used in {= =}: parallel echo '{= $arg[2] < 5 and
$_="j" =}' ::: 1 2 3 ::: 4 5 6
- Bug fixes and man page updates.
20150722
- If a record fits in a block, --block only reads upto the size of
--block. If not: --block reads a full --block more.
- Due to widely spread distributions still using --tollef even after a
year of being retired following a year of being obsolete, --tollef
now gives an error unless used with --gnu.
- --nice now uses a perl wrapper instead of the nice command.
- Bug fixes and man page updates.
20150622
- --halt has been rewritten completely. You can now combine
percentages with success or fail. See the man page.
- Exit values 102..254 have been removed. 101 means more than 100 jobs
failed.
- Killing through --timeout, --memfree, or --halt is now done as a
process group.
- --termseq determines which signals are sent when a job is killed.
- An empty argument would previously cause no string to be
inserted. This is now changed to '' being inserted, thus prepending
a space to the output of: parallel echo {} b ::: ''
- $PARALLEL_ENV can now be set to an environment prepending the
command. Used in env_parallel as mentioned in the manpage.
- --retry-failed will retry all failed jobs in a joblog. It will
ignore any command given.
- --ssh and $PARALLEL_SSH can be used to set the command used for
ssh. The command is assumed to behave the same way as ssh.
- --fifo now works in csh, too.
- Q(...) can be used in {= =} to shell quote a string.
- Bug fixes and man page updates.
20150522
- Security: The security issue for --sshlogin + --fifo/--cat has been
fixed. Thereby all issues with
http://lists.gnu.org/archive/html/parallel/2015-04/msg00045.html
have been fixed.
- Security: After further security analysis the issue fixed in
20150422 also fixed the problem for --tmux.
- Bug fixes and man page updates.
20150422
New in this release:
- Security fix. An attacker on the local system could make you
overwrite one of your own files with a single byte. The problem
exist when you use --compress or --tmux or --pipe or --cat or
--fifo. The attacker must figure out the randomly chosen file name
and create a symlink within a time window of 15 ms.
- --shuf will generate all jobs, and shuffle them before running
them. This is useful to get a quick preview of the results before
running the full batch.
- Bug fixes and man page updates.
20150322
New in this release:
- --number-of-cores respects 'taskset' on GNU/Linux.
- --joblog --pipe gives the data send and received in the log.
- Bug fixes and man page updates.
20150222
- --tmux has gotten a major overhaul.
- Bug fixes and man page updates.
20150122
- Remote jobs now send stderr (standard error) to stderr (standard
error) instead of stdout (standard output).
- Remote execution command is now packed using base64 encoded
bzip2. This means that big environments (app. 100 KB) can be
transferred. This change makes remote execution alpha quality.
- --semaphoretimeout now takes a negative value. This means give up if
the semaphore is not gotten within the timeout.
- --halt -1 and -2 now means halt if a job succeeds (so the opposite
of 1 and 2).
- --no-keep-order will reverse --keep-order.
- Bash's second fix of shellshock caused --env to break again when
exporting functions. This has been fixed again.
- A semibig refactoring of big functions. All non-trivial functions
are now less than 100 lines. The refactoring makes this release beta
quality.
- A description of the design decisions for GNU Parallel can be found
in 'man parallel_design'.
- A bug fix in replacement strings caused rewrite of the replacement
function. This makes use of replacement strings alpha quality.
- Bug fixes and man page updates.
20141122
- Remote systems can be divided into hostgroups (e.g. web and db) by
prepending '@groupname/' to the sshlogin. Multiple groups can be
given by separating groups with '+'. E.g. @web/www1 @web+db/www2
@db/mariadb
- Remote execution can be restricted to servers that are part of one
or more groups by '@groupname' as an sshlogin. Multiple groups can
be given by separating groups with '+'. E.g. -S @web or -S @db+web
- With --hostgroup you can restrict arguments to certain hostgroups by
appending '@groupname' to the argument. Multiple groups can be
given by separating groups with '+'. E.g. my_web_arg@web
db-or-web-arg@db+web db-only-arg@db Thanks to Michel Courtine for
developing a prototype for this.
- Bug fixes and man page updates.
20141022
- --env was changed to deal with Bash's new way of exporting a function.
- Bug fixes and man page updates.
20140922
- If the file give as --sshloginfile is changed it will be re-read
when a job finishes though at most once per second. This makes it
possible to add and remove hosts while running.
20140822
- GNU Parallel now uses the same shell it was started from as the
command shell for local jobs. So if GNU Parallel is started from
tcsh it will use tcsh as its shell even if the login $SHELL is
different. For remote jobs the login $SHELL will be used.
- The whole current environment in bash can be copied by using a shell
wrapper function (Search manual for env_parallel).
- --plus adds the replacement strings {+/} {+.} {+..} {+...} {..}
{...} {/..} {/...}. The idea being that '+foo' matches the opposite
of 'foo' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..}
= {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
- GNU Parallel now deals correctly with the combination
rsync-3.1.X-client and rsync-2.5.7-server
- Bug fixes and man page updates.
20140722
- {= perl expression =} can be used as replacement string. The
expression should modify $_. E.g. {= s/\.gz$// =} to remove .gz from
the string. This makes replacement strings extremely flexible.
- Positional perl expressions (similar to {2}) are given as {=2 perl
expression=} where 2 is the position.
- One small backwards incompatibility: {1}_{2} will replace {2} with
the empty string if there is only one argument. Previously {2} would
have been left untouched.
- Replacement strings can be defined using --rpl. E.g. parallel --rpl
'{.gz} s/\.gz$//' echo {.gz} ::: *.gz
- The parenthesis around {= perl expression =} can be changed with
--parens.
- --tmux will direct the output to a tmux session instead of
files. Each running jobs will be in its own window.
- --halt 10% will stop spawning new jobs if 10% failed so far.
- Bug fixes and man page updates.
20140622
- --shebang and --shebang-wrap now works on FreeBSD.
- --pipepart works with --header, --regexp, --cat and --fifo.
- ./configure --disable-documentation will not build documentation.
- {%} works as job slot.
- Bug fixes and man page updates.
20140522
- Use --semaphoretimeout to ignore the semaphore lock after a while.
- {%} introduced as job slot replacement string. It has known bugs.
- --slotreplace changes slot replacement string.
- Bug fixes and man page updates.
20140422
- --pipepart is a highly efficient alternative to --pipe if the input
is a real file and not a pipe.
- If using --cat or --fifo with --pipe the {} in the command will be
replaced with the name of a physical file and a fifo respectively
containing the block from --pipe. Useful for commands that cannot
read from standard input (stdin).
- --controlmaster has gotten an overhaul and is no longer
experimental.
- --env is now copied when determining CPUs on remote system. Useful
for copying $PATH if parallel is not in the normal path.
- --results now chops the argument if the argument is longer than the
allowed path length.
- Build now survives if pod2* are not installed.
- Bug fixes and man page updates.
20140322
- Bug fixes and man page updates.
20140222
- --tollef has been retired.
- --compress has be redesigned due to bugs.
- Format of --eta and --joblog has changed slightly.
- Bug fixes and man page updates.
20140122
- Bug fixes and man page updates.
20131222
- GNU Parallel now has a motto: For people who live life in the
parallel lane.
- Detect if the buffer dir $TMPDIR runs out of space. This changes a
central piece of code, making this release alpha quality.
- --keep-order --(n)onall will sort according to the sshlogin.
- Detect the number of CPUs on DEC Tru64.
- GNU sql: --list-tables for sqlite3.
- Bug fixes and man page updates.
20131122
- A citation notice is printed on stderr only if stderr is a terminal,
the user has not specified --no-notice and the user has not run
--bibtex once. This makes the release alpha quality.
- --compress will compress temporary files. If the output is big and
very compressible this will take up less disk space in $TMPDIR and
possibly be faster due to less disk I/O.
- --compress-program comtrols which program to use for compressing
temporary files.
- --bar show progress as a progress bar compatible with zenity.
- --resume can now be used with --result: Jobs already run will be
skipped.
- --transfer and --basefile support paths relative to the --workdir by
inserting /./ into the path.
- Bug fixes and man page updates.
20131022
- --transfer files with /./ in the path will copy the files relative
to the --workdir.
- The maximal command length is now cached in a file halfing the
startup time.
- Bug fixes and man page updates.
20130922
- PDF-files of documentation is now included.
- Bug fixes and man page updates.
20130822
- A tutorial showing much of GNU Parallel's functionality. Spend an
hour walking through the tutorial. Your commandline will love you
for it. man parallel_tutorial
- --line-buffer will buffer output on line basis. --group keeps the
output together for a whole job. --ungroup allows output to mixup
with half a line coming from one job and half a line coming from
another job. --line-buffer fits between these two; it prints a full
line, but will allow for mixing lines of different jobs.
- --record-env records all variables and functions set. Useful to
record the normal environment before using --env _.
- --env _ exports all functions and variables to the remote server
except for the ones recorded.
- New signing key. Due to recommendations from NIST
http://www.keylength.com/en/4/ the signing key was changed from
1024D/ID:FFFFFFF1 to 9888R/ID:88888888.
- Bug fixes and man page updates.
20130722
- --round-robin with --pipe will write all blocks to already running
jobs.
- --env can now transfer Bash function for remote execution. That is
pretty cool!
- Quite a few bug fixes and man page updates.
20130622
- --xapply now recycles arguments if an input source has more
arguments than others.
- The sleep time between jobs is now both increased and decreased
exponentially.
- 10 seconds installation check the signature using GnuPG if GnuPG is
installed.
- Bug fixes and man page updates.
20130522
- --ctrlc is now default if --pipe is not used; thus making it
possible to kill remotely started jobs simply by pressing Ctrl-C.
- --timeout 200% now means kill jobs that take more than twice the
time of the median run time of a job.
- Jobs are now distributed round robin when having mulitiple
--sshlogin instead of filling up one --sshlogin at a time.
- niceload: darwin version of --io --mem --swap. Thanks to Anders F
Bjorklund.
- Bug fixes and man page updates.
20130422
- 10 seconds installation: wget -O - pi.dk/3 | sh
- HPUX CPU/core detection. Thanks to Javier Tarradas.
- CPU detection for HURD, IRIX, SCO OpenServer and (old) HPUX.
- --ctrlc will send SIGINT to tasks running on remote computers and
thus killing them.
- --load now uses 'ps' to see immediately see the number of running
processes instead of 'uptime' thus making --load react much faster.
- Testing on Centos 3.9 and FreeBSD 7 revealed a lot of compability
bugs. Some of these required quite extensive changes making this
release beta quality.
- --retries works with --onall.
- The new --load computation now works on FreeBSD 7.
- --nice works under tcsh.
- Bug fixes and man page updates.
20130222
- --resume works with --pipe.
- --resume-failed will go through --joblog, redo the failed jobs and
then continue like --resume.
- Negative positional arguments count from the end: {-1} means the
last argument, {-2} the second to last.
- NetBSD CPU detection.
- --blocksize increases exponentially if it smaller than a full
record.
- Processing n-line records (--pipe -L n) is now much faster.
- --tollef is obsoleted after discussion on the mailing list.
- Bug fixes and man page updates.
f0f6c78401