Passes bup's test suite on netbsd-8/amd64, with TZ=UTC (needed to work
around what is likely a bug in git around DST changes).
Packaging changes/notes:
- bup depends directly on readline, GNU version required
- man pages have not yet been updated
Upstream changes include many minor improvements and bugfixes,
and:
Notable changes in 0.31 (since 0.30.1)
======================================
* Python 3 is now supported, and Python 2 support is deprecated. It's
possible that we'll stop new development for Python 2 fairly soon.
If so, we'll probably continue to fix bugs in the last Python 2
compatible version for a while, but please make plans to migrate.
* `bup features` has been added. It reports information about bup
itself, including the Python version, and the current availability
of features like readline or support for POSIX ACLs.
Build and install
-----------------
* `bup` itself is now located in now located in the cmd/ directory in
the install tree and finds sub-commands, etc. relative to its own
location.
packaging changes: switch to 0.30 man tarball
Notable changes in 0.30 as compared to 0.29.3
=============================================
May require attention
---------------------
* The minimum `git` version required is now 1.5.6.
* The `prune-older` command now keeps the most recent save in each
period group (day, week, month, ...) rather than the oldest.
* `bup` now adds a zero-padded suffix to the names of saves with the
same timestamp (e.g. 1970-01-01-214640-07) in order to avoid
duplicates. The sequence number currently represents the save's
reversed position in default `git rev-list` order, so that given:
/foo/1970-01-01-214640-09
/foo/1970-01-01-214640-10
In the normal case, the -10 save would be the next save made after
-09 (and the -09 save would be the single parent commit for -10).
* `bup` is not currently compatible with Python 3 and will now refuse
to run if the Python version is not 2 unless
`BUP_ALLOW_UNEXPECTED_PYTHON_VERSION=true` is set in the environment
(which can be useful for development and testing).
* `bup ls -s` now reports the tree hash for commits unless
`--commit-hash` is also specified.
General
-------
* `bup get` has been added. This command allows the transfer or
rewriting of data within and between repositories, local or remote.
Among other things, it can be used to append remote saves to a local
branch, which by extension supports merging repositories. See
`bup-get(1)` for further information, and please note, this is a new
*EXPERIMENTAL* command that can (intentionally) modify your data in
destructive ways. It is potentially much more dangerous than most
`bup` commands. Treat with caution.
* `bup` can now restore directly from a remote repository via `bup
restore -r host:path ...`. See `bup-restore(1)` for more
information.
* `bup ls` can now report information for remote repositories via `bup
ls -r host:path ...`. See `bup-ls(1)` for more information.
* `bup` should respect the git pack.packSizeLimit setting when writing
packfiles, though at the moment it will only affect a remote
repository when the option is set there directly.
* `bup save` now stores the size for all links and normal files. For
directories saved using this new format retrieving file sizes for
larger files should be notably less expensive. Among other things
this may improve the performance of commands like `bup ls -l` or
`find /some/fuse/dir -ls`.
* The VFS (Virtual File System) that underlies many operations, and
provides the basis for commands like `restore`, `ls`, etc. has been
rewritten in a way that makes remote repository access easier,
should decrease the memory footprint in some cases (e.g. for bup
fuse), and should make it easier to provide more selective caching.
At the moment, data is just evicted at random once a threshold is
reached.
* A `--noop <--blobs|--tree>` option has been added to `bup split`
which prints the resulting id without storing the data in the
repository.
Bugs
----
* The way `bup` handles output from subprocesses (diagnostics,
progress, etc.) has been adjusted in a way that should make it less
likely that bup might continue running after the main process has
exited, say via a C-c (SIGINT).
* `bup` should now respect the specified compression level when
writing to a remote repository.
* `bup restore` now creates FIFOs with mkfifo, not mknod, which is
more portable. The previous approach did not work correctly on (at
least) some versions of NetBSD.
* `bup` should no longer just crash when it encounters a commit with a
"mergetag" header. For the moment, it just ignores them, and
they'll be discarded whenever `bup` rewrites a commit, say via the
`rm`, `prune-older`, or `get` commands.
* The bloom command should now end progress messages with \r, not \n,
which avoids leaving spurious output lines behind at exit.
* A missing space has been added to the `bup split --bench` output.
* Various Python version compatibility problems have been fixed,
including some of the incompatibilities introduced by Python 3.
* Some issues with mincore on WSL have been fixed.
* Some Android build incompatibilities have been fixed.
Build system
------------
* The tests no longer assume pwd is in /bin.
* The tests should be less sensitive to the locale.
* `test-meta` should no longer try to apply chattr +T to files. 'T'
only works for directories, and newer Linux kernels actually reject
the attempt (as of at least 4.12, and maybe 4.10).
* `test-rm` should no longer fail when newer versions of git
automatically create packed-refs.
* `test-sparse-files` should be less likely to fail when run inside a
container.
* `test-index-check-device` and `test-xdev` now use separate files for
their loopback mounts. Previously each was mounting the same image
twice, which could produce the same device number.
Thanks to (at least)
====================
Alexander Barton, Artem Leshchev, Ben Kelly, Fabian 'xx4h' Melters,
Greg Troxel, Jamie Wyrick, Julien Goodwin, Mateusz Konieczny,
Nathaniel Filardo, Patrick Rouleau, Paul Kronenwetter, Rob Browning,
Robert Evans, Tim Riemenschneider, and bedhanger
Upstream changes (other than bugfixes minor improvements):
0.29.3:
* Newer versions of par2 can process in parallel using multiple
threads/cores, and they do so automatically. This ends up competing
with `bup fsck`s own parallelism, enabled by `-j`, in such a way
that the command can actually run much *slower* (and be much more
expensive) than it would have been with no parallelism at all.
When appropriate `bup fsck` now disables par2's competing
parallelism (via its `-t1` option) to avoid the contention.
0.29.0:
* An experimental `prune-older` command has been added. It removes
(permanently deletes) all saves except those preserved by a set of
arguments like `--keep-monthlies-for 3y`. See `bup help
prune-older` for further information.
Upstream changes:
* The index format has changed, which will trigger a full index
rebuild on the next index run, making that run more expensive than
usual.
* When given `--xdev`, `bup save` should no longer skip directories
that are explicitly listed on the command line when the directory is
both on a separate filesystem, and a subtree of another path listed
on the command line. Previously `bup save --xdev / /usr` could skip
"/usr" if it was on a separate filesystem from "/".
* Tags along a branch are no longer shown in the branch's directory in
the virtual filesystem (VFS). i.e. given `bup tag special
/foo/latest`, "/foo/special" will no longer be visible via `bup ls`,
`bup web`, `bup fuse`, etc., but the tag will still be available as
"/.tag/special".
* bup now provides experimental `rm` and `gc` subcommands, which
should allow branches and saves to be deleted, and their storage
space reclaimed (assuming nothing else refers to the relevant data).
For the moment, these commands require an `--unsafe` argument and
should be treated accordingly. Although if an attempt to `join` or
`restore` the data you still care about after a `gc` succeeds,
that's a fairly encouraging sign that the commands worked correctly.
(The `t/compare-trees` command in the source tree can be used to
help test before/after results.)
Note that the current `gc` command is probabilistic, which means it
may not remove *all* of the obsolete data from the repository, but
also means that the command should be fairly efficient, even for
large repositories.
* bup may have less impact on the filesystem cache. It now attempts
to leave the cache roughly the way it found it when running a `save`
or `split`.
* A specific Python can be specified at `./configure` time via PYTHON,
i.e. `PYTHON=/some/python ./configure`, and that Python will be
embedded in all of the relevant scripts as an explicit "#!/..." line
during `make install`.
* The way bup writes the data to disk (the packfiles in particular),
should be a bit safer now if there is a coincident power failure or
system crash.
* bup should be more likely to get the data to permanent storage
safely on OS X, which appears to follow a surprising interpretation
of the `fsync()` specification.
* The build system now creates and uses cmd/bup-python which refers to
the `./configure` selected python.
This version primarily fixes bugs, but it includes a few new features,
and hopefully will provide a solid base for 0.28, which may finally
include rm/gc support.
Of note since 0.26:
- Files can be restored sparsely via "restore --sparse". See
bup-sparse(1) for the details. Given that this option is
completely new and affects the data written, some
caution/verification may be wise (i.e. there were bugs in some of
the development revisions that could mangle the output).
- Duplicity backups can be imported via the *experimental* command
"bup import-duplicity". Please exercise caution, and let us know
if you have any trouble.
- Tests can be run in parallel now via "make -j check".
As before, the metadata support is somewhat immature, but it seems to
be holding up fairly well. Though we still need to add better support
for cross-filesystem-type save/restore (which can be too noisy), etc.
This release doesn't include anything as significant as the metadata
support added in 0.25, but it has quite a few bug fixes and internal
improvements, in addition to these notable changes:
- When --meta is specified to the fuse command, instead of generic
data, the originally saved mode, uid, git, atime, mtime, and ctime
will be reported for the archive paths.
- When --browser is specified to the web command, a browser window
will be opened for the repository.
- The -x/--xdev/--one-filesystem options now include the mountpoint
itself in the traversal (matching rsyc, tar, etc.).
- Empty lines in --exclude-rx-from files will be ignored.
Previously they would cause all paths to be excluded.
- The index and restore commands now support --exclude-rx-from.
- Relative filesystem --excludes like "--exclude bar" should now
work. Previously --excludes had to be absolute.
- The drecurse command now supports --exclude-rx and
--exclude-rx-from.
- The --compress option should now work for remote repositories.
- Streams saved via "bup split" will now show up as a single file
named "data" at the top level of the VFS, instead of as a subtree
(i.e. when examined via ftp, ls, and fuse).
- The ls command now supports -n, -A, -F, --file-type,
--numeric-ids, and detailed -l options.
- The save dates are now taken from the corresponding git commit's
author date, not the committer date.
- The tornado server, required by the web command, is no longer
included. See the README for installation instructions.
Note that the metadata support is still somewhat immature. For
example, we still need to add better support for cross-filesystem-type
save/restore (which is too noisy), etc.
Please give this release a try and let us know what's broken. If
you're new to bup, start with the README (and then HACKING if you'd
like to help further):
https://github.com/bup/bup/blob/master/README.mdhttps://github.com/bup/bup/blob/master/HACKINGhttp://anonscm.debian.org/gitweb/?p=users/rlb/bup.git;a=blob;f=README.md;hb=refs/heads/masterhttp://anonscm.debian.org/gitweb/?p=users/rlb/bup.git;a=blob;f=HACKING;hb=refs/heads/master
And although I probably sound like a broken record -- while we expect
bup to work fairly well, I still don't recommend it as your sole
backup strategy. I'd still suggest a periodic
tar/rsync/etc. backstop.
This release includes a veritable ton of work, but the biggest change
is probably the addition of real (YMMV) metadata support, including
hardlinks, and the ability to directly save/restore trees without tar
and split/join. See bup-index(1), bup-save(1), bup-restore(1),
bup-meta(1), bup-xstat(1), and the updated "Using bup" section in the
README.
Note though, that the metadata support really is our first pass, and
there are already things that we know need fixing (i.e. better support
for cross-filesystem-type save/restore (too noisy), etc.). And check
the "Notes on ..." sections in the README for some platform-specific
limitations.
An incomplete list of other notable changes since 0.24b:
- bup will no longer create ~/.bup implicitly; "bup init" is
required.
- "bup split" now supports a compression-level option (-#).
- "bup tag" now supports "-f".
- "bup ls" now supports "-a", "-s", "--human-readable", etc., and
reports more information.
- "bup web" now supports "--human-readable".
- "bup import-rdiff-backup" has been added.
- "bup cat-file" has been added.
- The default "/usr" installation prefix can be overridden via PREFIX.
- Python 2.4 is no longer supported.
And we have at least one (likely minor) known issue:
- *Very* large (i.e. probably greater than MAX_LONG), or negative
filesystem timestamps may not always be handled correctly at the
moment. We'll fix that soon. And related -- some of the tests
may fail on FUSE filesystems.
bup is a program that backs things up. bup has a few advantages
over other backup software:
It uses a rolling checksum algorithm (similar to rsync) to split
large files into chunks. The most useful result of this is you can
backup huge virtual machine (VM) disk images, databases, and XML
files incrementally, even though they're typically all in one huge
file, and not use tons of disk space for multiple versions.
It uses the packfile format from git (the open source version
control system), so you can access the stored data even if you
don't like bup's user interface.
Unlike git, it writes packfiles directly (instead of having a
separate garbage collection / repacking stage) so it's fast even
with gratuitously huge amounts of data. bup's improved index formats
also allow you to track far more filenames than git (millions) and
keep track of far more objects (hundreds or thousands of gigabytes).
Data is "automagically" shared between incremental backups without
having to know which backup is based on which other one - even if
the backups are made from two different computers that don't even
know about each other. You just tell bup to back stuff up, and it
saves only the minimum amount of data needed.
You can back up directly to a remote bup server, without needing
tons of temporary disk space on the computer being backed up. And
if your backup is interrupted halfway through, the next run will
pick up where you left off. And it's easy to set up a bup server:
just install bup on any machine where you have ssh access.
Bup can use "par2" redundancy to recover corrupted backups even if
your disk has undetected bad sectors.
Even when a backup is incremental, you don't have to worry about
restoring the full backup, then each of the incrementals in turn;
an incremental backup acts as if it's a full backup, it just takes
less disk space.
You can mount your bup repository as a FUSE filesystem and access
the content that way, and even export it over Samba.
