Important note about pre-1.0.4 potential repo corruption
--------------------------------------------------------
Some external errors (like network or disk I/O errors) could lead to
corruption of the backup repository due to issue #1138.
A sign that this happened is if "E" status was reported for a file that can
not be explained by problems with the source file. If you still have logs from
"borg create -v --list", you can check for "E" status.
Here is what could cause corruption and what you can do now:
1) I/O errors (e.g. repo disk errors) while writing data to repo.
This could lead to corrupted segment files.
Fix::
# check for corrupt chunks / segments:
borg check -v --repository-only REPO
# repair the repo:
borg check -v --repository-only --repair REPO
# make sure everything is fixed:
borg check -v --repository-only REPO
2) Unreliable network / unreliable connection to the repo.
This could lead to archive metadata corruption.
Fix::
# check for corrupt archives:
borg check -v --archives-only REPO
# delete the corrupt archives:
borg delete --force REPO::CORRUPT_ARCHIVE
# make sure everything is fixed:
borg check -v --archives-only REPO
3) In case you want to do more intensive checking.
The best check that everything is ok is to run a dry-run extraction::
borg extract -v --dry-run REPO::ARCHIVE
Version 1.0.7 (2016-08-19)
--------------------------
Security fixes:
- borg serve: fix security issue with remote repository access, #1428
If you used e.g. --restrict-to-path /path/client1/ (with or without trailing
slash does not make a difference), it acted like a path prefix match using
/path/client1 (note the missing trailing slash) - the code then also allowed
working in e.g. /path/client13 or /path/client1000.
As this could accidentally lead to major security/privacy issues depending on
the pathes you use, the behaviour was changed to be a strict directory match.
That means --restrict-to-path /path/client1 (with or without trailing slash
does not make a difference) now uses /path/client1/ internally (note the
trailing slash here!) for matching and allows precisely that path AND any
path below it. So, /path/client1 is allowed, /path/client1/repo1 is allowed,
but not /path/client13 or /path/client1000.
If you willingly used the undocumented (dangerous) previous behaviour, you
may need to rearrange your --restrict-to-path pathes now. We are sorry if
that causes work for you, but we did not want a potentially dangerous
behaviour in the software (not even using a for-backwards-compat option).
Bug fixes:
- fixed repeated LockTimeout exceptions when borg serve tried to write into
a already write-locked repo (e.g. by a borg mount), #502 part b)
This was solved by the fix for #1220 in 1.0.7rc1 already.
- fix cosmetics + file leftover for "not a valid borg repository", #1490
- Cache: release lock if cache is invalid, #1501
- borg extract --strip-components: fix leak of preloaded chunk contents
- Repository, when a InvalidRepository exception happens:
- fix spurious, empty lock.roster
- fix repo not closed cleanly
New features:
- implement borg debug-info, fixes#1122
(just calls already existing code via cli, same output as below tracebacks)
Other changes:
- skip the O_NOATIME test on GNU Hurd, fixes#1315
(this is a very minor issue and the GNU Hurd project knows the bug)
- document using a clean repo to test / build the release
Version 1.0.7rc2 (2016-08-13)
-----------------------------
Bug fixes:
- do not write objects to repository that are bigger than the allowed size,
borg will reject reading them, #1451.
Important: if you created archives with many millions of files or
directories, please verify if you can open them successfully,
e.g. try a "borg list REPO::ARCHIVE".
- lz4 compression: dynamically enlarge the (de)compression buffer, the static
buffer was not big enough for archives with extremely many items, #1453
- larger item metadata stream chunks, raise archive item limit by 8x, #1452
- fix untracked segments made by moved DELETEs, #1442
Impact: Previously (metadata) segments could become untracked when deleting data,
these would never be cleaned up.
- extended attributes (xattrs) related fixes:
- fixed a race condition in xattrs querying that led to the entire file not
being backed up (while logging the error, exit code = 1), #1469
- fixed a race condition in xattrs querying that led to a crash, #1462
- raise OSError including the error message derived from errno, deal with
path being a integer FD
Other changes:
- print active env var override by default, #1467
- xattr module: refactor code, deduplicate, clean up
- repository: split object size check into too small and too big
- add a transaction_id assertion, so borg init on a broken (inconsistent)
filesystem does not look like a coding error in borg, but points to the
real problem.
- explain confusing TypeError caused by compat support for old servers, #1456
- add forgotten usage help file from build_usage
- refactor/unify buffer code into helpers.Buffer class, add tests
- docs:
- document archive limitation, #1452
- improve prune examples
Version 1.0.7rc1 (2016-08-05)
-----------------------------
Bug fixes:
- fix repo lock deadlocks (related to lock upgrade), #1220
- catch unpacker exceptions, resync, #1351
- fix borg break-lock ignoring BORG_REPO env var, #1324
- files cache performance fixes (fixes unneccessary re-reading/chunking/
hashing of unmodified files for some use cases):
- fix unintended file cache eviction, #1430
- implement BORG_FILES_CACHE_TTL, update FAQ, raise default TTL from 10
to 20, #1338
- FUSE:
- cache partially read data chunks (performance), #965, #966
- always create a root dir, #1125
- use an OrderedDict for helptext, making the build reproducible, #1346
- RemoteRepository init: always call close on exceptions, #1370 (cosmetic)
- ignore stdout/stderr broken pipe errors (cosmetic), #1116
New features:
- better borg versions management support (useful esp. for borg servers
wanting to offer multiple borg versions and for clients wanting to choose
a specific server borg version), #1392:
- add BORG_VERSION environment variable before executing "borg serve" via ssh
- add new placeholder {borgversion}
- substitute placeholders in --remote-path
- borg init --append-only option (makes using the more secure append-only mode
more convenient. when used remotely, this requires 1.0.7+ also on the borg
server), #1291.
Other changes:
- Vagrantfile:
- darwin64: upgrade to FUSE for macOS 3.4.1 (aka osxfuse), #1378
- xenial64: use user "ubuntu", not "vagrant" (as usual), #1331
- tests:
- fix fuse tests on OS X, #1433
- docs:
- FAQ: add backup using stable filesystem names recommendation
- FAQ about glibc compatibility added, #491, glibc-check improved
- FAQ: 'A' unchanged file; remove ambiguous entry age sentence.
- OS X: install pkg-config to build with FUSE support, fixes#1400
- add notes about shell/sudo pitfalls with env. vars, #1380
- added platform feature matrix
- implement borg debug-dump-repo-objs
Version 1.0.6 (2016-07-12)
--------------------------
Bug fixes:
- Linux: handle multiple LD_PRELOAD entries correctly, #1314, #1111
- Fix crash with unclear message if the libc is not found, #1314, #1111
Other changes:
- tests:
- Fixed O_NOATIME tests for Solaris and GNU Hurd, #1315
- Fixed sparse file tests for (file) systems not supporting it, #1310
- docs:
- Fixed syntax highlighting, #1313
- misc docs: added data processing overview picture
Version 1.0.6rc1 (2016-07-10)
-----------------------------
New features:
- borg check --repair: heal damaged files if missing chunks re-appear (e.g. if
the previously missing chunk was added again in a later backup archive),
#148. (*) Also improved logging.
Bug fixes:
- sync_dir: silence fsync() failing with EINVAL, #1287
Some network filesystems (like smbfs) don't support this and we use this in
repository code.
- borg mount (FUSE):
- fix directories being shadowed when contained paths were also specified,
#1295
- raise I/O Error (EIO) on damaged files (unless -o allow_damaged_files is
used), #1302. (*)
- borg extract: warn if a damaged file is extracted, #1299. (*)
- Added some missing return code checks (ChunkIndex._add, hashindex_resize).
- borg check: fix/optimize initial hash table size, avoids resize of the table.
Other changes:
- tests:
- add more FUSE tests, #1284
- deduplicate fuse (u)mount code
- fix borg binary test issues, #862
- docs:
- changelog: added release dates to older borg releases
- fix some sphinx (docs generator) warnings, #881
Notes:
(*) Some features depend on information (chunks_healthy list) added to item
metadata when a file with missing chunks was "repaired" using all-zero
replacement chunks. The chunks_healthy list is generated since borg 1.0.4,
thus borg can't recognize such "repaired" (but content-damaged) files if the
repair was done with an older borg version.
Version 1.0.5 (2016-07-07)
--------------------------
Bug fixes:
- borg mount: fix FUSE crash in xattr code on Linux introduced in 1.0.4, #1282
Other changes:
- backport some FAQ entries from master branch
- add release helper scripts
- Vagrantfile:
- centos6: no FUSE, don't build binary
- add xz for redhat-like dists
Version 1.0.4 (2016-07-07)
--------------------------
New features:
- borg serve --append-only, #1168
This was included because it was a simple change (append-only functionality
was already present via repository config file) and makes better security now
practically usable.
- BORG_REMOTE_PATH environment variable, #1258
This was included because it was a simple change (--remote-path cli option
was already present) and makes borg much easier to use if you need it.
- Repository: cleanup incomplete transaction on "no space left" condition.
In many cases, this can avoid a 100% full repo filesystem (which is very
problematic as borg always needs free space - even to delete archives).
Bug fixes:
- Fix wrong handling and reporting of OSErrors in borg create, #1138.
This was a serious issue: in the context of "borg create", errors like
repository I/O errors (e.g. disk I/O errors, ssh repo connection errors)
were handled badly and did not lead to a crash (which would be good for this
case, because the repo transaction would be incomplete and trigger a
transaction rollback to clean up).
Now, error handling for source files is cleanly separated from every other
error handling, so only problematic input files are logged and skipped.
- Implement fail-safe error handling for borg extract.
Note that this isn't nearly as critical as the borg create error handling
bug, since nothing is written to the repo. So this was "merely" misleading
error reporting.
- Add missing error handler in directory attr restore loop.
- repo: make sure write data hits disk before the commit tag (#1236) and also
sync the containing directory.
- FUSE: getxattr fail must use errno.ENOATTR, #1126
(fixes Mac OS X Finder malfunction: "zero bytes" file length, access denied)
- borg check --repair: do not lose information about the good/original chunks.
If we do not lose the original chunk IDs list when "repairing" a file
(replacing missing chunks with all-zero chunks), we have a chance to "heal"
the file back into its original state later, in case the chunks re-appear
(e.g. in a fresh backup). Healing is not implemented yet, see #148.
- fixes for --read-special mode:
- ignore known files cache, #1241
- fake regular file mode, #1214
- improve symlinks handling, #1215
- remove passphrase from subprocess environment, #1105
- Ignore empty index file (will trigger index rebuild), #1195
- add missing placeholder support for --prefix, #1027
- improve exception handling for placeholder replacement
- catch and format exceptions in arg parsing
- helpers: fix "undefined name 'e'" in exception handler
- better error handling for missing repo manifest, #1043
- borg delete:
- make it possible to delete a repo without manifest
- borg delete --forced allows to delete corrupted archives, #1139
- borg check:
- make borg check work for empty repo
- fix resync and msgpacked item qualifier, #1135
- rebuild_manifest: fix crash if 'name' or 'time' key were missing.
- better validation of item metadata dicts, #1130
- better validation of archive metadata dicts
- close the repo on exit - even if rollback did not work, #1197.
This is rather cosmetic, it avoids repo closing in the destructor.
- tests:
- fix sparse file test, #1170
- flake8: ignore new F405, #1185
- catch "invalid argument" on cygwin, #257
- fix sparseness assertion in test prep, #1264
Other changes:
- make borg build/work on OpenSSL 1.0 and 1.1, #1187
- docs / help:
- fix / clarify prune help, #1143
- fix "patterns" help formatting
- add missing docs / help about placeholders
- resources: rename atticmatic to borgmatic
- document sshd settings, #545
- more details about checkpoints, add split trick, #1171
- support docs: add freenode web chat link, #1175
- add prune visualization / example, #723
- add note that Fnmatch is default, #1247
- make clear that lzma levels > 6 are a waste of cpu cycles
- add a "do not edit" note to auto-generated files, #1250
- update cygwin installation docs
- repository interoperability with borg master (1.1dev) branch:
- borg check: read item metadata keys from manifest, #1147
- read v2 hints files, #1235
- fix hints file "unknown version" error handling bug
- tests: add tests for format_line
- llfuse: update version requirement for freebsd
- Vagrantfile:
- use openbsd 5.9, #716
- do not install llfuse on netbsd (broken)
- update OSXfuse to version 3.3.3
- use Python 3.5.2 to build the binaries
- glibc compatibility checker: scripts/glibc_check.py
- add .eggs to .gitignore
Version 1.0.3 (2016-05-20)
--------------------------
Bug fixes:
- prune: avoid that checkpoints are kept and completed archives are deleted in
a prune run), #997
- prune: fix commandline argument validation - some valid command lines were
considered invalid (annoying, but harmless), #942
- fix capabilities extraction on Linux (set xattrs last, after chown()), #1069
- repository: fix commit tags being seen in data
- when probing key files, do binary reads. avoids crash when non-borg binary
files are located in borg's key files directory.
- handle SIGTERM and make a clean exit - avoids orphan lock files.
- repository cache: don't cache large objects (avoid using lots of temp. disk
space), #1063
Other changes:
- Vagrantfile: OS X: update osxfuse / install lzma package, #933
- setup.py: add check for platform_darwin.c
- setup.py: on freebsd, use a llfuse release that builds ok
- docs / help:
- update readthedocs URLs, #991
- add missing docs for "borg break-lock", #992
- borg create help: add some words to about the archive name
- borg create help: document format tags, #894
c437def87b