Changes in Version 2.40.61
Changes since 2.40.1:
* Added "BelowPath" patterns, that match a path as well as all paths
below (convenient to use with nodeletion,update,creationpartial
preferences)
* Added a "fat" preference that makes Unison use the right options
when one of the replica is on a FAT filesystem.
* Allow "prefer/force=newer" even when not synchronizing
modification times. (The reconciler will not be aware of the
modification time of unchanged files, so the synchronization
choices of Unison can be different from when "times=true", but the
behavior remains sane: changed files with the most recent
modification time will be propagated.)
* Minor fixes and improvements:
+ Compare filenames up to decomposition in case sensitive mode
when one host is running MacOSX and the unicode preference is
set to true.
+ Rsync: somewhat faster compressor
+ Make Unicode the default on all architectures (it was only
the default when a Mac OS X or Windows machine was involved).
Changes since 2.32:
* Major enhancement: Unicode support.
+ Unison should now handle unicode filenames correctly on all
platforms.
+ This functionality is controlled by a new preference unicode.
+ Unicode mode is now the default when one of the hosts is
under Windows or MacOS. This may make upgrades a bit more
painful (the archives cannot be reused), but this is a much
saner default.
* Partial transfer of directories. If an error occurs while
transferring a directory, the part transferred so far is copied
into place (and the archives are updated accordingly). The
"maxerrors" preference controls how many transfer error Unison
will accept before stopping the transfer of a directory (by
default, only one). This makes it possible to transfer most of a
directory even if there are some errors. Currently, only the first
error is reported by the GUIs.
Also, allow partial transfer of a directory when there was an
error deep inside this directory during update detection. At the
moment, this is only activated with the text and GTK UIs, which
have been modified so that they show that the transfer is going to
be partial and so that they can display all errors.
* Improvement to the code for resuming directory transfers:
+ if a file was not correctly transferred (or the source has
been modified since, with unchanged size), Unison performs a
new transfer rather than failing
+ spurious files are deleted (this can happen if a file is
deleted on the source replica before resuming the transfer;
not deleting the file would result in it reappearing on the
target replica)
* Experimental streaming protocol for transferring file contents
(can be disabled by setting the directive "stream" to false): file
contents is transfered asynchronously (without waiting for a
response from the destination after each chunk sent) rather than
using the synchronous RPC mechanism. As a consequence:
+ Unison now transfers the contents of a single file at a time
(Unison used to transfer several contents simultaneously in
order to hide the connection latency.)
+ the transfer of large files uses the full available bandwidth
and is not slowed done due to the connection latency anymore
+ we get performance improvement for small files as well by
scheduling many files simultaneously (as scheduling a file
for transfer consume little ressource: it does not mean
allocating a large buffer anymore)
* Changes to the internal implementation of the rsync algorithm:
+ use longer blocks for large files (the size of a block is the
square root of the size of the file for large files);
+ transmit less checksum information per block (we still have
less than one chance in a hundred million of transferring a
file incorrectly, and Unison will catch any transfer error
when fingerprinting the whole file)
+ avoid transfer overhead (which was 4 bytes per block)
For a 1G file, the first optimization saves a factor 50 on the
amount of data transferred from the target to the source (blocks
are 32768 bytes rather than just 700 bytes). The two other
optimizations save another factor of 2 (from 24 bytes per block
down to 10).
* Implemented an on-disk file fingerprint cache to speed-up update
detection after a crash: this way, Unison does not have do
recompute all the file fingerprints from scratch.
+ When Unison detects that the archive case-sensitivity mode
does not match the current settings, it populates the
fingerprint cache using the archive contents. This way,
changing the case-sensitivity mode should be reasonably fast.
* New preferences "noupdate=root", "nodeletion=root",
"nocreation=root" that prevent Unison from performing files
updates, deletions or creations on the given root. Also 'partial'
versions of 'noupdate', 'nodeletion' and 'nocreation'
* Limit the number of simultaneous external copy program ("copymax"
preference)
* New "links" preference. When set to false, Unison will report an
error on symlinks during update detection. (This is the default
when one host is running Windows but not Cygwin.) This is better
than failing during propagation.
* Added a preference "halfduplex" to force half-duplex communication
with the server. This may be useful on unreliable links (as a more
efficient alternative to "maxthreads = 1").
* Renamed preference "pretendwin" to "ignoreinodenumbers" (an alias
is kept for backwards compatibility).
* Ignore one-second differences when synchronizing modification
time. (Technically, this is an incompatible archive format change,
but it is backward compatible. To trigger a problem, a user would
have to synchronize modification times on a filesystem with a
two-second granularity and then downgrade to a previous version of
Unison, which does not work well in such a case. Thus, it does not
seem worthwhile to increment the archive format number, which
would impact all users.)
* Do not keep many files simultaneously opened anymore when the
rsync algorithm is in use.
* Add "ignorearchives" preference to ignore existing archives (to
avoid forcing users to delete them manually, in situations where
one archive has gotten deleted or corrupted).
* Mac OS
+ fixed rsync bug which could result in an "index out of
bounds" error when transferring resource forks.
+ Fixed bug which made Unison ignore finder information and
resource fork when compiled to 64bit on Mac OSX.
+ should now be 64 bit clean (the Growl framework is not up to
date, though)
+ Made the bridge between Objective C and Ocaml code GC
friendly (it was allocating ML values and putting them in an
array which was not registered with the GC)
+ use darker grey arrows (patch contributed by Eric Y. Kow)
* GTK user interface
+ assistant for creating profiles
+ profile editor
+ pop up a summary window when the replicas are not fully
synchronized after transport
+ display estimated remaining time and transfer rate on the
progress bar
+ allow simultaneous selection of several items
+ Do not reload the preference file before a new update
detection if it is unchanged
+ disabled scrolling to the first unfinished item during
transport. It goes way too fast when lot of small files are
synchronized, and it makes it impossible to browse the file
list during transport.
+ take into account the "height" preference again
+ the internal list of selected reconciler item was not always
in sync with what was displayed (GTK bug?); workaround
implemented
+ Do not display "Looking for change" messages during
propagation (when checking the targe is unchanged) but only
during update detection
+ Apply patch to fix some crashes in the OSX GUI, thanks to
Onne Gorter.
* Text UI
+ During update detection, display status by updating a single
line rather than generating a new line of output every so
often. Should be less confusing.
* Windows
+ Fastcheck is now the default under Windows. People mostly use
NTFS nowadays and the Unicode API provides an equivalent to
inode numbers for this filesystem.
+ Only use long UNC path for accessing replicas (as '..' is not
handled with this format of paths, but can be useful)
+ Windows text UI: now put the console into UTF-8 output mode.
This is the right thing to do when in Unicode mode, and is no
worse than what we had previously otherwise (the console use
some esoteric encoding by default). This only works when
using a Unicode font instead of the default raster font.
+ Don't get the home directory from environment variable HOME
under Windows (except for Cygwin binaries): we don't want the
behavior of Unison to depends on whether it is run from a
Cygwin shell (where HOME is set) or in any other way (where
HOME is usually not set).
* Miscellaneous fixes and improvements
+ Made a server waiting on a socket more resilient to
unexpected lost connections from the client.
+ Small patch to property setting code suggested by Ulrich
Gernkow.
+ Several fixes to the change transfer functions (both the
internal ones and external transfers using rsync). In
particular, limit the number of simultaneous transfer using
an rsync (as the rsync algorithm can use a large amount of
memory when processing huge files)
+ Keep track of which file contents are being transferred, and
delay the transfer of a file when another file with the same
contents is currently being transferred. This way, the second
transfer can be skipped and replaced by a local copy.
+ Experimental update detection optimization: do not read the
contents of unchanged directories
+ When a file transfer fails, turn off fastcheck for this file
on the next sync.
+ Fixed bug with case insensitive mode on a case sensitive
filesystem:
o if file "a/a" is created on one replica and directory
"A" is created on the other, the file failed to be
synchronized the first time Unison is run afterwards, as
Unison uses the wrong path "a/a" (if Unison is run
again, the directories are in the archive, so the right
path is used);
o if file "a" appears on one replica and file "A" appears
on the other with different contents, Unison was unable
to synchronize them.
+ Improved error reporting when the destination is updated
during synchronization: Unison now tells which file has been
updated, and how.
+ Limit the length of temporary file names
+ Case sensitivity information put in the archive (in a
backward compatible way) and checked when the archive is
loaded
+ Got rid of the 16mb marshalling limit by marshalling to a
bigarray.
+ Resume copy of partially transferred files.
Set license.
Changes since 2.32.44:
* Improvement to the code for resuming directory transfers: (1) make sure file information (permissions, ...) has been properly set when using a previously transferred temp file (2) make sure previously transferred directories are writable (other changes made in the developer version of Unison require a protocol change)
* Got rid of the 16MiB marshalling limit by marshalling to a bigarray
* Ignore one hour differences for deciding whether a file may have been updated. This avoids slow update detection after daylight saving time changes under Windows. This makes it slightly more likely to miss an update, but that should be safe enough.
* Improved Unison icon under Windows
* Case sensitivity information put in the archive (in a backward compatible way) and checked when the archive is loaded
* Uses improved emulation of "select" call provided by Ocaml 3.11 under Windows (the GUI does not freeze as much during synchronization)
* Upgraded to GPL version 3 and added copyright notice to documentation files.
* Unison can sometimes fail to transfer a file, giving the unhelpful message "Destination updated during synchronization" even though the file has not been changed. This can be caused by programs that change either the file's contents *or* the file's extended attributes without changing its modification time. I'm not sure what is the best fix for this – it is not Unison's fault, but it makes Unison's behavior puzzling – but at least Unison can be more helpful about suggesting a workaround (running once with 'fastcheck' set to false). The failure message has been changed to give this advice.
* Text UI
o During update detection, display status by updating a single line rather than generating a new line of output every so often. That should be less confusing.
o In repeat mode, don't save the archives when there is no update. Indeed, in this mode, we should minimize the amount of work performed and it is unlikely that the archives have changed much.
* Bugfixes
o Fixed quotation of paths and names when writing to a preference file
o Fixed bug resulting in slow performances when transferring a file using our rsync implementation from a 64-bit architecture to a 32-bit architecture.
o Fixed bug in Lwt_unix.run which could make it fail with a Not_found exception (see [Not_found raised in tryCopyMovedFile] errors)
o Properly deals with non-conformant AppleDouble files produced by Mac OS X.
o Fixed bug that results in Unison missing ressource fork changes
o Applied a patch from Karl M to make the GTK2 version build with OCaml 3.11 on Windows.
o Added some extra debugging code to remote.ml to give more informative error messages when people encounter the longstanding "assert failed during file transfer" bug.
o Applied patch from Antoine Reilles for NetBSD compilation
o Resizing the update window vertically no longer moves the status label. Fix contributed by Pedro Melo.
Changes since 2.31:
* Minor fixes and improvements:
o Ignore one hour differences when deciding whether a file may have been updated. This avoids slow update detection after daylight saving time changes under Windows. This makes Unison slightly more likely to miss an update, but it should be safe enough.
o Fix a small bug that was affecting mainly windows users. We need to commit the archives at the end of the sync even if there are no updates to propagate because some files (in fact, if we've just switched to DST on windows, a LOT of files) might have new modtimes in the archive. (Changed the text UI only. It's less clear where to change the GUI.)
o Don't delete the temp file when a transfer fails due to a fingerprint mismatch (so that we can have a look and see why!) We've also added more debugging code togive more informative error messages when we encounter the dreaded and longstanding "assert failed during file transfer" bug
Changes since 2.27:
* If Unison is interrupted during a directory transfer, it will now leave the partially transferred directory intact in a temporary location. (This maintains the invariant that new files/directories are transferred either completely or not at all.) The next time Unison is run, it will continue filling in this temporary directory, skipping transferring files that it finds are already there.
* We've added experimental support for invoking an external file transfer tool for whole-file copies instead of Unison's built-in transfer protocol. Three new preferences have been added:
o copyprog is a string giving the name (and command-line switches, if needed) of an external program that can be used to copy large files efficiently. By default, rsync is invoked, but other tools such as scp can be used instead by changing the value of this preference. (Although this is not its primary purpose, rsync is actually a pretty fast way of copying files that don't already exist on the receiving host.) For files that do already exist on (but that have been changed in one replica), Unison will always use its built-in implementation of the rsync algorithm.
o Added a "copyprogrest" preference, so that we can give different command lines for invoking the external copy utility depending on whether a partially transferred file already exists or not. (Rsync doesn't seem to care about this, but other utilities may.)
o copythreshold is an integer (-1 by default), indicating above what filesize (in megabytse the external copying utility specified by copyprog. Specifying 0 will cause ALL copies to use the external program; a negative number will prevent any files from using it. (Default is -1.)
Thanks to Alan Schmitt for a huge amount of hacking and o an anonymous sponsor for suggesting and underwriting this extension.
* Small improvements:
o Added a new preference, dontchmod. By default, Unison uses the chmod system call to set the permission bits of files after it has copied them. Butin some circumstances (and under some operating systems), the chmod call always fails. Setting this preference completely prevents Unison from ever calling chmod.
o Don't ignore files that look like backup files if the backuplocation preference set to central
o Shortened the names of several preferences. The old names are also still supported, for backwards compatibility, but they do not appear in the documentation.
o Lots of little documentation tidying. (In particular, preferences are separated into Basic and Advanced! This should hopefully make Unison a little more approachable for new users.
o Unison can sometimes fail to transfer a file, giving the unhelpful message "Destination updated during synchronization"
#
* Further improvements to the OS X GUI (thanks to Alan Schmitt and Craig Federighi).
# Very preliminary support for triggering Unison from an external filesystem-watching utility. The current implementation is very simple, not efficient, and almost completely untested—not ready for real users. But if someone wants to help improve it (e.g., by writing a filesystem watcher for your favorite OS), please make yourself known!
On the Unison side, the new behavior is very simple:
* use the text UI
* start Unison with the command-line flag "-repeat FOO", where FOO is name of a file where Unison should look for notifications of changes
* when it starts up, Unison will read the whole contents of this file (on both hosts), which should be a newline-separated list of paths (relative to the root of the synchronization) and synchronize just these paths, as if it had been started with the "-path=xxx" option for each one of them
* when it finishes, it will sleep for a few seconds and then examine the watchfile again; if anything has been added, it will read the new paths, synchronize them, and go back to sleep
* that's it!
To use this to drive Unison "incrementally," just start it in this mode and start up a tool (on each host) to watch for new changes to the filesystem and append the appropriate paths to the watchfile. Hopefully such tools should not be too hard to write.
# Bug fixes:
* Fixed a bug that was causing new files to be created with permissions 0x600 instead of using a reasonable default (like 0x644), if the 'perms' flag was set to 0. (Bug reported by Ben Crowell.)
* Follow maxthreads preference when transferring directories.
Enable full destdir support.
Changes since 2.17:
* Several small fixes to the GTK2 UI to make it work better under Windows
* The backup functionality has been completely rewritten. The external
interface has not changed, but numerous bugs, irregular behaviors, and
cross-platform inconsistencies have been corrected.
* Some important safety improvements:
o Added a new mountpoint preference, which can be used to specify a path
that must exist in both replicas at the end of update detection
o The confirmation of “big deletes” is now controlled by a boolean
preference confirmbigdeletes. Default is true, which gives the same behavior
as previously. (This functionality is at least partly superceded by the
mountpoint preference, but it has been left in place in case it is useful to
some people.)
o If Unison is asked to “follow” a symbolic link but there is nothing at
the other end of the link, it will now flag this path as an error, rather
than treating the symlink itself as missing or deleted. This avoids a
potentially dangerous situation where a followed symlink points to an
external filesystem that might be offline when Unison is run (whereupon
Unison would cheerfully delete the corresponding files in the other
replica!).
* Smaller changes:
o Added forcepartial and preferpartial preferences, which behave like force
and prefer but can be specified on a per-path basis. [Thanks to Alan Schmitt
for this.]
o A bare-bones self test feature was added, which runs unison through some
of its paces and checks that the results are as expected. The coverage of the
tests is still very limited, but the facility has already been very useful in
debugging the new backup functionality (especially in exposing some subtle
cross-platform issues).
o Refined debugging code so that the verbosity of individual modules can be
controlled separately. Instead of just putting '-debug verbose' on the
command line, you can put '-debug update+', which causes all the extra
messages in the Update module, but not other modules, to be printed. Putting
'-debug verbose' causes all modules to print with maximum verbosity.
o Removed mergebatch preference. (It never seemed very useful, and its
semantics were confusing.)
o Rewrote some of the merging functionality, for better cooperation with
external Harmony instances.
o Changed the temp file prefix from .# to .unison.
o Compressed the output from the text user interface (particularly when run
with the -terse flag) to make it easier to interpret the results when Unison
is run several times in succession from a script.
o Changed the order of arguments to the default diff command (so that the +
and - annotations in diff's output are reversed).
o Small fix to ctime (non-)handling in update detection under windows with
fastcheck.
RECOMMENDED is removed. It becomes ABI_DEPENDS.
BUILDLINK_RECOMMENDED.foo becomes BUILDLINK_ABI_DEPENDS.foo.
BUILDLINK_DEPENDS.foo becomes BUILDLINK_API_DEPENDS.foo.
BUILDLINK_DEPENDS does not change.
IGNORE_RECOMMENDED (which defaulted to "no") becomes USE_ABI_DEPENDS
which defaults to "yes".
Added to obsolete.mk checking for IGNORE_RECOMMENDED.
I did not manually go through and fix any aesthetic tab/spacing issues.
I have tested the above patch on DragonFly building and packaging
subversion and pkglint and their many dependencies.
I have also tested USE_ABI_DEPENDS=no on my NetBSD workstation (where I
have used IGNORE_RECOMMENDED for a long time). I have been an active user
of IGNORE_RECOMMENDED since it was available.
As suggested, I removed the documentation sentences suggesting bumping for
"security" issues.
As discussed on tech-pkg.
I will commit to revbump, pkglint, pkg_install, createbuildlink separately.
Note that if you use wip, it will fail! I will commit to pkgsrc-wip
later (within day).
directory or try to create it: it causes the build to fail under darwin
More: the INSTALL file is not necessary, since unison -doc install will show it
Main changelog entries:
- Incompatible change: Archive format has changed.
- Substantial reworking of the internal logic of the text UI
- Finder flags now synchronized (for mac osx users)
- Workaround for transferring large files using rsync.
Approved by cube@
- Lots of changes, in summary:
- out of the box OS X and OpenBSD support
- limit the number of simultaneous file transfers
- support for ocaml >3.07
- Files larger than 2Gb are now supported
- Major tidying and enhancement of 'merge' functionality
in the process. (More information on tech-pkg.)
Bump PKGREVISION and BUILDLINK_DEPENDS of all packages using libtool and
installing .la files.
Bump PKGREVISION (only) of all packages depending directly on the above
via a buildlink3 include.
