Preamble
--------

This software is ready for testing by a wider audience. Some parts of
this documentation describe how features are supposed to work although
they have not been implemented yet. They are marked with [not
implemented].

There's no home page yet. Please refer to the SourceForge page instead:
    http://sourceforge.net/projects/sync4jevolution/
Because the original name "sync4jevolution" is not really appropriate,
it was changed to SyncEvolution. The web page and project name will
be changed eventually, too.


Introduction
------------

SyncEvolution synchronizes Evolution's contact, calender and task
items with a SyncML server. The items are exchanged in the vCard 2.1
or 3.0 format and iCalender 2.0 format via the Funambol Sync4j C++
client API library, which should make SyncEvolution compatible with
the majority of SyncML servers. Full, one-way and incremental
synchronization of items are supported.

SyncEvolution does not synchronize with another SyncML capable
device directly. A SyncML server that that device and SyncEvolution
can talk to is needed. There are several options for that:
- using a web service like www.scheduleworld.com which stores the
  data to be synchronized on a scheduleworld server and provides
  access to it via SyncML
- installing a SyncML server like Funambol's Sync4j on one's own server
- installing a SyncML server on the desktop

The recommended solution is www.scheduleworld.com because it is easier
than setting up a server and provides better support for vCard and
iCalendar data than the stock Sync4j installation. Setting up a server
on the desktop has the additional problem that not all mobile devices
can communicate with the desktop via HTTP.

[ not quite yet: scheduleworld still has some problems that the
  admin is working on - the text above was written for SyncEvolution
  0.3, which will be released once the problems have been resolved ]

The remainder of this document assumes that either Funambol's Sync4j
server bundle for Linux V2.3 was installed using the default
configuration or scheduleworld is used: because scheduleworld uses
Sync4j, configuration and usage are often similar.

With a server that fully supports SyncML and vCard/iCalender
the following works:
- copy a complete database to the server and restore it
  from the server later
- delete or modify an item locally, then make the same change
  on the server
- delete, modify or add items on the server (by synchronizing with
  another client), then make the same change locally
- conflict resolution (where two clients modify the same item,
  then sync with the server) is handled by the server, but
  SyncEvolution has support which ensures that no data is lost
  by creating duplicates (see "Conflict Resolution" below)

For conflict resolution and synchronization between clients which
support different attributes of items the server needs an
understanding of the format of items. The Sync4j server supports that
for contacts and calendars but not tasks; see "Configuration with
Sync4j" below for more information. scheduleworld also supports that
for calendars and tasks.


Installation
------------

To install SyncEvolution, just unpack an archive with a precompiled
binary for your platform in a directory of your choice. Then create
a configuration in $HOME/.sync4j/evolution as described below under
"Configuration". No special environment variables are needed, although
one might want to add the directory with contains the "syncevolution"
binary to the shell's PATH variable.

When a binary packages is not available for the target system
and/or is not up-to-date, compiling from source can also be used
to produce a binary. See below in "Compiling from Source" for details.

You also need a working SyncML server. If you do not have and/or
cannot get an account on an existing one, installing the Funambol
Sync4j server bundle is very easy. See:
   http://www.funambol.com/opensource/downloads.html

For Sync4j V2.3, an additional patch is recommended to preserve
line breaks of items on the server:
   http://forge.objectweb.org/tracker/index.php?func=detail&aid=304718&group_id=96&atid=300096

Although all of the features are covered by unit testing and
have been verified to work, it is highly recommended that you
make a backup of your
   $HOME/.evolution/addressbook
   $HOME/.evolution/calender
   $HOME/.evolution/tasks
directories before running it for the first time. In older Evolution
versions the same data is found in $HOME/evolution.


Usage
-----

Currently SyncEvolution comes as a simple command line tool which is
configured via files. A graphical interface via an Evolution plugin
would also be possible, but is not implemented yet. As command line
parameters SyncEvolution only supports one option which specifies the
configuration file that drives the synchronization run:

   syncevolution [<server>] [<source> ...]

The <server> string is used to find the configuration which determines
how synchronization is going to proceed. Selection of sources of
Evolution data which are to be synchronized with that server is done
via configuration files. It is possible to configure sources without
activating their synchronization: if the "sync" property of a source
is set to "none", it will be ignore. Explicitely listing one or more
source names after the server chooses just those sources for
synchronization with configured sync mode or "two-way" if a source
is currently set to "none".

If the SyncML server is not specified, SyncEvolution lists all
available Evolution backend databases.

Progress and error messages are written into a log file that is
preserved for each synchronization run. Details about that is found
in the "Automatic Backups and Logging" section below. Immediately
before quitting SyncEvolution will show all errors or warnings
encountered and print a summary of how the databases were modified.
This is done with the "synccompare" utility script described
in the "Exchanging Data" section.

In case of an error the synchronization run is aborted prematurely and
SyncEvolution will return a non-zero value. Recovery from failed
synchronization is done by forcing a full synchronization during the
next run, i.e. by sending all items and letting the SyncML server
compare against the ones it already knows. If the error is detected
by SyncEvolution itself, then it attempts to cause the full
synchronization only for the failed database, but if the SyncML
client library itself runs into a problem it might cause a full
synchronization of all databases even if only one of them failed.
Therefore it might be less risky to invoke syncevolution for each
database separately.

After a successful synchronization the server's configuration file is
updated so that the next run can be done incrementally.  If the
configuration file has to be recreated e.g. because it was lost, the
next run recovers from that by doing a full synchronization. The risk
associated with this is that the server might not recognize items that
it already has stored previously which then would lead to duplication
of items.


Configuration
-------------

The configuration file of a certain <server> is stored in
   $HOME/.sync4j/evolution/<server>/spds/syncml/config.txt

The format is a simple list of
   <property> = <value>
pairs with one pair per line. Leading spaces and space around the
equals character are skipped. <value> then runs until the the end of
the line. In other words, it cannot start with spaces nor contain line
breaks. Do not put quotation marks around <value>, they would be
treated as part of the value itself. Lines starting with a hash (#)
after optional leading spaces are treated as comments and skipped.

Each data source is configured in
   $HOME/.sync4j/evolution/<server>/spds/sources/<source>/config.txt

See "etc/example/spds/syncml/config.txt" for options in the server
configuration and "etc/example/spds/sources/addressbook_1/config.txt"
for options in the data source configuration. Without changing this
example configuration can be used for testing the operation of
SyncEvolution, see "Exchanging Data" below.

Normally at least the following configuration options need to be adapted:
  spds/syncml
      syncURL
      deviceId
      username
      password
  spds/sources
      uri
      evolutionsource

Each data source corresponds to one database at the SyncML server.
The Evolution data source is determined by the type of data given in
"type" and uniquely identified with the "evolutionurl" property.

One can synchronize with multiple server databases in one run, but
the same server database can only be accessed once. To synchronize
the same server database with multiple local databases, one has
to setup two independent configurations with different "deviceId"
settings and synchronize them separately.

If the Evolution data source requires authentication, the
"evolutionuser" and "evolutionpassword" are used as credentials.  In
this case the directory that contains the source's config.txt should
only be accessible by the user. [NOT IMPLEMENTED YET]


Automatic Backups and Logging
-----------------------------

To support recovery from a synchronization which damaged the
local database or modified it in an unexpected way, SyncEvolution
always creates the following files during a synchronization:
- a dump of the database in a format which can be imported
  back into Evolution, e.g. .vcf for address books
- a full log file with debug information
- a dump of the database after the synchronization for
  automatic comparison of the before/after state with
  "normalize_vcard"

If the source configuration option "logdir" is set, then
a new directory will be created for each synchronization
in that directory, using the format
  SyncEvolution-<server>-<yyyy>-<mm>-<dd>-<hh>-<mm>[-<seq>]
with the various fields filled in with the time when the
synchronization started. The sequence suffix will only be
used when necessary to make the name unique. By default,
SyncEvolution will never delete any data in that log
directory unless explicitly asked to keep only a limited
number of previous log directories.

This is done by setting the "maxlogdirs" limit to something
different than the empty sring or 0: if a limit is set,
then SyncEvolution will only keep that many log directories
and start removing the oldest ones when it reaches the limit.
This cleanup is only done after a successful synchronization
and is limited to directories starting with the
  SyncEvolution-<server>
prefix, so it is safe to put other files or directories
into the configured log directory.

If that option is not set, then the directory will be
created as
  $TMPDIR/SyncEvolution-<username>-<server>
with access allowed for the user only. Files from a
previous synchronization will be overwritten. This is
a lot less useful because the data will probably
be lost during the next reboot.


Configuration with Sync4j
-------------------------

A default Sync4j 2.3 installation already contains databases which
SyncEvolution can synchronize with Evolution address books and
calendars. They are adressed with
  uri = card
for contacts and
  uri = cal
for calendars in the source config.

WARNING: the current Sync4j 2.3 cannot parse calendar entries
as sent by SyncEvolution. Until these issues are resolved in the
server, one has to fall back to a setup where the server does not
have to convert tasks. This is similar to the configuration for
tasks described in the next paragraph.

Although Evolution tasks are sent and received as special calendar
events, the server cannot parse them. Therefore it is necessary to
store each task as files on the server without having the server parse
them. One can use the existing
  uri = snote
file source, but if that source is also used for real notes or
files, then one has to setup another FileSyncSource. Adding
a new "todo" sync source is best done via the graphical Sync4j admin
tool:
- connect to the server
- unfold the tree and select "Modules/pdi/FileSystem SyncSource"
- enter the following values:
  Source URI: todo
  Name: todo
  Type: text/x-todo
  Source Directory: <server install dir>/db/todo
  Supported types: text/x-todo
  Supported versions: text/x-todo
  Encoded: deselected
  MultiUser: deselected if all users access the same database,
             selected if every user gets his own database
- add the new sync source
- create the <server install dir>/db/todo directory

The drawback of the file based approach is that the server always
keeps tasks exactly as the clients send them: imagine that
SyncEvolution stores a task on the server and then a phone fetches
that task from the server, possibly throwing away attributes that it
does not support. When now the task is modified on the phone and sent
back to the server, only the attributes supported by the phone remain
on the server and thus after another synchronization with
SyncEvolution also in Evolution. For calendar entries and contacts the
Sync4j server avoids this kind of data loss by updating its copy of an
item instead of replacing it wholesale.


Exchanging Data
---------------

SyncEvolution transmits address book entries as vCard 2.1 or 3.0
depending on the type chosen in the configuration. Evolution uses
3.0 internally, so SyncEvolution converts between the two formats
as needed. Calendar items and tasks have to be sent and received
in iCalendar 2.0 format. Only the UTF-8 character set is supported.

How the server stores the items depends on its implementation and
configuration. In the default Sync4j server installation, vCards
and calendar items are converted into an internal format, but at
least for vCards it preserves most of the properties used by
Evolution.

To check which data is preserved, one can use this procedure:
1. synchronize the address book with the server
2. create an new address book in Evolution
3. add a configuration for that second address book and the
   same database on the SyncML server
4. synchronize again, this time using the other data source

The "etc/localhost_1" directory contains a configuration for a default
Sync4j installation on the local host and Evolution address book,
calendar and tasks all called "SyncEvolution test #1". The
"etc/localhost_2" is the pendant for a second client which
synchronizes against the same server and Evolution databases called
"SyncEvolution test #2". Both configurations can be copied directly to
".sync4j/evolution":
  mkdir -p ~/.sync4j/evolution
  cp -a etc/localhost* ~/.sync4j/evolution

Note that the second client pretends to be a "sc-pim-ppc" client to
avoid the need to reconfigure the default Sync4j installation. This
implies that you cannot use this predefined Evolution config if you
actually synchronize against a PocketPC client.

The same configuration also exists for scheduleworld, but beware
that username/password must be adapted in the spds/syncml/config.txt
files.

For them to work, also create the two address books/calendars/tasks
  SyncEvolution test #1
  SyncEvolution test #2
inside Evolution. SyncEvolution never creates databases
itself.

Steps 1 above then becomes an invocation of
  syncevolution localhost_1 addressbook_1
and step 4
  syncevolution localhost_2 addressbook_2

This copies all contacts into the server and from there into the new
address book. Now one can either compare the address books in
Evolution or do that automatically:
- save the complete address books: mark all entries, save as vCard
- invoke synccompare with two file names as arguments and it will
  normalize and compare them automatically

Normalizing is necessary because the order of cards and their
properties as well as other minor formatting aspects may be
different. The output comes from a "diff --side-by-side", but
is augmented by the script so that the context of each change
is always the complete item that was modified. Lines or items
following a ">" on the right side were added, those on the
left side followed by a "<" were removed, and those with
a "!" between text on the left and right side were modified.

The automatic unit testing (see HACKING) contains a "testItems"
test which verifies the copying of special entries.

Modifying either address book and synchronizing back and forth
can be used to verify that SyncEvolution works as expected. If
you do not trust SyncEvolution or the server, then it is prudent
to run these checks with a copy of the original address book.
Make a backup of the .evolution/addressbook directory.

Did I mention that you should make backups?


Conflict Resolution
-------------------

If two clients make changes to the same item, the first one to
sync will get the server to copy its changes. The second one
then runs into a conflict when it tries to push its own changes
into the server.

The SyncML server now has to decide how to proceed. If the server
decides to continue with its own copy and asks to overwrite the
locally modified copy (the default with Sync4j), SyncEvolution will
make a local copy first. This leads to duplicates which have to be
merged manually on the client side where the conflict
occurred. Currently there is no support for that inside SyncEvolution:
there is only an ERROR entry in the log and the summary will show
the duplicated items.

Merging items on the server is difficult because the SyncML protocol
does not specify which parts of a conflicting item were updated.
In general a server can only make more or less educated guesses
which might lead to data loss. It is better to avoid this situation
in the first place by synchronizing before making changes.


Tracking Changes inside Evolution
---------------------------------

The SyncML protocol requires that a client knows which items have been
added, modified and deleted since the last sync. This is supported by
the Evolution database server, albeit in a limited way:

the same function lists changes and also moves the so called "change
marker" forward. Therefore asking for changes twice in a row will only
list changes the first time and not report the same changes a second
time. SyncEvolution delays asking for changes as long as possible
and only does it when synchronization has really started. Then
if synchronization completed and items where added, modified or
deleted on behalf of the server, the change marker is moved forward.

If synchronization fails for some or all items, then SyncEvolution
cannot mark individual items for retransmission during the next
sync and forces the next sync to execute in slow mode.

The change marker that SyncEvolution uses is a string which is
composed as "SyncEvolution:<syncURL>/<name>" where <syncURL> comes
from the server config file and <name> from the source config
file. This implies that changes are tracked separately for each server
and server database that Evolution might be synchronized with.


Known Problems
--------------

ObjectWeb #<num> refers to the Funambol issue tracker at:
http://forge.objectweb.org/tracker/?group_id=96&atid=100096

- Evolution 2.0.4 and 2.4.2.1 still display the old content of a contact
  which was updated during a certain test (TestEvolution::testMerge).
  Exact reason unknown, needs to be investigated.

- refresh-server results in a Sync4j client library error at the end
  of the sync with the client library shipped with 0.2:

  TestEvolution.cpp:534:Assertion
  Test name: TestEvolution::testDeleteAll
  assertion failed
  - Expression: !res

  With the current library source that is avoided, but a refresh-server
  with empty local address book does not delete all items on the
  server as it should, leading to:

  TestEvolution.cpp:674:Assertion
  Test name: TestEvolution::testDeleteAll
  assertion failed
  - Expression: countItems( source ) == 0

  ObjectWeb #304806

- various vcard and special character related problems in the
  Sync4j server and client library:
  TestEvolution::testVCard fails the check that items
  are identical after copying them to the server and back

  Many of these issues have been fixed in the server already,
  but even in 2.3 plus patch some information is still lost
  (see "testing vcard conversion and copying" on the sync4j-user
   mailing list for details).
  Characters with special meaning in XML like & < > cannot be
  exchanged.

  ObjectWeb #304828, #304786, #304784, #304782

- error handling could be improved

  ObjectWeb #304805, #304562

- calendar conversion fails resp. loses information

  ObjectWeb #304946, #304993, #304994, #304995, #304996, #304997

- Removing a field and then synchronizing with the Sync4j server
  will not remove that field on the server. The server will preserve
  the old value instead.

  This is necessary because the server cannot distinguish between
  removed fields and fields that a client does not store. To avoid
  losing data when copying back items from a less capable client the
  server preserves missing fields, even in situation where the field
  was intentionally removed on the client.

  The workaround for this conceptual problem is to never clean
  (= remove) a field - better fill it with e.g. a single space.


Support
-------

If you would like to ask questions, please use the Sync4j
users mailing list. You can subscribe at
   http://sourceforge.net/mail/?group_id=30236
or mail
   sync4j-users@lists.sourceforge.net
directly. If you mail the list without subscribing, please ask
to get replies to you directly, as some people might reply
only to the list otherwise.

However, before asking a question make sure that it has not
been answered already (archives are linked to from the list
page) and is not covered in this document. There is no FAQ yet.

If you run into any kind of issue during synchronization with
SyncEvolution, please try to determine as good as you can
whether it is caused by SyncEvolution, the Sync4j client
library or the server that you talk to.

Issues in the server should be reported using the issue tracker
for it. For Sync4j, it is located at
   http://forge.objectweb.org/tracker/?group_id=96
There you can also report issues with the Sync4j client library.

Because binary releases of SyncEvolution will contain
that library and have to be updated, please also create another
issue in the SyncEvolution tracker linking to the objectweb
issue.

If unsure, just open a ticket in the SyncEvolution tracker
and it will be forwarded appropriately after an initial analysis.
That tracker is hosted at SourceForge and can also be used
for feature requests:
   http://sourceforge.net/tracker/?group_id=146288&atid=764733

Be sure to always include the following information:
- version of SyncEvolution
- version of Evolution
- server and its version
- Linux distribution
- client log if appropriate,
  server log if appropriate and available
- a description of what you do,
  what you expect to see,
  what you get instead


Compiling from Source
---------------------

To compile the code the 3.x version of the Sync4j C++ client library
is needed. A compatible snapshot of it is included in SyncEvolution
source packages and will be used automatically. Instructions for
working with CVS sources directly are contained in the HACKING
document.

Also needed are the Evolution and libcurl development files. On
Debian 3.1 (Sarge) you can install them with
   apt-get install libcurl3-dev evolution-data-server-dev
The code was tested with Evolution 2.0.4 and 2.4.2.1. It is unclear
which other versions it is compatible with.

The build system is the normal autotools system.  See INSTALL for
general instructions how to use that and "./configure --help" for
SyncEvolution specific options. For the convenience of those checking
out from CVS directly, the files generated with "autogen.sh" are also
available from CVS.


Author
------

Patrick Ohly
patrick.ohly@gmx.de
http://ohly.home.pages.de/