syncevolution/src/backends/webdav

Usage
=====

The "webdav" backend provides sync sources for CalDAV and
CardDAV. They can be selected with backend=CalDAV
resp. backend=CardDAV.

In contrast to other backends, these sources need additional
information about a peer:

* syncURL:
  Specifies URL of calendar or contacts collection. %u gets
  replaced with the username.

  A ?SyncEvolution=<keyword>,<keyword>,... parameter provides
  further control. Currently implemented keywords:
  UpdateHack = work around Google's REV CalDAV quirks
  AlarmHack = when storing a VEVENT without VALARM, store it
              multiple times, because otherwise Google Calendar
              adds a default alarm
  Google = enables all hacks needed for Google

  Specifying a syncURL is optional. If not given, then DNS SRV
  lookups based on the domain name in the username are used
  to find the right host. In theory, .well-known URIs are then
  used to find the right path on that host, but in practice none
  of the peers that were tested (Google and Yahoo) seemed to
  support that. Therefore the code contains a fallback for
  well-known Yahoo paths when .well-known URIs fail.

* username/password:
  credentials, if available use the email address (needed
  for auto discovery of CalDAV or CardDAV server)

They also use:
* loglevel:
  >= 3: basic logging of HTTP traffic
  >= 4: also logging of HTTP body
  >= 5: also SSL and WebDAV lock handling
  >= 6: detailed information about XML parsing
  >= 11: plaintext HTTP authentication

The recommended way of using the CalDAV and CardDAV backends is to
configure a context with the "target-config" peer inside it. Such a
context then can be used as part of a local sync with other
sources. See the Google section below for examples.


Multiple calendars/address books
================================

This is not yet implemented. The code always ends up picking the
default collection. Eventually the goal is to enable:
1. listing of available collections
2. selecting them via the "database" property

Only the second point is currently implemented.

The "database" property now can hold the final URL of the collection
(aka database) on the WebDAV server which is used for the
source. Setting it skips the entire auto-discovery process, which
makes access quite a bit faster and more reliable.

Because most UIs won't know initially how to find these URLs (listing
them not supported by the core SyncEvolution) and/or won't have the
necessary user dialogs, the property is set automatically after a
successful sync. This avoids the accidental switching between
databases when the user adds or removes databases on the server (which
can lead to different results of the auto discovery).

There's still no guarantee that the database picked by default is the
"right" one. That can only be solved in the UI because servers
typically don't have a "default" or "personal" flag for their
collections.


Concurrency
===========

The original plan was to lock the server's collection while
manipulating it during a sync. But Google Calendar does not support
locks, so that was not pursued further. An alternative would be to run
a sync without locking and detect concurrent modifications, but that
is not implemented yet.

Therefore it is possible (although not likely) that changes get lost
when some other client changes an item after SyncEvolution started a
sync and before that same item gets modified by SyncEvolution as part
of that sync.

Change tracking itself copes with changes made while a sync
runs. They'll be synchronized as part of the next sync.


Google
======

Google supports CalDAV access to its calendars. Several quirks were
encountered, which SyncEvolution tries to work around as good as
possible.

Use syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=Google
to enable these workarounds.

# create context for accessing Google CalDAV server,
# not visible in sync UI (consumerReady = 0):
syncevolution --configure \
              --template SyncEvolution \
              backend=CalDAV \
              syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=Google \
              username=<your account> \
              password=<your password> \
              consumerReady=0 \
              target-config@google calendar

# list events in the server
syncevolution --print-items target-config@google calendar

# show the content of these events on stdout
syncevolution --export - target-config@google calendar

# set up synchronization, using "google-calendar" as peer name
# because "google" is typically used for the SyncML-based
# contact sync, which uses a different syncURL;
# username/password can be left empty to use the credentials
# configured above
syncevolution --configure \
              --template "SyncEvolution Client" \
              syncURL=local://@google \
              consumerReady=1 \
              username= \
              password= \
              google-calendar calendar

# run a sync for the first time (allow slow sync)
syncevolution --sync slow google-calendar

# normal sync, also possible via sync UI
syncevolution google-calendar


Yahoo
=====

Yahoo supports both CalDAV and CardDAV. CalDAV is available at this
time (beginning 2011) if and only if the user has switched to the
"Yahoo! Calendar Beta". It is stable enough to be useful.

CardDAV works in read/write mode, but has severe limitations:
* Contacts sent by Yahoo encode special characters with
  HTML/XML entities, in one case even multiple times
  (\ -> &#92; -> &amp;#92;). At the moment SyncEvolution unconditionally
  works around that by replacing these entities, which is wrong for
  servers working correctly.
* Contacts are sent with UTF-8 encoding by Yahoo, but uploading such
  contacts leads to non-ASCII characters being replaced with a question
  mark on the server.

Both of these issues can be reproduced with iOS 4 when configuring
Yahoo as generic CardDAV server.

Yahoo uses different hosts for CalDAV and CardDAV. Leave the syncURL
empty and use the @yahoo email address as username to let the backend
do the host lookup.

# configure CalDAV and CardDAV with Yahoo,
# with host lookup via the domain in the username
syncevolution --configure \
              --template SyncEvolution \
              calendar/backend=CalDAV \
              addressbook/backend=CardDAV \
              syncURL= \
              username=<your account@yahoo.com or some other yahoo domain> \
              password=<your password> \
              consumerReady=0 \
              target-config@yahoo addressbook calendar

# list items, use "--export -" to see content
syncevolution --print-items target-config@yahoo calendar
syncevolution --print-items target-config@yahoo addressbook

# configure synchronization, with addressbook not enabled
syncevolution --configure \
              --template "SyncEvolution Client" \
              addressbook/sync=disabled \
              syncURL=local://@yahoo \
              username= \
              password= \
              yahoo addressbook calendar 


Debugging/Troubleshooting
=========================

Run the commands above with "loglevel=4" or higher. Set
SYNCEVOLUTION_DEBUG=1 to see the full neon output directly (instead of
having it filtered and, if present, written into a logfile) and run
the operation locally with --daemon=no (instead of inside the
syncevo-dbus-server).

Example:
SYNCEVOLUTION_DEBUG=1 syncevolution --daemon=no loglevel=4 --print-items @google calendar

Each sync produces two log files, one for the main config
("google-calendar"), one for the target config
("target-config@google"). It is possible to choose different log levels:

syncevolution --run \
              loglevel=1 \
              loglevel@google=11 \
              google-calendar

Occassionally Google Calendar gets confused and reports VEVENTs in
response to a REPORT which then are not returned for a
GET. Symptoms of that are:
  [ERROR @google] @google/calendar: event not found
SyncEvolution tries hard to avoid such situations, but some calendars
already seem to be in such a state to begin with. Deleting offending events
via the web interface helps in such cases.

When running under debugger, then beware that the WebDAV part runs
inside a forked process. Use gdb's "set follow-fork-mode child" to
enter the child. Make sure that you don't accidentally follow
execution into one of the external shell commands:

$ gdb ./syncevolution
(gdb) set follow-fork-mode child
(gdb) b SyncEvo::LocalTransportAgent::run
Breakpoint 1 at 0x846755: file /home/pohly/syncevolution/syncevolution/src/syncevo/LocalTransportAgent.cpp, line 159.
(gdb) run --daemon=no --run google-calendar
Starting program: /home/pohly/work/syncevolution/src/syncevolution --daemon=no --run google-calendar
[Thread debugging using libthread_db enabled]
[INFO] @default/addressbook: inactive
[INFO] @default/calendar+todo: inactive
[INFO] @default/memo: inactive
[INFO] @default/todo: inactive
[New process 4655]
[Thread debugging using libthread_db enabled]
[Switching to Thread 0x7ffff7fc27e0 (LWP 4655)]

Breakpoint 1, SyncEvo::LocalTransportAgent::run (this=0xc2a310)
    at /home/pohly/syncevolution/syncevolution/src/syncevo/LocalTransportAgent.cpp:159
159	    const char *delay = getenv("SYNCEVOLUTION_LOCAL_CHILD_DELAY");
(gdb) set follow-fork-mode parent
(gdb) b SyncEvo::CalDAVSource::readSubItem
Breakpoint 2 at 0x69d47c: file /home/pohly/syncevolution/syncevolution/src/backends/webdav/CalDAVSource.cpp, line 382.
(gdb) c
Continuing.
[...]
Breakpoint 2, SyncEvo::CalDAVSource::readSubItem (this=0xec7430, davLUID=..., subid=..., item=...)
    at /home/pohly/syncevolution/syncevolution/src/backends/webdav/CalDAVSource.cpp:382
382	    Event &event = loadItem(davLUID);
(gdb) ...

Alternatively, setting SYNCEVOLUTION_LOCAL_CHILD_DELAY=<seconds> in
the environment and the attaching to the second "syncevolution"
process also works.


Implementation
==============

The neon library is used for HTTP/WebDAV. CalDAV and CardDAV queries
can be done with it, although not supported directly. SyncEvolution's
NeonCXX provides a C++ API on top of neon, with exceptions thrown for
errors, Boost function pointers for callbacks and handles that track
the underlying pointers.

WebDAVSource contains the main WebDAV logic. It is customized via
virtual methods by derived CalDAVSource and CardDAVSource whenever
necessary.

These sources use the resource name/ETag pair to detect changes. So
for CardDAV retrieving the meta information is sufficient to detect
changes.

With CalDAV the situation is a bit more complex. A CalDAV item
contains multiple VEVENTs, whereas SyncEvolution works on one VEVENT
at a time during synchronization. The MapSyncSource uses a
CalDAVSource and exposes calendar data with one VEVENT per
item. MapSyncSource is generic enough to be used by sources which
implement the SubSyncSource interface.

The downside is that the number of VEVENTs inside each CalDAV item and
their UID/RECURRENCE-ID properties need to be known before a sync. In
the default mode of SyncEvolution (with automatic data backups before
and after a sync), this additional information is collected as part of
creating the backup, so no additional overhead is incurred, but when
disabling backups, a sync still needs to download the calendar data
first. In theory, CalDAV can report just the necessary
UID/RECURRENCE-ID properties and SyncEvolution uses that, but in
practice, servers return the full data.

DNS SRV lookup is implemented using the syncevo-webdav-lookup utility
script which calls "host", "adnshost" or "nslookup", depending on what
is available. This approach avoids a hard dependency on a specfic
resolver library and (for some of these) writing quite a bit of
additional code.


Testing
=======

Testing via client-test is possible for the Client::Source tests. They
cover importing/exporting of items and change tracking.

client-test needs to be told in the CLIENT_TEST_WEBDAV env variable
which CalDAV or CardDAV servers are available and the sync properties
for them. It then creates a suitable
"target-config@client-test-<server>" config with "caldav" and/or
"carddav" sources. Sync properties can be specified; if not given, the
relevant ones must be set inside that config already.

In addition to sync properties, the following properties can be used
to influence the test setup itself:
  testconfig = name of a test config known to ClientTest::getTestData(),
               default is "eds_contact" (for CardDAV) and "eds_event" (for CalDAV)
  testcases  = name of file holding data for testImport (eds_contact.vcf and eds_event.ics)

The format of CLIENT_TEST_WEBDAV is:
  <server> [caldav] [carddav] [<testconfig>] <prop>=<val> ...; ...
  <prop> = sync property or [caldav/carddav/<testconfig>]/testcases

Example:

env PATH=backends/webdav/:$PATH \
    "CLIENT_TEST_WEBDAV=
     yahoo caldav carddav username=<my @yahoo.com email> password=<password>
           caldav/testcases=testcases/yahoo_event.ics
           carddav/testcases=testcases/yahoo_contact.vcf ;
     google caldav username=<email> password=<password>
            testcases=testcases/google_event.ics
            syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=Google" \
    ./client-test \
    Client::Source::yahoo_caldav::testOpen \
    Client::Source::yahoo_carddav::testOpen \
    Client::Source::google_caldav::testOpen

Note that this assumes that the test is run inside the "src" directory
without installing it, so the PATH must include the backend dir which
contains syncevo-webdav-lookup.