Starting a neon Session is slow, in particular when libproxy is
involved (seems to search for proxy servers, which involves a DNS
timeout when none are around). Another delay is opening the TCP
connection and determining which kind of autentication is needed.
For a normal program run there's not much that can be done to speed
this up, but in client-test it helps a lot to reuse an existing
Session instance between different tests. This patch enables that
by keeping one Session instance around and returning that if the new
one has the same parameters.
The Settings pointer is a boost::shared_ptr, but this patch avoids
using it unless the Session is in active use, just in case that a user
wants to merge the settings into a class with a different life time.
It was observed once (but could no longer be reproduced after reducing the
number of requests) that Google Calendar started to reject requests
with valid credentials with a 401 credential error. Presumably that
was a counter-measure for excessive number of requests.
This patch adds a retry mechanism for this particular case, with
exponentially increasing delays before retrying the same request until it
succeeds or one minute has passed. A credential error on the very first
request in a session still fails immediately. This is implemented as
part of the normal neon "get credentials" callback, so the logic
on top of neon is unchanged.
Both CardDAV and CalDAV allow collections inside the collection of
items which are to be synchronized. They may not contain other contact
resp. calendar data and simply can be ignored.
This patch is only relevant for CardDAV; CalDAV uses its own method
for finding calendar items which already ignores non-calendar items.
This patch moves some of the CalDAV-specific code out of WebDAVSource
into CalDAVSource and adds the corresponding CardDAV code in the new
CardDAVSource. It improves searching for the right collection.
A Yahoo Contacts specific problem is that vCard UID and resource name
have to match, with ".vcf" used as suffix in the resource name. The
new createResourceName() and setResourceName() virtual methods ensure
that. They are nops in CalDAV and use simple string manipulation for
vCard.
Another Yahoo Contacts odity is merging of contacts in PUT, without
telling the client. This can be detected by checking the expected
resource name with PROPFIND: the server then replies with the real
resource name. This check is done only if no href is returned in the
response to PUT, so it should only be triggered for Yahoo Contacts.
CardDAV support is still experimental. It was tested with Yahoo
Contacts, using https://carddav.address.yahoo.com/dav/%u as sync
URL. This is different from the CalDAV URL, which currently prevents
using the same @yahoo context for both calendar and contacts.
Working on CardDAV showed that resource name comparisons had issues
due to different encoding of special characters. Added
canonicalization of paths to fix this problem. This also includes
appending the trailing slash for collections
(Neon::URI::normalizePath()).
Note that Buteo only serializes sync sessions based on their type, so
as soon as we allow CardDAV in addition to CalDAV, it might happen
that SyncEvolution is invokved multiple times in parallel => need to
remove global variables first.
If proxy settings are turned off in the config, the default system
proxy settings are used. This depends on ne_session_system_proxy(),
introduced in neon 0.29. The version is checked at configure time. A
fallback if only an older version is found is intentionally not
present yet, the feature is meant to work in all cases.
The username might contain special characters. It must be escaped
before adding it to the URL. Found when the code was used with an
the default "Your SyncML Username" username string.
Yahoo works without any of the Google specific hacks. Because these
have side effects (performance, change data), they shouldn't be used
when talking to other CalDAV servers.
This patch adds a "SyncEvolution" URL parameter with a comma separated
list of flags:
"UpdateHack" = increase SEQUENCE and LAST-MODIFIED
"ChildHack" = replace RECURRENCE-ID in stand-alone child events
This needs to be appended to the syncURL, like this:
syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=UpdateHack,ChildHack
The reason for using "SyncEvolution" here is that other parameters
might one day be passed through to the underlying server; at the
moment, anything other than "SyncEvolution" causes an error.
They need to specified explicitly. Hard-coding a check for www.google.com
did not feel right.
Content-Type must be set, otherwise Yahoo CalDAV rejects the
items with "Bad Request".
The resource name must match the UID + .ics. It is okay to
escape special characters like @ in the PUT, even though the
server doesn't do the same in PROPFIND.
For change tracking we need a canonical form for DAV luids. Always
unescaping the relative path component is safe. For absolute paths, we
use the value as suggested by the server. There's no risk of mismatch
here, because we never create such a luid ourself.
This patch implements a dynamic search for the "default" calendar
of a user, starting with the user's principal URL:
https://caldav.calendar.yahoo.com/dav/<user>/
https://www.google.com/calendar/dav/<user>/user/
Stops when:
- current path is calendar collection (= contains VEVENTs)
Gives up:
- when running in circles
- nothing else to try out
- tried 10 times
Follows:
- CalDAV calendar-home-set
- collections
This implies that a specific calendar can be selected by using its URL
directly. The search for "default" calendar is a bit random, because
it depends on the naming of calendars on the server: calendars with
lexicographically lower name are tested (and thus found) first.
Without any kind odf <filter>, Yahoo rejected the request with 400
"Bad Request". Now pass a filter for VEVENT without actually
filtering on any property.
Yahoo requests a client certificate as part of setting up the SSL
connection. Without such a certificate, gnutls aborts setting up
the connection.
As a workaround this patch reads "client.p12" in the current
directory and uses that as certificate. Cannot be the final
solution - replace by dynamically generated certificate or use
ne_ssl_provide_clicert(), with empty list of certificates?
When a child is added without the corresponding parent, Google gets
confused (child is added, but cannot be retrieved). This patch turns a
child event into a normal event by replacing RECURRENCE-ID with
X-SYNCEVOLUTION-RECURRENCE-ID. The inverse operation is applied when
importing data back from Google.
Google requires that all items sent to it have a higher SEQUENCE
number than the copy on the server. Calendar applications are allowed
to make changes to an item even when they are not the organizer, in
which case they are *not* meant to increase the SEQUENCE.
This patch works around the issue by increasing the SEQUENCE number
artifically before sending to the CalDAV server (regardless whether
it is Google or something else).
It does that by keeping track of UID and SEQUENCE number in the
cache. The UID is used to find the resource before sending in case that
the caller didn't know, the SEQUENCE to know what the outgoing item
must have.
The sequence number is shared among all sub items. That is an
intentional simplification of the implementation which may or may not
be necessary to work around the issue. If we have to touch the
sequence number, we might as well do it for all sub items.
This patch enhances error handling so that a more specific
TransportStatusException with the right SyncMLStatus inside is
thrown. This is necessary to report back authentication errors.
libical needs to be checked for explicitly, instead of assuming
that it is pulled in via the ecal backend.
The neon package check can be deferred until the backend really
is needed.
Relying on the MapSyncSource and our own caching hides problems in
both. A backup mechanism taking data directly from the CalDAV server
is more reliable.
This patch implements that by running a REPORT and dumping the
returned data. It stores each VCALENDAR as one file, which may contain
multiple VEVENTs with the same UID. synccompare handles that.
The patch refactors the REPORT issuing because the new backupData()
and listAllSubItems() can share most of that code.
Restore is not implemented yet. The start of a sync could be optimized
by adding all backed up items to a revision list and reusing it in
listAllSubItems(), like SyncSourceRevisions does.
This implements logging like in EvolutionCalendarSource: SUMMARY
plus optional LOCATION. If possible, the item is taken from the cache.
A better solution would move the common "work with icalcomponent"
code into a shared part of libsyncevolution... left for later.
Implemented with a single REPORT request, instead of one query for
resource ID/etag pairs followed by one GET per item.
In theory also less data needs to be transmitted, because we only ask
for the minimum number of iCalendar 2.0 properties required at that
point. In practice, Google delivers full iCalendar 2.0 items. Because
we cannot rely on that, we do not populate the cache with these and
have to fetch those anew which are later involved in operations.
Overall this should still be more efficient.
When a single sub item in a merged item is updated, we modify the
calendar and sent the complete item to the server. Google didn't
like that and rejected the update because of "409 - Can only store an
event with a newer DTSTAMP." Found in testLinkedItemsInsertBothUpdateParent.
The solution is to update SEQUENCE and DTSTAMP/LAST-MODIFIED to that of
the recently updated sub item, then Google accepts the update. Updating
just one or the other was not enough.
This change should work even if more than one item is updated, as long
as a) item changes are reported in the right order (oldest change
first) or b) all sub items get updated.
a) is not guaranteed, b) might be. To be tested...
This patch implements the SubSyncSource interface in CalDAVSource,
then instantiates that class as backend for a MapSyncSource which
bridges between the core engine (one VEVENT per item) and CalDAV (all
VEVENTs with same UID in the same item).
The tests must run with a modified ical20.ics.google.tem which
does not contain RECURRENCE-ID, because that is not yet supported.
Also, our complex 1234567890!@#$%^&*()<>@dummy must be avoided
because Google has an issue with it (doesn't find resulting resource):
CLIENT_TEST_SERVER=google CLIENT_TEST_SIMPLE_UID=1 ./client-test Client::Source::caldav_ical20::testImport
This patch fixes some (but not all) issues found in testImport:
- readItem() did not clear the item string before appending
the new item.
- Detection of "item existed already" cannot use the alternative name
returned by the server, because Google does that also for items
which are really new.
- The test registered itself as "ical20", which conflicts with the EDS
backend. Now use "caldav_ical20".
This replaces the WEBDAV_* env variables with access to the
normal per-peer properties, passed into the source via the new
SyncSourceParams context member.
When used on the command line, @foobar context configs are not
enough to use the WebDAV sources. A properly configured peer
context (foo@bar) is needed. There is no explicit check for this,
the resulting error is about not being able to access the empty
URL `' (Neon uses these quotation marks).
Mapping removeItem() to DELETE is straight-forward. Except that CalDAV
will require additional support for removing individual VEVENTs from
a resource that contains more than one VEVENT...
This works for updating and adding an item. Storing a new item under a
different resouce URI is also handled, which can happen with CalDAV
when there is already an item with the same UID.
It would be nice if the server returned a ETag in response to a PUT,
but Google Calendar doesn't do that. The header is checked neverthless
(untested!), with an explicit PROPGET as fallback.
This patch also clarifies LUID (relative if possible, absolute if not)
and revision string handling (without W/ and quotation marks). We
don't care about weak ETags at the moment, Google doesn't use
them. Even if a server did, what would we do with the information that
the item wasn't stored verbatim?
The current implementation is not yet good enough for CalDAV:
- only one VEVENT per UID supported: must merge and split
related VEVENTs into one CalDAV calendar resource
- adding or updating VEVENT without UID probably fails:
must insert new or old UID
Listing items in WebDAV consists of getting the ETag for each URI
directly underneath the collection. Checking for "empty source"
is done by reading all pairs and then checking for non-empty map.
It would be nice to abort the processing once an error is encountered
or once "not empty" has been observed, but Neon does not support that.
Probably for a good reason, because it would render the TCP connection
useless.
This wraps the "simple" Neon propfind method which handles plain
strings as values. For more complex values, the value is an XML
fragment which should better be parsed using the "complex" Neon
propfind method with XML handlers.
The callbacks are defined by boost::function, which allows attaching
additional parameters to each callback. Exceptions must not cross Neon
library calls.
The Neon::Session wraps most of the relevant calls. It is
parameterized by Neon::Settings. It is uncertain where all of these
are meant to come from, because there is no peer configuration in many
cases. Perhaps we can enforce that a WebDAV source may only be created
in a context which has one and exactly one peer config?
The current intermediate solution in WebDAVSourceRegister.cpp grabs
all settings from WEBDAV_* env variables.
Disabling SSL verification and Neon debug logging are implemented.
Opening the source runs a few checks on the URL. Disabling SSL
certificate checking turned out to be necessary, probably because of
the known issue of gnutls not trusting the weak Google certificate
chain.
This patch maps QContactDetails which have no mapping to vCard to
X-SYNCEVO-QTCONTACTS. The value of that extension property is a dump
of the complete QContactDetail. Reading it again later can restore
that detail.
The goal is to make this additional information available to the
Synthesis engine such that it can preserve the additional information.
Some peers might even be able to store them.
One of the features of the Synthesis engine is that it can preserve
properties locally which are not supported by the
peer. "testExtensions" covers that, by updating all items on the
server (via client B) and reimporting them as updates into client A.
The data comparison then is done without the usual "allow server to
drop data" simplifications in synccompare.
This test is not enabled yet. "config.update" needs to be set for it,
but doing so then trips over other changes introduced by servers, like
for example re-encoding photos. Needs some more thoughts and testing...
The QVersitContactHandlerFactory::ProfileBackup profile enables a
vCard flavor which contains all information stored in a QtContact. It
might make sense to use that format for raw access (--import/export).
Not currently enabled though because the format is pretty verbose.
clang 2.8 compiles SyncEvolution + Synthesis faster than g++ 4.4.5
(3:40min instead of 4:10min on my laptop) and produces more useful
error reports. This patch fixes the code so that it compiles cleanly
with clang when using "-Wall -Werror -Wno-unknown-pragmas". Note that
clang 2.6 (Debian Squeeze) goes into an infinite recursion compiling
code using gdbus-cxx-bridge.h and dies eventually with a stack
overflow - can't be used.
Changes necessary for clang:
- eptr pointer referencing ambiguous, use *x.get() instead
- boost::intrusive_ptr* must be defined before code using it
- two-phase template checking requires explicitly specifying
members in base classes
- name clashes with plain C structs (DBusServer, DBusWatch) are
an error and need to be avoided (done with namespaces GDBusCXX and
SyncEvo)
- floats cannot be inline constants
- unused methods in local classes are warned about (left() in SyncML.cpp)
A lot of the code was roled out explicitly. This patch replaces
that with an approach similar to iostreams: remember context, then
work on one parameter at a time.
Because the decision what to do with a certain method parameter
depends on the type of the parameter, that type must be used to pick
between Get/Set classes which skip resp. work on the parameter.
