Check whether configs already exist instead of overwriting them
each time. Allows local modifications of the properties which
are set (useful for debugging) and is more efficient (always
useful).
Interfacing with the Cmdline class turned out to be the easiest
way of implementing the functionality. It is a bit indirect, but
replicating the logic for setting up and updating the config
would have required more code.
Currently the config is updated each time the plugin runs. Should
change that so that an existing config is left unchanged.
Error handling is incomplete.
When parsing data received from the server, we may end up with
X-LIC-ERROR error messages from libical in the internal
representation. This will get added to the server when combining
multiple items, which is undesirable (waste of bandwith, breaks
comparison).
This patch solves this by filtering out such errors.
Google Calendar adds a VALARM when a VEVENT is created for the first
time on the server via CalDAV. This is not what we want, if the VEVENT
had no VALARM, that's how it should be stored.
As a workaround we detect this special case (new VEVENT without
VALARM) and send the original data again. Having to resend with higher
SEQUENCE and LAST-MODIFIED/DTSTAMP values makes this a bit complex and
slow, because we need to get the actual data from the server (cannot
guess what time stamps were assigned).
This leads to an interesting question: if the clock on the local side
is unsynchronized, its LAST-MODIFIED time stamps might end up being
lower than the values on the server, which prevents sending local
updates. There's currently no solution for this in the backend.
Starting a neon Session is slow, in particular when libproxy is
involved (seems to search for proxy servers, which involves a DNS
timeout when none are around). Another delay is opening the TCP
connection and determining which kind of autentication is needed.
For a normal program run there's not much that can be done to speed
this up, but in client-test it helps a lot to reuse an existing
Session instance between different tests. This patch enables that
by keeping one Session instance around and returning that if the new
one has the same parameters.
The Settings pointer is a boost::shared_ptr, but this patch avoids
using it unless the Session is in active use, just in case that a user
wants to merge the settings into a class with a different life time.
It was observed once (but could no longer be reproduced after reducing the
number of requests) that Google Calendar started to reject requests
with valid credentials with a 401 credential error. Presumably that
was a counter-measure for excessive number of requests.
This patch adds a retry mechanism for this particular case, with
exponentially increasing delays before retrying the same request until it
succeeds or one minute has passed. A credential error on the very first
request in a session still fails immediately. This is implemented as
part of the normal neon "get credentials" callback, so the logic
on top of neon is unchanged.
Both CardDAV and CalDAV allow collections inside the collection of
items which are to be synchronized. They may not contain other contact
resp. calendar data and simply can be ignored.
This patch is only relevant for CardDAV; CalDAV uses its own method
for finding calendar items which already ignores non-calendar items.
This patch moves some of the CalDAV-specific code out of WebDAVSource
into CalDAVSource and adds the corresponding CardDAV code in the new
CardDAVSource. It improves searching for the right collection.
A Yahoo Contacts specific problem is that vCard UID and resource name
have to match, with ".vcf" used as suffix in the resource name. The
new createResourceName() and setResourceName() virtual methods ensure
that. They are nops in CalDAV and use simple string manipulation for
vCard.
Another Yahoo Contacts odity is merging of contacts in PUT, without
telling the client. This can be detected by checking the expected
resource name with PROPFIND: the server then replies with the real
resource name. This check is done only if no href is returned in the
response to PUT, so it should only be triggered for Yahoo Contacts.
CardDAV support is still experimental. It was tested with Yahoo
Contacts, using https://carddav.address.yahoo.com/dav/%u as sync
URL. This is different from the CalDAV URL, which currently prevents
using the same @yahoo context for both calendar and contacts.
Working on CardDAV showed that resource name comparisons had issues
due to different encoding of special characters. Added
canonicalization of paths to fix this problem. This also includes
appending the trailing slash for collections
(Neon::URI::normalizePath()).
Note that Buteo only serializes sync sessions based on their type, so
as soon as we allow CardDAV in addition to CalDAV, it might happen
that SyncEvolution is invokved multiple times in parallel => need to
remove global variables first.
If proxy settings are turned off in the config, the default system
proxy settings are used. This depends on ne_session_system_proxy(),
introduced in neon 0.29. The version is checked at configure time. A
fallback if only an older version is found is intentionally not
present yet, the feature is meant to work in all cases.
The username might contain special characters. It must be escaped
before adding it to the URL. Found when the code was used with an
the default "Your SyncML Username" username string.
Yahoo works without any of the Google specific hacks. Because these
have side effects (performance, change data), they shouldn't be used
when talking to other CalDAV servers.
This patch adds a "SyncEvolution" URL parameter with a comma separated
list of flags:
"UpdateHack" = increase SEQUENCE and LAST-MODIFIED
"ChildHack" = replace RECURRENCE-ID in stand-alone child events
This needs to be appended to the syncURL, like this:
syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=UpdateHack,ChildHack
The reason for using "SyncEvolution" here is that other parameters
might one day be passed through to the underlying server; at the
moment, anything other than "SyncEvolution" causes an error.
They need to specified explicitly. Hard-coding a check for www.google.com
did not feel right.
Content-Type must be set, otherwise Yahoo CalDAV rejects the
items with "Bad Request".
The resource name must match the UID + .ics. It is okay to
escape special characters like @ in the PUT, even though the
server doesn't do the same in PROPFIND.
For change tracking we need a canonical form for DAV luids. Always
unescaping the relative path component is safe. For absolute paths, we
use the value as suggested by the server. There's no risk of mismatch
here, because we never create such a luid ourself.
This patch implements a dynamic search for the "default" calendar
of a user, starting with the user's principal URL:
https://caldav.calendar.yahoo.com/dav/<user>/
https://www.google.com/calendar/dav/<user>/user/
Stops when:
- current path is calendar collection (= contains VEVENTs)
Gives up:
- when running in circles
- nothing else to try out
- tried 10 times
Follows:
- CalDAV calendar-home-set
- collections
This implies that a specific calendar can be selected by using its URL
directly. The search for "default" calendar is a bit random, because
it depends on the naming of calendars on the server: calendars with
lexicographically lower name are tested (and thus found) first.
Without any kind odf <filter>, Yahoo rejected the request with 400
"Bad Request". Now pass a filter for VEVENT without actually
filtering on any property.
Yahoo requests a client certificate as part of setting up the SSL
connection. Without such a certificate, gnutls aborts setting up
the connection.
As a workaround this patch reads "client.p12" in the current
directory and uses that as certificate. Cannot be the final
solution - replace by dynamically generated certificate or use
ne_ssl_provide_clicert(), with empty list of certificates?
When a child is added without the corresponding parent, Google gets
confused (child is added, but cannot be retrieved). This patch turns a
child event into a normal event by replacing RECURRENCE-ID with
X-SYNCEVOLUTION-RECURRENCE-ID. The inverse operation is applied when
importing data back from Google.
Google requires that all items sent to it have a higher SEQUENCE
number than the copy on the server. Calendar applications are allowed
to make changes to an item even when they are not the organizer, in
which case they are *not* meant to increase the SEQUENCE.
This patch works around the issue by increasing the SEQUENCE number
artifically before sending to the CalDAV server (regardless whether
it is Google or something else).
It does that by keeping track of UID and SEQUENCE number in the
cache. The UID is used to find the resource before sending in case that
the caller didn't know, the SEQUENCE to know what the outgoing item
must have.
The sequence number is shared among all sub items. That is an
intentional simplification of the implementation which may or may not
be necessary to work around the issue. If we have to touch the
sequence number, we might as well do it for all sub items.
This patch enhances error handling so that a more specific
TransportStatusException with the right SyncMLStatus inside is
thrown. This is necessary to report back authentication errors.
libical needs to be checked for explicitly, instead of assuming
that it is pulled in via the ecal backend.
The neon package check can be deferred until the backend really
is needed.
Relying on the MapSyncSource and our own caching hides problems in
both. A backup mechanism taking data directly from the CalDAV server
is more reliable.
This patch implements that by running a REPORT and dumping the
returned data. It stores each VCALENDAR as one file, which may contain
multiple VEVENTs with the same UID. synccompare handles that.
The patch refactors the REPORT issuing because the new backupData()
and listAllSubItems() can share most of that code.
Restore is not implemented yet. The start of a sync could be optimized
by adding all backed up items to a revision list and reusing it in
listAllSubItems(), like SyncSourceRevisions does.
This implements logging like in EvolutionCalendarSource: SUMMARY
plus optional LOCATION. If possible, the item is taken from the cache.
A better solution would move the common "work with icalcomponent"
code into a shared part of libsyncevolution... left for later.
Implemented with a single REPORT request, instead of one query for
resource ID/etag pairs followed by one GET per item.
In theory also less data needs to be transmitted, because we only ask
for the minimum number of iCalendar 2.0 properties required at that
point. In practice, Google delivers full iCalendar 2.0 items. Because
we cannot rely on that, we do not populate the cache with these and
have to fetch those anew which are later involved in operations.
Overall this should still be more efficient.
When a single sub item in a merged item is updated, we modify the
calendar and sent the complete item to the server. Google didn't
like that and rejected the update because of "409 - Can only store an
event with a newer DTSTAMP." Found in testLinkedItemsInsertBothUpdateParent.
The solution is to update SEQUENCE and DTSTAMP/LAST-MODIFIED to that of
the recently updated sub item, then Google accepts the update. Updating
just one or the other was not enough.
This change should work even if more than one item is updated, as long
as a) item changes are reported in the right order (oldest change
first) or b) all sub items get updated.
a) is not guaranteed, b) might be. To be tested...
This patch implements the SubSyncSource interface in CalDAVSource,
then instantiates that class as backend for a MapSyncSource which
bridges between the core engine (one VEVENT per item) and CalDAV (all
VEVENTs with same UID in the same item).
The tests must run with a modified ical20.ics.google.tem which
does not contain RECURRENCE-ID, because that is not yet supported.
Also, our complex 1234567890!@#$%^&*()<>@dummy must be avoided
because Google has an issue with it (doesn't find resulting resource):
CLIENT_TEST_SERVER=google CLIENT_TEST_SIMPLE_UID=1 ./client-test Client::Source::caldav_ical20::testImport
This patch fixes some (but not all) issues found in testImport:
- readItem() did not clear the item string before appending
the new item.
- Detection of "item existed already" cannot use the alternative name
returned by the server, because Google does that also for items
which are really new.
- The test registered itself as "ical20", which conflicts with the EDS
backend. Now use "caldav_ical20".
This replaces the WEBDAV_* env variables with access to the
normal per-peer properties, passed into the source via the new
SyncSourceParams context member.
When used on the command line, @foobar context configs are not
enough to use the WebDAV sources. A properly configured peer
context (foo@bar) is needed. There is no explicit check for this,
the resulting error is about not being able to access the empty
URL `' (Neon uses these quotation marks).
Mapping removeItem() to DELETE is straight-forward. Except that CalDAV
will require additional support for removing individual VEVENTs from
a resource that contains more than one VEVENT...
This works for updating and adding an item. Storing a new item under a
different resouce URI is also handled, which can happen with CalDAV
when there is already an item with the same UID.
It would be nice if the server returned a ETag in response to a PUT,
but Google Calendar doesn't do that. The header is checked neverthless
(untested!), with an explicit PROPGET as fallback.
This patch also clarifies LUID (relative if possible, absolute if not)
and revision string handling (without W/ and quotation marks). We
don't care about weak ETags at the moment, Google doesn't use
them. Even if a server did, what would we do with the information that
the item wasn't stored verbatim?
The current implementation is not yet good enough for CalDAV:
- only one VEVENT per UID supported: must merge and split
related VEVENTs into one CalDAV calendar resource
- adding or updating VEVENT without UID probably fails:
must insert new or old UID
Listing items in WebDAV consists of getting the ETag for each URI
directly underneath the collection. Checking for "empty source"
is done by reading all pairs and then checking for non-empty map.
It would be nice to abort the processing once an error is encountered
or once "not empty" has been observed, but Neon does not support that.
Probably for a good reason, because it would render the TCP connection
useless.
