This patch moves the files for the glib-based SyncEvolution client library
into src/dbus/glib. This will make it possible to add src/dbus/qt and
enable/disable various D-Bus related components in a sane way.wq
The GTK sync-ui does CheckSource() calls on all configured sources,
even when not directly opening a config in the UI. This was
problematic for a CalDAV/CardDAV source because it ended up talking to
the server in WebDAVSource::open().
This violates the principle that open() should be a fast, local
operation. Reorganized the code so that the expensive code is done as
part of WebDAVSource::beginSync() -> WebDAVSource::contactServer()
(for CardDAV) resp. MapSyncSource::beginSync() ->
SubSyncSource::begin() -> WebDAVSource::contactServer() (for CalDAV).
The main usage is for derived classes which cannot do all of their
initialization in open() because it is too heavy. The implementation
of the D-Bus CheckSource() call expects open() to be fast and local.
The more expensive initialization needs to be done in beginSync(),
which requires calling TrackingSyncSource::beginSync() when
done.
This could be caused by a race condition (item already deleted by some
other client). In practice it was seen with a confused Google Calendar
server: event listed by REPORT, but both GET and DELETE fail for
it. Not much that can be done about that event via CalDAV then, so let's
ignore it.
Copied eds_contact.vcf to yahoo_contact.vcf in preparation
for adapting it to Yahoo. Use with
"CLIENT_TEST_WEBDAV=yahoo carddav testcases=testcases/yahoo_contact.vcf"
Currently identical to eds_event.ics. Use this with
"CLIENT_TEST_WEBDAV=google caldav testcases=testcases/google_event.ics"
Will be modified in further commits to let testImport pass without
differences.
409 "Can't delete a recurring event except on its organizer's calendar"
was seen once in response to a DELETE operation. This code tries
to deal with that failure by removing the RRULE first with a PUT, then
doing the DELETE again.
In further testing, this error was not encountered again, so this code
is not verified yet.
Client::Source tests with different servers may need different test
data. "testconfig" allows choosing something other than eds_event/contact,
"testcases" (more likely to be useful) the testcases/* file.
For servers which support both CalDAV and CardDAV, carddav/testcases
and caldav/testcases can be used.
Because the UI checked for the no-longer existing "type" property,
it didn't handle any sources at all. Replacing with the new
"backend" property name solves the problem.
Boost's split iterator on Debian Testing returns an empty
string, which wasn't seen before. Caused an invalid
complaint about an unknown empty parameter in the sync URL.
Checking for empty string now.
Failed to build because getPeerMimeType() now uses
a std::string as return type. getMimeType[Version]()
are no longer needed for sources derived from SyncSource.
The distinction between vcard21 and vcard30 became mute in the Evolution
backend a while ago. Both tests ended up using the vCard 3.0 Evolution
tests data and the default uri for each server. This patch removes
the vCard 2.1 special case.
It also renames the tests and test data to reflect that they always
were Evolution specific. The new naming convention, also applied
to file, QtContacts, KCalExtended, XMLRPC, Maemo and Akonadi backends, is
now <backend>_contact/event/task/memo, with eds/file/qt/kcal/maemo/kde
as backend names.
The reasoning is:
- results in unique string (in particular no overlap with
backend type names), easier to search for
- underscore already used before (in contrast to hyphen)
- no plural-s to keep the name shorter
The Akonadi backend should be using its own test data instead of
the Evolution ones.
Running "syncevolution" without parameters ran into a null
pointer access in a Boost shared pointer, leading to an abort
of the program with a Boost exception.
Fix this by checking whether the backend really has a context
node. Long term we need a solution which provides the necessary
information even when running without a sync config.
When adding Ubuntu Lucid to the nightly testing, it was observed
that on different platforms ical20.ics.nokia_7210c.tem.patch was
created differently, leading to a dirty source directory.
This patch avoids this by setting time stamps of the generated
file such that testcase2patch does not attempt to regenerate the
patch file unless the patched test cases are really getting edited
after creating them.
Also cleaned up logging of the operations a bit.
The recent change around WebDAVCredentialsOkay broke compilation
when WebDAV was disabled: the declaration and inclusion of
relevant header files also must be outside of the ifdef.
The "source-config" peer in the target context used to be required for
local sync. Now it is optional. The reasoning is:
- For WebDAV, copying username/password from the main config may
already be enough to find the server.
- For local file sources, no sync properties ared needed
at all.
In both cases, "source-config" is used if available. That can be useful
to set loglevel separately, for example.
For SyncEvolution<->SyncEvolution synchronization there should be
no need to set the uri property of each source explicitly, because
typically it is the same as the source name.
With this patch, the source name is used as fallback. Because some
code needs to check whether the URI is set, getURI() returns the
setting directly (as before) whereas getURINonEmpty() always returns a
non-empty result with the name as fallback.
Specifying an invalid template name continued with an empty template
if enough properties were specified. That's not correct and leads
to unexpected results:
- peerIsClient might have been part of the desired template.
- Set of sources to be created is defined by template.
Now an error is printed for this case.
Printing of templates was changed to use underscores instead of
spaces, but retrieving a template was not. Turned out that
it doesn't compare against the list that it builds up, but instead
does a matching against the fingerprint.
Now the fingerprint comparison treats spaces like underscores,
so that "SyncEvolution_Client" matches against the "SyncEvolution Client"
fingerprint. Better for backwards compatibility, too.
The device templates contained spaces ("Nokia N900", for example),
which was hard to use on the command line and in scripts. Now
an underscore is used instead of spaces.
This is a better fix for the webDAVCredentialsOkay property
registration: always do it, regardless whether the backend is enabled,
because then the config migration code will always know about it.
Also fixed automated testing to include that property.
Logging calls might end up calling functions which modify errno.
Seen once where logging reported a "pipe error" and the following
exception a "resource temporarily unavailable".
testOneWayFromClient was still failing in server mode because of a
similar issue as with slow sync as fallback for refresh-from-server:
the client can ask for a two-way sync and then withhold all of its own
changes to achieve the same effect.
Detect this based on the requested sync mode.
recv() may return -1 and EAGAIN/EWOULDBLOCK, according to the man
page. That was treated as a fatal error instead of retrying, which
was the original intention.
For writev() it is less clear whether EAGAIN/EWOULDBLOCK need to be
handled. Let's catch them anyway.
The following code crashed if m_length was read after executing
the release() calls:
realloc(buffer.m_message.release(), buffer.m_message->m_length)
That's because release() resets the m_message pointer. It seems that
clang orders execution like that whereas gcc doesn't.
Some of our extensions are optional, for example X-EVOLUTION-UI-SLOT
(because Nokia phones had problems with that). CalDAV/CardDAV servers
are meant to preserve all extensions, so let's send them everything we
have.
For calendar, this is not yet relevant in practice (rule doesn't do
anything).
Apple Calendar server + CardDAV really preserves the slot parameter.
Yahoo CardDAV does not, but at least accepts it.
When deleting items on the server due to "refresh-from-client" sync,
we want to have these items counted in the statistics, just as on
a client in a "refresh-from-server".
The code doing this had to be updated to distinguish between client and
server mode.
The main bug fix is for partial writes: the base address wasn't getting
updated, so the same data would have been sent multiple times. Not sure
whether this really occurred in practice.
fcntl() was called for random file descriptors due to a buffer
overflow. If a valid file descriptor was hit, the real user of it
might have gotten confused.
writev() was passed unitialized memory. Okay in this case (hole in struct),
but ugly in valgrind and easily fixed with memset().
This patch relies on libsynthesis reporting a stable sync mode
of each source in the PEV_ALERTED progress event. This was added
recently to libsynthesis in server mode.
This patch deals with a client that falls back to "delete data + slow
sync" as a way of doing a refresh-from-server (as Synthesis engine
itself does) at least in one case: server-alerted syncs know the originally
requested sync mode and assume that a slow sync is the intended mode,
which then gets translated back into refresh-from-server.
A session with such a sync initiated by a client is still recorded as
"slow sync".
An empty "database" property for a virtual source lead to a useless
error about instantiating a source with an empty name. Now the error
is reported for the virtual source and includes the "database" value.
A sanity check that the source really had exactly two sub-sources
was missing. Added.
The file backend refuses to create a source if backendFormat is unset.
This was reported as "<source>: backend 'file' not supported",
which was confusing because the backend was in fact supported.
Now the error message is "<source>: backend 'file' not supported or
not fully configured (format '<format>')".
Exceptions encountered in the child process were printed as [ERROR],
but not reported to the parent process and thus did not end up in the
final sync report.
Now they are, which makes finding the root cause of a failed local
sync a bit easier.
The error message "@<context>/<source>: source not configured" becames
confusing when <source> is empty. Now "missing URI for one of the
sources" is printed instead.
The command line instantiated sources with the name "list" (cut-and-paste
error?) instead of using the proper name. As a result, error messages
created by these sources didn't refer back to the configuration. Now
the real name is used.
The Akonadi source reused the EDS test names (vcard30, ical20, ...).
This has the effect that the configuration of these tests was
overwritten each time client-test was started. Each backend must
use its own names here.
For Akonadi, kde_events/tasks/memos were chosen. "calendar" was avoided
because some people include tasks in that category.
This re-adds the code from a3d7412972
which was temporarily removed in revert commit
4ff3777c70.
This time, the changes are done in such a way that other, non-KDE
backends are not affected. The properties fall into three categories:
1. The KDE extension maps to a field which has already an property mapping
that is understood by SyncML peers (example: X-messaging/xmpp-All KDE
property maps to X-JABBER). In these cases, the external
property is not used with KDE (empty property with rule="KDE").
A second property entry exclusively for KDE (rule="KDE") defines
the mapping to the KDE property.
2. The KDE extension has no corresponding mapping (X-KADDRESSBOOK-CRYPTO*,
X-messaging/irc-All for IRC chat handle). In these cases the KDE
property name is used also with SyncML peers.
3. An Evolution extension has no corresponding extension in KDE (X-EVOLUTION-VIDEO-URL).
Here the Evolution extension is passed to KDE.
Whether KDE extensions like X-messaging/irc-All should be exposed as
they are to other peers is uncertain. It might be more natural to follow
the X-AIM/JABBER/MSM/... pattern and introduce X-IRC. Or follow RFC 4770,
which defines the IMPP property... at the end of the day, this depends on
the peers which we want to send these extensions to.
Same rationale as with KWallet: don't surprise traditional
users by enableing it without their consent. Also the automatism
enabled it on Ubuntu Hardy where it didn't compile.
Nightly testing shows that the changes in
a3d7412972 affect other backends. This
patch removes the changes temporarily to have more time for
investigating that.
The OPTIONS request doesn't use the retry mechanismn and thus
failed hard when Google started the 401 throttling. Now such
any error is simply ignored.
The outer retry loops seems to handle the 401 Google throttling
fine, no need anymore for the identical code which was invoked
by Neon inside the credentials callback.
The REPORT request which retrieved calendar data was only sent
once. If it failed, creating the backup failed. Now this request
is tried a few times similar to the other requests. Before each
retry, the potentially incomplete backup is removed.
The retry mechanism for Google's request throttling with incorrect
401 responses broke when introducing forced authentication, because
Neon never asked for credentials in that case, which bypassed the
retry logic.
Moved the 401 handling into the outer retry loop of run() + retry.
Cleaned up operation handling: the deadline is now set once
before entering the retry logic. This is also a convenient place
to record the name of the upcoming operation (used for debugging
and error messages).
Made the information that credentials were valid before persistent as
"WebDAVCredentialsOkay" in the .internal.ini file of the
"source-config" peer. This turned out to be necessary for unit testing,
because running the next test had no knowledge that retrying is
okay and gave up too early.
The inner retry loop in the credentials callback is probably obsolete
now; will be removed.
The intention always was that backends may add their own
properties. For sync properties this seemed less relevant,
so sources only got read access to it.
But the WebDAV backends run with their own "source-config" context
and now need read/write access to store information that is
shared between all sources.
Therefore this patch removes the "const" qualifier from SyncConfig.
The logic for detecting a KDE session was reversed: if KDE_FULL_SESSION
is *not* set, then the session is not KDE. As a result, KWallet was
used inside a GNOME session instead of GNOME keyring if both were
enabled.
The problem with eGroupware is that it doesn't have one fixed syncURL.
Users of the template must remember to change it. To make this more
obvious, it is set to http://set.your.domain.here/rpc.php
The failure to fix the URL is leading to fairly obvious network
errors.
Various people have reported success with eGroupware, the latest
one here: http://www.ruinelli.ch/how-to-sync-egroupware-with-a-tablet-n900-with-syncevolution
Therefore it makes sense to also mark this template as "consumerReady"
and have it shown in the sync UI.
When Google calendar has temporary problems, it redirects to
an HTML error page. That defeated our "retry on error" logic.
Now redirects are checked inside Session::check(), with a special
case in place for this Google redirect.
When capturing lines from stdout or stderr, strip the trailing newline
before calling message() because that introduces an additional newline
by itself.
It is useful to know how long certain operations take. Instead of
adding random time stamps at the log calls, this should better be
done consistently for all lines.
In a normal sync log, such a time stamp is added by the Synthesis
logging code. For stdout logging, this patch adds it if the log
level is >= DEBUG. That way it won't distract normal users but will
be available automatically when producing a log for developers.
Because a full time dump can be long and because the delta between
events is more interesting, only relative time since the start of
logging is printed.
This is motivated by the observation that the Yahoo server becomes
unresponsive (various 5xx errors) quite often, which used to abort
the sync. Now there are better chances that it'll complete eventually,
although the root cause (server not responsive enough) remains.
PROPFIND, PUT and GET are all tried again. For a failed PUT this is
problematic because it is simply not known whether the command was
already executed. A check for "item does not exist" therefore can only
be done on the first attempt. Likewise, any future "eTag matches"
check (not yet implemented) will be also impossible.
Because of the retry, a PUT is no longer marked as not
idem-potent. The advantage is that an existing connection will be
reused for it.
The implementation is based around the idea that the higher levels in
the stack define a deadline until which the operation must succeed,
and then the lower levels (individual PROPFIND/PUT/GET) retry inside
that allowed time interval. That ensures that the initial service
discovery never takes longer that the configured timeout, which
wouldn't be the case if the timeout was applied to every single
PROPFIND inside the semantic operation.
When both KWallet and libnotify support were enabled, the redefinition
of "signals" in the Qt headers broke the GTK headers that are pulled in
by notify.h, because one of the GTK structs has a "signals" member.
Fixed by asking Qt to not define these special keywords as
preprocessor symbols.
The scripting code seemed a bit overly complex. There shouldn't be
a need for the temporary variable.
The original patch also had the side-effect of always setting
REV to its normalized version, even if it wasn't empty. That
may or may not influence other syncs, but it is better to not
take the risk.
Added preliminary Support for KJots's custom format of storing Akonadi Notes.
Currently it is assumed that all the notes it stores is of Plain Text format
(as happens in the AkoNotes Plasmoid).
Later on will have to dig into implementing support for notes in HTML and other
formats too?
Problems yet to be solved:
1) Support for KBC's custom fields.
2) The 1st ; in ADR is causing it to leave out address as a blank....
ADR;TYPE=dom;TYPE=home;TYPE=intl;TYPE=parcel;TYPE=postal;TYPE=pref;TYPE=work:123;;Planet Earth;Planet Earth;Planet Earth;Planet Earth;Home
3) "\" being expanded to "\\" and "," ot "\," in X-KADDRESSBOOK-CRYPTOPREF (not creating any problems though)
also URL is being deleted locally , but appearing remotely,
EMAIL is being replaced by FN , when tested against Ovi servers.
KApplication starts of it's own DBus service of the format
org.syncevolution.syncevolution-processId
Sending a patch upstream to stop this needs a lot more work and
should be out by KDE 4.8 (hope so).
Also
cleaned up the ugly
if (getenv("KDE_FULL_SESSION")!= true)
with
if (getenv("KDE_FULL_SESSION"))
These servers need VCards to have a Revision string. Contacts
created in KABC don't have one by default. So this patch appends a
dummy timestamp to the outgoing VCards if the RevString is empty.
Currently not enabled in compilation by default. If KWallet support is
enabled, it will check whether KDE is running and otherwise fall back
to GNOME keyring.
On the backend side, use a new "KDE" rule to enable only
the X-KADDRESSBOOK- variant of ASSISTENT/MANAGER/SPOUSE/ANNIVERSARY.
On the SyncML side, continue to use both X- and X-EVOLUTION-
because that is what SyncML servers understand.
This patch adds the necessary boiler-plate to compile
Sascha's Akonadi backend as part of SyncEvolution.
It changes the change tracking so that it is now based on the revision
number maintained by Akonadi. This removes the need to keep the sync
process running all the time to record changes.
Finding local databases (= collections) is implemented inside the
backend, with several TODOs in the code to make this nicer.
Tests were added as part of copying the boiler-plate code from the
Evolution backend. However, this depends on being able to open local
databases following the name pattern <prefix>_<type>_[12], with
<prefix> from CLIENT_TEST_EVOLUTION_PREFIX and <type> one of
ical20/vcard30/itodo20/text. This does not work at the moment.
Because the implementation of isEmpty() always returns "false", the
logic which checks whether a slow sync is acceptable will err on the
side of caution and reject slow sync, even if the local side has no
data.
This is a verbatim copy of svn://anonsvn.kde.org/home/kde/trunk/playground/pim/syncml/clientagent/funambol
revision 1072301, the latest as of January 9th 2010.
The source implements the Funambol C++ client API SyncSource API,
accessing Akonadi as the underlying database. Change tracking is based
on the end time stamp of the last sync.
The source still needs to be adapted to SyncEvolution before it can
be used there.
clang 2.9 does not support -u<symbol> and ignores it, leading to incomplete
client-test and static syncevolution binaries. Fixed by explicitly passing it
to the linker stage with -Wl,-u.
clang 2.9 pointed out that getConnection() is overloaded with
different method signatures. One is about low-level D-Bus, the other
about (HTTP) server stub communication. Distinguish that by calling
the more high-level concept "stub connection".
The methods related to sync username/password were just called
get/setUsername/Password(), with additional check/savePassword()
methods. The savePassword() method clashed with a semantically
different method for arbitrary passwords (as pointed out by clang
2.9), so let's make it more clear that one set of methods specifically
is for the sync username/password.
clang 2.9 warns about comparisons which are always true. For the
range check in a template that cannot be avoided because the type
may or may not be unsigned. Suppress the warning with clang pragmas.
clang 2.9 pointed out that getLogDir() didn't have the desired effect
anymore after a signature change in the base class. Not sure why the
test passed regardless of that, but it was indeed broken.
The callback for "source B" wasn't bound to the creator method,
which was the originall intention. Therefore such sources
were created with the default method and incomplete configs,
breaking some of the Client::Source tests.
Found in nightly testing with Valgrind. One is a boolean check on an
uninitialized variable which might have caused Basic authorization to
be sent when not desired, the other a harmless end-of run leak.
Some error scenarios, like "Google user has no calendar => 404 error",
only led to a general 20043 "communication error". This patch
preserves the remote status code by passing it from the request to
Session::check() where it can be passed on as a status exception.
Ubuntu Hardy's boost does not have reset() in intrusive_ptr.
Can be replaced with constructing a ptr followed by swap().
glib lacks the flag for combining certain change events. We don't
need this (although it reduces overhead a bit), so let's simply drop it.
g++ on Ubuntu Hardy did not agree with clang++ on scoping rules.
Both seem to work with global ref/unref functions (outside of
boost namespace), so let's use that.
This patch also covers the case that the server needs to restart
because it has to run automatic syncs. The restart is implemented as a
brute-force execve() after removing output redirection, using the
original argv and env arrays (Restart class).
syncevo-dbus-server must restart after its installation was updated or removed.
Otherwise further sync attempts can fail. This was seen in practice when
SyncEvolution 1.0 was updated to 1.1 (Debian bug #617320): the in-memory daemon
used an old libsynthesis, but the on-disk XML files required more recent
libsynthesis features.
In general, *any* update of something loaded into memory should
trigger a shutdown or restart. A shutdown alone is okay when no
automatic sync scheduling is needed (auto sync off for all
configs). Clients will restart the daemon on demand. A restart is
needed otherwise because without it, automatic syncs would stop to
run.
This patch implements the shutdown part. Restart still needs to be
implemented. A 10 second delay is chosen between "file modified" and
"shut down". This is meant to ensure that a future restart does not
occur too soon (before all file changes are done). It's still a bit
racy, but a better solution would depend integration into
distro-specific hooks ("package update complete"), which is hard or
impossible (installation via "make install" or "tar xf").
This new feature is tested by test-dbus.py, including several corner
cases:
- testShutdown: files modified in regular intervals for a while
- testSession: a running session prevents the shutdown
- testSession2: same as testSession, with different timing
The Boost::intrusive_ptr works with GObject's own reference counting.
Defines in the global name space are needed for each GObject type to
tell Boost that the corresponding C struct is indeed a GObject.
The file notification class couples the C callbacks with
boost::function.
gcc 4.5.2 complained that a "case" value is not among those valid for
the "switch" enum. That's correct, the enum might be smaller than the
value. The root cause is an incorrect use of the enum type in the
callback: the signal itself is defined with gint.
SyncEvolution used to insist on having a template when creating a
config from scratch. This was meant to address typos like "--configure
sheduleworld". But when the command line contains enough information,
no template is needed.
The same logic was applied to configuring a source: if a source was not
among those defined by the template, configuring it was rejected as a
typo.
With this patch, there are two ways around these checks:
- --template none and/or
- specifying required properties on the command line
This patch also removes the special cases for "your SyncML server
account name/password". Username/password aren't always needed
(previous patch), nor are they always for a SyncML server anymore.
The value is a QtContacts URI for an address book, composed of
qtcontacts:<backend>:<URL encoded parameters>
Example values for "database":
qtcontacts:tracker:
qtcontacts:eds:source=local:/system
Fedora now includes a shared libgdbus under /usr/lib. This leads to a
conflict based on the *name* of the library, not just because of
*symbols* (which was resolved earlier).
Therefore this patch renames the version that SyncEvolution depends on
to libgdbusyncevo. All SyncEvolution patches were already sent
upstream, but it is uncertain whether libgdbus will be see further
releases. Patches to make SyncEvolution use a suitable system libgdbus
library are welcome.
The intention is that backends shall be able to bind additional
parameters to their implementation of the callbacks. The default
implementation used by most (all?) backends was adapted, so no
changes there are necessary.
This reverts commit ad608ca221.
vCard 3.0 has problems with other peers, for example Mobical. Better
stay conservative and use vCard 2.1 by default, as before. MemoToo
is dealt with by the previous commit which explicitly sets vCard 3.0
for MemoToo.
This patch fixes a problem that occured if:
* a peer was configured with a ifferently named template (--configure --template foo bar)
* and there was a config foo@xyz.
Then the nonsense error message "ERROR: template foo must not specify a context."
was printed (nonsense because "foo" does not specify a context) and the operation
failed.
The root cause was that SyncConfig::normalizeConfigString() expanded
the "foo" string to "foo@xyz" (desired when "foo" is a shorthand for
an existing config). This behavior now can be turned off with a
flag. The patch does that by replacing the bool parameter with a bit
field.
When SyncEvolution prefers vCard 3.0, additional properties like
NICKNAME work with Memotoo. This patch a) makes that the default
syncFormat in the Memotoo template and b) automatically sets it
in the "addressbook" source of all migrated configs which sync
with Memotoo (detected based on the syncURL).
The automatic config rewriting is possible because users upgrading
from the previous stable SyncEvolution release need to migrate their
configs anyway. Normally SyncEvolution does not modify configs like
this.
The reason for this change is that MemoToo works better
when vCard 3.0 is announced as the preferred format by
the client. Only then does it support the NICKNAME property,
for example.
The reason for keeping vCard 2.1 as the preferred format
where legacy servers. Not sure whether that still is an
issue. If it is, then these servers must have their
"syncFormat" property set explicitly in their templates
and existing configs.
Asking for .well-known wasn't leading to a redirect with Yahoo.
The RFC recommends trying the root path instead, which happens
to work.
A slight tweak was necessary to handle Yahoo's returning of
"/dav" properties when "/" is queried. With that it is possible
to unify the caldav/carddav code paths and remove the hard-coded
Yahoo paths.
The automatic service discovery was incomplete resp. faulty. Changes
necessary to make it work with Apple Calendar Server (both CalDAV and
CardDAV):
- properly detect redirects in PROPFIND
- send credentials as Basic authorization for PROPFIND, to
get "current-user-principal" properly set by server (<unauthenticated>
is returned by Apple Calendar Server otherwise)
- use the "current-user-principal" href to find calendar/addressbook-home-set
- extracting that href must be more intelligent to cope with
<DAV:href xmlns=...>
- avoid shared collections (we want the user's personal collection at this time)
Sending invalid credentials over HTTP was considered, but fails with
Apple Calendar Server if only Basic authentication is enabled, without
causing the server to send a challenge. Falling back to sending valid
credenticals would make the code more complex for a sitation which is
already insecure (plain text transmission of PIM data!) and thus
wasn't implemented.
See also:
http://tools.ietf.org/html/rfc4918#appendix-Ehttp://lists.w3.org/Archives/Public/w3c-dist-auth/2005OctDec/0243.htmlhttp://thread.gmane.org/gmane.comp.web.webdav.neon.general/717/focus=719
ne_request_create() copies all parameters, but due to a bug in neon,
Digest authentication ended up using the original parameter. With
SyncEvolution freeing it before running the request, the ensuing
read-after-use error caused authentication problems with servers using
Digest authentication (Apple Calendar Server) or might even have
crashed.
Patch submitted and accepted upstream, but better work around it by
keeping the affected "method" string valid throughout the lifetime of
the request.
libneon 0.27 doesn't have ne_options2() and
ne_session_system_proxy(). ne_options2() is purely for debugging,
disabled in builds on old distros. ne_session_system_proxy() is needed
to enable the use of the system proxy settings. If a recent enough
libneon.so.27 is found at runtime (= one which has the function), then
it is called.
libical was contained in libecal in Ubuntu Hardy, at least the way how
SyncEvolution is compiled there. Thus use libecal-1.2 when libical is
not found. Also handle the different ical string allocation semantic
via the icalstrdup() wrapper.
Use a more specific error code instead STATUS_FATAL because otherwise
the UI cannot inform the user.
CalDAV sync attempt while offline => DNS SRV fails => STATUS_TRANSPORT_FAILURE.
No sync URL (as with Yahoo) + no domain in user name => DNS SRV impossible => STATUS_UNAUTHORIZED.
Yahoo CardDAV and CalDAV config only works with full domain (needed
for DNS SRV lookup). Add it if not present. If the resulting
username is invalid, we'll get a credentials error, as desired.
The RetryDuration value now defines the time after which the Neon
library is told to give up. Previously some Neon-internal default
was used.
Resending a pending request earlier still needs to be implemented.
As with SyncML over HTTP, a loss of network connectivity can lead
to a half-open TCP connection which the client cannot detect when
relying on TCP alone.
Non-ASCII characters and pictures get lost. X- Extensions can be
uploaded and are shown on the server, but are not returned by it.
These issues are all on the server side and it is unclear how to
work around them.
Therefore this patch disables contact support in the Yahoo! profile.
Can be enabled manually for testing purposes.
SyncEvolution itself is careful to not send a VEVENT with
RECURRENCE-ID unless the parent item is also in the Google Calendar.
If that is done, then the Google CalDAV server reports the item
in a REPORT, but cannot GET or DELETE it (404 error).
Because other clients might get the server into such a state (Android
client was mentioned), we must deal with it nevertheless. Retrieving
such an item uses a REPORT with UID filter as fallback. A
calendar-multiget had the same issue as a single GET. That code is in
the patch, but not enabled.
Deleting has no fallback. The item will simply remain on the server
and no error will be shown to users. Note that the same 404 error
might also occur in case of a concurrent delete by some other client,
so it makes sense to ignore this.
Just to be sure, flush stderr after running a neon operation and give
the OS a chance to process output redirection, if neon prints output
and redirection is active. It is uncertain whether this is really
necessary, but because it is off most of the time, it doesn't hurt
either.
The "error line" debug output from neon was recognized as potential
error by LogRedirect and shown to users at higher log levels. This patch
uses the new LogRedirect::addIgnoreError() to suppress this line.
A GET of a vCard returns data that seems to have gone through an
encoding step for transmission as part of HTML or XML: special
characters are replaced by entity codes. Worse, that step seems to be
done multiple times, leading to:
backslash &#92; single quote ' double quote "
instead of:
backslash \ single quote ' double quote "
Note the double encoding of \ as &#92; - really broken.
This patch works around that by unconditionally and repeatedly
replacing XML entities with the corresponding characters until no
entities are left in the item data. It would be nice to only do that
when such a broken server is detected, but detecting that is hard and
not implemented.
The downside obviously is that entities that the user (for whatever
reason) wants to have in his contact data will be replaced.
Enabled support for Yahoo! Contacts, by adding it to the configuration
created for Yahoo. Sending data to it requires suppressing empty
properties, which it doesn't seem to handle in all cases (empty
X-GENDER, for example).
When the syncURL setting is empty, the username is checked for
a domain part and that is used to find the CalDAV or CardDAV service.
The actual DNS lookup is done by invoking a shell script and wrapping
some common tools (host, dnslookup, adnshost). This approach avoids a)
keeps the feature optional (no hard library dependency) and b) is more
portable across platforms (availability of DNS resolver libraries varies,
sometimes the API for SRV lookup is poor).
No attempt is made to also find the right URL path via a TXT lookup,
because the services which this patch was tested with didn't seem to
support it.
Instead of the TXT lookup, .well-known paths are tried. But even that
didn't work with the services, so the search code in WebDAV still
needs to hard-code some well-known paths and tries them in a (now)
depth-first search. Services with full support for a redirect of these
known paths should work (but untested).
Previously, the credentials were read directly when instantiating
the ButeoBridge. That happened before the SSO patch in msyncd
looked up the real credentials. Now the credentials are retrieved
in startSync(), where they are available.
When backends are compiled as modules, then the SyncSource
ScannedModules instance must be initialized before the backends
are available. This can be achieved by accessing information
obtained from that class via SyncSource::backendsInfo().
Not doing so resulted in "no backend" errors when
trying to init the configs.
This patch reduces the number of header files needed from core
SyncEvolution. TransportException is expected to be in util.h
instead of TransportAgent.h.
Compile flags are taken from SYNCEVOLUTION_CFLAGS/LIBS instead of
hard-cording paths as they are when compiling together with
SyncEvolution.
Check whether configs already exist instead of overwriting them
each time. Allows local modifications of the properties which
are set (useful for debugging) and is more efficient (always
useful).
Interfacing with the Cmdline class turned out to be the easiest
way of implementing the functionality. It is a bit indirect, but
replicating the logic for setting up and updating the config
would have required more code.
Currently the config is updated each time the plugin runs. Should
change that so that an existing config is left unchanged.
Error handling is incomplete.
When parsing data received from the server, we may end up with
X-LIC-ERROR error messages from libical in the internal
representation. This will get added to the server when combining
multiple items, which is undesirable (waste of bandwith, breaks
comparison).
This patch solves this by filtering out such errors.
Google Calendar adds a VALARM when a VEVENT is created for the first
time on the server via CalDAV. This is not what we want, if the VEVENT
had no VALARM, that's how it should be stored.
As a workaround we detect this special case (new VEVENT without
VALARM) and send the original data again. Having to resend with higher
SEQUENCE and LAST-MODIFIED/DTSTAMP values makes this a bit complex and
slow, because we need to get the actual data from the server (cannot
guess what time stamps were assigned).
This leads to an interesting question: if the clock on the local side
is unsynchronized, its LAST-MODIFIED time stamps might end up being
lower than the values on the server, which prevents sending local
updates. There's currently no solution for this in the backend.
Starting a neon Session is slow, in particular when libproxy is
involved (seems to search for proxy servers, which involves a DNS
timeout when none are around). Another delay is opening the TCP
connection and determining which kind of autentication is needed.
For a normal program run there's not much that can be done to speed
this up, but in client-test it helps a lot to reuse an existing
Session instance between different tests. This patch enables that
by keeping one Session instance around and returning that if the new
one has the same parameters.
The Settings pointer is a boost::shared_ptr, but this patch avoids
using it unless the Session is in active use, just in case that a user
wants to merge the settings into a class with a different life time.
It was observed once (but could no longer be reproduced after reducing the
number of requests) that Google Calendar started to reject requests
with valid credentials with a 401 credential error. Presumably that
was a counter-measure for excessive number of requests.
This patch adds a retry mechanism for this particular case, with
exponentially increasing delays before retrying the same request until it
succeeds or one minute has passed. A credential error on the very first
request in a session still fails immediately. This is implemented as
part of the normal neon "get credentials" callback, so the logic
on top of neon is unchanged.
Both CardDAV and CalDAV allow collections inside the collection of
items which are to be synchronized. They may not contain other contact
resp. calendar data and simply can be ignored.
This patch is only relevant for CardDAV; CalDAV uses its own method
for finding calendar items which already ignores non-calendar items.
This patch moves some of the CalDAV-specific code out of WebDAVSource
into CalDAVSource and adds the corresponding CardDAV code in the new
CardDAVSource. It improves searching for the right collection.
A Yahoo Contacts specific problem is that vCard UID and resource name
have to match, with ".vcf" used as suffix in the resource name. The
new createResourceName() and setResourceName() virtual methods ensure
that. They are nops in CalDAV and use simple string manipulation for
vCard.
Another Yahoo Contacts odity is merging of contacts in PUT, without
telling the client. This can be detected by checking the expected
resource name with PROPFIND: the server then replies with the real
resource name. This check is done only if no href is returned in the
response to PUT, so it should only be triggered for Yahoo Contacts.
CardDAV support is still experimental. It was tested with Yahoo
Contacts, using https://carddav.address.yahoo.com/dav/%u as sync
URL. This is different from the CalDAV URL, which currently prevents
using the same @yahoo context for both calendar and contacts.
Working on CardDAV showed that resource name comparisons had issues
due to different encoding of special characters. Added
canonicalization of paths to fix this problem. This also includes
appending the trailing slash for collections
(Neon::URI::normalizePath()).
Note that Buteo only serializes sync sessions based on their type, so
as soon as we allow CardDAV in addition to CalDAV, it might happen
that SyncEvolution is invokved multiple times in parallel => need to
remove global variables first.
If proxy settings are turned off in the config, the default system
proxy settings are used. This depends on ne_session_system_proxy(),
introduced in neon 0.29. The version is checked at configure time. A
fallback if only an older version is found is intentionally not
present yet, the feature is meant to work in all cases.
The username might contain special characters. It must be escaped
before adding it to the URL. Found when the code was used with an
the default "Your SyncML Username" username string.
Yahoo works without any of the Google specific hacks. Because these
have side effects (performance, change data), they shouldn't be used
when talking to other CalDAV servers.
This patch adds a "SyncEvolution" URL parameter with a comma separated
list of flags:
"UpdateHack" = increase SEQUENCE and LAST-MODIFIED
"ChildHack" = replace RECURRENCE-ID in stand-alone child events
This needs to be appended to the syncURL, like this:
syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=UpdateHack,ChildHack
The reason for using "SyncEvolution" here is that other parameters
might one day be passed through to the underlying server; at the
moment, anything other than "SyncEvolution" causes an error.
They need to specified explicitly. Hard-coding a check for www.google.com
did not feel right.
Content-Type must be set, otherwise Yahoo CalDAV rejects the
items with "Bad Request".
The resource name must match the UID + .ics. It is okay to
escape special characters like @ in the PUT, even though the
server doesn't do the same in PROPFIND.
For change tracking we need a canonical form for DAV luids. Always
unescaping the relative path component is safe. For absolute paths, we
use the value as suggested by the server. There's no risk of mismatch
here, because we never create such a luid ourself.
This patch implements a dynamic search for the "default" calendar
of a user, starting with the user's principal URL:
https://caldav.calendar.yahoo.com/dav/<user>/
https://www.google.com/calendar/dav/<user>/user/
Stops when:
- current path is calendar collection (= contains VEVENTs)
Gives up:
- when running in circles
- nothing else to try out
- tried 10 times
Follows:
- CalDAV calendar-home-set
- collections
This implies that a specific calendar can be selected by using its URL
directly. The search for "default" calendar is a bit random, because
it depends on the naming of calendars on the server: calendars with
lexicographically lower name are tested (and thus found) first.
Without any kind odf <filter>, Yahoo rejected the request with 400
"Bad Request". Now pass a filter for VEVENT without actually
filtering on any property.
Yahoo requests a client certificate as part of setting up the SSL
connection. Without such a certificate, gnutls aborts setting up
the connection.
As a workaround this patch reads "client.p12" in the current
directory and uses that as certificate. Cannot be the final
solution - replace by dynamically generated certificate or use
ne_ssl_provide_clicert(), with empty list of certificates?
When a child is added without the corresponding parent, Google gets
confused (child is added, but cannot be retrieved). This patch turns a
child event into a normal event by replacing RECURRENCE-ID with
X-SYNCEVOLUTION-RECURRENCE-ID. The inverse operation is applied when
importing data back from Google.
Google requires that all items sent to it have a higher SEQUENCE
number than the copy on the server. Calendar applications are allowed
to make changes to an item even when they are not the organizer, in
which case they are *not* meant to increase the SEQUENCE.
This patch works around the issue by increasing the SEQUENCE number
artifically before sending to the CalDAV server (regardless whether
it is Google or something else).
It does that by keeping track of UID and SEQUENCE number in the
cache. The UID is used to find the resource before sending in case that
the caller didn't know, the SEQUENCE to know what the outgoing item
must have.
The sequence number is shared among all sub items. That is an
intentional simplification of the implementation which may or may not
be necessary to work around the issue. If we have to touch the
sequence number, we might as well do it for all sub items.
This patch enhances error handling so that a more specific
TransportStatusException with the right SyncMLStatus inside is
thrown. This is necessary to report back authentication errors.
libical needs to be checked for explicitly, instead of assuming
that it is pulled in via the ecal backend.
The neon package check can be deferred until the backend really
is needed.
Relying on the MapSyncSource and our own caching hides problems in
both. A backup mechanism taking data directly from the CalDAV server
is more reliable.
This patch implements that by running a REPORT and dumping the
returned data. It stores each VCALENDAR as one file, which may contain
multiple VEVENTs with the same UID. synccompare handles that.
The patch refactors the REPORT issuing because the new backupData()
and listAllSubItems() can share most of that code.
Restore is not implemented yet. The start of a sync could be optimized
by adding all backed up items to a revision list and reusing it in
listAllSubItems(), like SyncSourceRevisions does.
This implements logging like in EvolutionCalendarSource: SUMMARY
plus optional LOCATION. If possible, the item is taken from the cache.
A better solution would move the common "work with icalcomponent"
code into a shared part of libsyncevolution... left for later.
Implemented with a single REPORT request, instead of one query for
resource ID/etag pairs followed by one GET per item.
In theory also less data needs to be transmitted, because we only ask
for the minimum number of iCalendar 2.0 properties required at that
point. In practice, Google delivers full iCalendar 2.0 items. Because
we cannot rely on that, we do not populate the cache with these and
have to fetch those anew which are later involved in operations.
Overall this should still be more efficient.
When a single sub item in a merged item is updated, we modify the
calendar and sent the complete item to the server. Google didn't
like that and rejected the update because of "409 - Can only store an
event with a newer DTSTAMP." Found in testLinkedItemsInsertBothUpdateParent.
The solution is to update SEQUENCE and DTSTAMP/LAST-MODIFIED to that of
the recently updated sub item, then Google accepts the update. Updating
just one or the other was not enough.
This change should work even if more than one item is updated, as long
as a) item changes are reported in the right order (oldest change
first) or b) all sub items get updated.
a) is not guaranteed, b) might be. To be tested...
This patch implements the SubSyncSource interface in CalDAVSource,
then instantiates that class as backend for a MapSyncSource which
bridges between the core engine (one VEVENT per item) and CalDAV (all
VEVENTs with same UID in the same item).
The tests must run with a modified ical20.ics.google.tem which
does not contain RECURRENCE-ID, because that is not yet supported.
Also, our complex 1234567890!@#$%^&*()<>@dummy must be avoided
because Google has an issue with it (doesn't find resulting resource):
CLIENT_TEST_SERVER=google CLIENT_TEST_SIMPLE_UID=1 ./client-test Client::Source::caldav_ical20::testImport
This patch fixes some (but not all) issues found in testImport:
- readItem() did not clear the item string before appending
the new item.
- Detection of "item existed already" cannot use the alternative name
returned by the server, because Google does that also for items
which are really new.
- The test registered itself as "ical20", which conflicts with the EDS
backend. Now use "caldav_ical20".
This replaces the WEBDAV_* env variables with access to the
normal per-peer properties, passed into the source via the new
SyncSourceParams context member.
When used on the command line, @foobar context configs are not
enough to use the WebDAV sources. A properly configured peer
context (foo@bar) is needed. There is no explicit check for this,
the resulting error is about not being able to access the empty
URL `' (Neon uses these quotation marks).
Mapping removeItem() to DELETE is straight-forward. Except that CalDAV
will require additional support for removing individual VEVENTs from
a resource that contains more than one VEVENT...
This works for updating and adding an item. Storing a new item under a
different resouce URI is also handled, which can happen with CalDAV
when there is already an item with the same UID.
It would be nice if the server returned a ETag in response to a PUT,
but Google Calendar doesn't do that. The header is checked neverthless
(untested!), with an explicit PROPGET as fallback.
This patch also clarifies LUID (relative if possible, absolute if not)
and revision string handling (without W/ and quotation marks). We
don't care about weak ETags at the moment, Google doesn't use
them. Even if a server did, what would we do with the information that
the item wasn't stored verbatim?
The current implementation is not yet good enough for CalDAV:
- only one VEVENT per UID supported: must merge and split
related VEVENTs into one CalDAV calendar resource
- adding or updating VEVENT without UID probably fails:
must insert new or old UID
Listing items in WebDAV consists of getting the ETag for each URI
directly underneath the collection. Checking for "empty source"
is done by reading all pairs and then checking for non-empty map.
It would be nice to abort the processing once an error is encountered
or once "not empty" has been observed, but Neon does not support that.
Probably for a good reason, because it would render the TCP connection
useless.
This wraps the "simple" Neon propfind method which handles plain
strings as values. For more complex values, the value is an XML
fragment which should better be parsed using the "complex" Neon
propfind method with XML handlers.
The callbacks are defined by boost::function, which allows attaching
additional parameters to each callback. Exceptions must not cross Neon
library calls.
The Neon::Session wraps most of the relevant calls. It is
parameterized by Neon::Settings. It is uncertain where all of these
are meant to come from, because there is no peer configuration in many
cases. Perhaps we can enforce that a WebDAV source may only be created
in a context which has one and exactly one peer config?
The current intermediate solution in WebDAVSourceRegister.cpp grabs
all settings from WEBDAV_* env variables.
Disabling SSL verification and Neon debug logging are implemented.
Opening the source runs a few checks on the URL. Disabling SSL
certificate checking turned out to be necessary, probably because of
the known issue of gnutls not trusting the weak Google certificate
chain.
This patch maps QContactDetails which have no mapping to vCard to
X-SYNCEVO-QTCONTACTS. The value of that extension property is a dump
of the complete QContactDetail. Reading it again later can restore
that detail.
The goal is to make this additional information available to the
Synthesis engine such that it can preserve the additional information.
Some peers might even be able to store them.
One of the features of the Synthesis engine is that it can preserve
properties locally which are not supported by the
peer. "testExtensions" covers that, by updating all items on the
server (via client B) and reimporting them as updates into client A.
The data comparison then is done without the usual "allow server to
drop data" simplifications in synccompare.
This test is not enabled yet. "config.update" needs to be set for it,
but doing so then trips over other changes introduced by servers, like
for example re-encoding photos. Needs some more thoughts and testing...
The QVersitContactHandlerFactory::ProfileBackup profile enables a
vCard flavor which contains all information stored in a QtContact. It
might make sense to use that format for raw access (--import/export).
Not currently enabled though because the format is pretty verbose.
clang 2.8 compiles SyncEvolution + Synthesis faster than g++ 4.4.5
(3:40min instead of 4:10min on my laptop) and produces more useful
error reports. This patch fixes the code so that it compiles cleanly
with clang when using "-Wall -Werror -Wno-unknown-pragmas". Note that
clang 2.6 (Debian Squeeze) goes into an infinite recursion compiling
code using gdbus-cxx-bridge.h and dies eventually with a stack
overflow - can't be used.
Changes necessary for clang:
- eptr pointer referencing ambiguous, use *x.get() instead
- boost::intrusive_ptr* must be defined before code using it
- two-phase template checking requires explicitly specifying
members in base classes
- name clashes with plain C structs (DBusServer, DBusWatch) are
an error and need to be avoided (done with namespaces GDBusCXX and
SyncEvo)
- floats cannot be inline constants
- unused methods in local classes are warned about (left() in SyncML.cpp)
