The outer retry loops seems to handle the 401 Google throttling
fine, no need anymore for the identical code which was invoked
by Neon inside the credentials callback.
The REPORT request which retrieved calendar data was only sent
once. If it failed, creating the backup failed. Now this request
is tried a few times similar to the other requests. Before each
retry, the potentially incomplete backup is removed.
The retry mechanism for Google's request throttling with incorrect
401 responses broke when introducing forced authentication, because
Neon never asked for credentials in that case, which bypassed the
retry logic.
Moved the 401 handling into the outer retry loop of run() + retry.
Cleaned up operation handling: the deadline is now set once
before entering the retry logic. This is also a convenient place
to record the name of the upcoming operation (used for debugging
and error messages).
Made the information that credentials were valid before persistent as
"WebDAVCredentialsOkay" in the .internal.ini file of the
"source-config" peer. This turned out to be necessary for unit testing,
because running the next test had no knowledge that retrying is
okay and gave up too early.
The inner retry loop in the credentials callback is probably obsolete
now; will be removed.
The intention always was that backends may add their own
properties. For sync properties this seemed less relevant,
so sources only got read access to it.
But the WebDAV backends run with their own "source-config" context
and now need read/write access to store information that is
shared between all sources.
Therefore this patch removes the "const" qualifier from SyncConfig.
The logic for detecting a KDE session was reversed: if KDE_FULL_SESSION
is *not* set, then the session is not KDE. As a result, KWallet was
used inside a GNOME session instead of GNOME keyring if both were
enabled.
The problem with eGroupware is that it doesn't have one fixed syncURL.
Users of the template must remember to change it. To make this more
obvious, it is set to http://set.your.domain.here/rpc.php
The failure to fix the URL is leading to fairly obvious network
errors.
Various people have reported success with eGroupware, the latest
one here: http://www.ruinelli.ch/how-to-sync-egroupware-with-a-tablet-n900-with-syncevolution
Therefore it makes sense to also mark this template as "consumerReady"
and have it shown in the sync UI.
When Google calendar has temporary problems, it redirects to
an HTML error page. That defeated our "retry on error" logic.
Now redirects are checked inside Session::check(), with a special
case in place for this Google redirect.
When capturing lines from stdout or stderr, strip the trailing newline
before calling message() because that introduces an additional newline
by itself.
It is useful to know how long certain operations take. Instead of
adding random time stamps at the log calls, this should better be
done consistently for all lines.
In a normal sync log, such a time stamp is added by the Synthesis
logging code. For stdout logging, this patch adds it if the log
level is >= DEBUG. That way it won't distract normal users but will
be available automatically when producing a log for developers.
Because a full time dump can be long and because the delta between
events is more interesting, only relative time since the start of
logging is printed.
This is motivated by the observation that the Yahoo server becomes
unresponsive (various 5xx errors) quite often, which used to abort
the sync. Now there are better chances that it'll complete eventually,
although the root cause (server not responsive enough) remains.
PROPFIND, PUT and GET are all tried again. For a failed PUT this is
problematic because it is simply not known whether the command was
already executed. A check for "item does not exist" therefore can only
be done on the first attempt. Likewise, any future "eTag matches"
check (not yet implemented) will be also impossible.
Because of the retry, a PUT is no longer marked as not
idem-potent. The advantage is that an existing connection will be
reused for it.
The implementation is based around the idea that the higher levels in
the stack define a deadline until which the operation must succeed,
and then the lower levels (individual PROPFIND/PUT/GET) retry inside
that allowed time interval. That ensures that the initial service
discovery never takes longer that the configured timeout, which
wouldn't be the case if the timeout was applied to every single
PROPFIND inside the semantic operation.
When both KWallet and libnotify support were enabled, the redefinition
of "signals" in the Qt headers broke the GTK headers that are pulled in
by notify.h, because one of the GTK structs has a "signals" member.
Fixed by asking Qt to not define these special keywords as
preprocessor symbols.
The scripting code seemed a bit overly complex. There shouldn't be
a need for the temporary variable.
The original patch also had the side-effect of always setting
REV to its normalized version, even if it wasn't empty. That
may or may not influence other syncs, but it is better to not
take the risk.
Added preliminary Support for KJots's custom format of storing Akonadi Notes.
Currently it is assumed that all the notes it stores is of Plain Text format
(as happens in the AkoNotes Plasmoid).
Later on will have to dig into implementing support for notes in HTML and other
formats too?
Problems yet to be solved:
1) Support for KBC's custom fields.
2) The 1st ; in ADR is causing it to leave out address as a blank....
ADR;TYPE=dom;TYPE=home;TYPE=intl;TYPE=parcel;TYPE=postal;TYPE=pref;TYPE=work:123;;Planet Earth;Planet Earth;Planet Earth;Planet Earth;Home
3) "\" being expanded to "\\" and "," ot "\," in X-KADDRESSBOOK-CRYPTOPREF (not creating any problems though)
also URL is being deleted locally , but appearing remotely,
EMAIL is being replaced by FN , when tested against Ovi servers.
KApplication starts of it's own DBus service of the format
org.syncevolution.syncevolution-processId
Sending a patch upstream to stop this needs a lot more work and
should be out by KDE 4.8 (hope so).
Also
cleaned up the ugly
if (getenv("KDE_FULL_SESSION")!= true)
with
if (getenv("KDE_FULL_SESSION"))
These servers need VCards to have a Revision string. Contacts
created in KABC don't have one by default. So this patch appends a
dummy timestamp to the outgoing VCards if the RevString is empty.
Currently not enabled in compilation by default. If KWallet support is
enabled, it will check whether KDE is running and otherwise fall back
to GNOME keyring.
On the backend side, use a new "KDE" rule to enable only
the X-KADDRESSBOOK- variant of ASSISTENT/MANAGER/SPOUSE/ANNIVERSARY.
On the SyncML side, continue to use both X- and X-EVOLUTION-
because that is what SyncML servers understand.
This patch adds the necessary boiler-plate to compile
Sascha's Akonadi backend as part of SyncEvolution.
It changes the change tracking so that it is now based on the revision
number maintained by Akonadi. This removes the need to keep the sync
process running all the time to record changes.
Finding local databases (= collections) is implemented inside the
backend, with several TODOs in the code to make this nicer.
Tests were added as part of copying the boiler-plate code from the
Evolution backend. However, this depends on being able to open local
databases following the name pattern <prefix>_<type>_[12], with
<prefix> from CLIENT_TEST_EVOLUTION_PREFIX and <type> one of
ical20/vcard30/itodo20/text. This does not work at the moment.
Because the implementation of isEmpty() always returns "false", the
logic which checks whether a slow sync is acceptable will err on the
side of caution and reject slow sync, even if the local side has no
data.
This is a verbatim copy of svn://anonsvn.kde.org/home/kde/trunk/playground/pim/syncml/clientagent/funambol
revision 1072301, the latest as of January 9th 2010.
The source implements the Funambol C++ client API SyncSource API,
accessing Akonadi as the underlying database. Change tracking is based
on the end time stamp of the last sync.
The source still needs to be adapted to SyncEvolution before it can
be used there.
clang 2.9 does not support -u<symbol> and ignores it, leading to incomplete
client-test and static syncevolution binaries. Fixed by explicitly passing it
to the linker stage with -Wl,-u.
clang 2.9 pointed out that getConnection() is overloaded with
different method signatures. One is about low-level D-Bus, the other
about (HTTP) server stub communication. Distinguish that by calling
the more high-level concept "stub connection".
The methods related to sync username/password were just called
get/setUsername/Password(), with additional check/savePassword()
methods. The savePassword() method clashed with a semantically
different method for arbitrary passwords (as pointed out by clang
2.9), so let's make it more clear that one set of methods specifically
is for the sync username/password.
clang 2.9 warns about comparisons which are always true. For the
range check in a template that cannot be avoided because the type
may or may not be unsigned. Suppress the warning with clang pragmas.
clang 2.9 pointed out that getLogDir() didn't have the desired effect
anymore after a signature change in the base class. Not sure why the
test passed regardless of that, but it was indeed broken.
The callback for "source B" wasn't bound to the creator method,
which was the originall intention. Therefore such sources
were created with the default method and incomplete configs,
breaking some of the Client::Source tests.
Found in nightly testing with Valgrind. One is a boolean check on an
uninitialized variable which might have caused Basic authorization to
be sent when not desired, the other a harmless end-of run leak.
Some error scenarios, like "Google user has no calendar => 404 error",
only led to a general 20043 "communication error". This patch
preserves the remote status code by passing it from the request to
Session::check() where it can be passed on as a status exception.
Ubuntu Hardy's boost does not have reset() in intrusive_ptr.
Can be replaced with constructing a ptr followed by swap().
glib lacks the flag for combining certain change events. We don't
need this (although it reduces overhead a bit), so let's simply drop it.
g++ on Ubuntu Hardy did not agree with clang++ on scoping rules.
Both seem to work with global ref/unref functions (outside of
boost namespace), so let's use that.
This patch also covers the case that the server needs to restart
because it has to run automatic syncs. The restart is implemented as a
brute-force execve() after removing output redirection, using the
original argv and env arrays (Restart class).
syncevo-dbus-server must restart after its installation was updated or removed.
Otherwise further sync attempts can fail. This was seen in practice when
SyncEvolution 1.0 was updated to 1.1 (Debian bug #617320): the in-memory daemon
used an old libsynthesis, but the on-disk XML files required more recent
libsynthesis features.
In general, *any* update of something loaded into memory should
trigger a shutdown or restart. A shutdown alone is okay when no
automatic sync scheduling is needed (auto sync off for all
configs). Clients will restart the daemon on demand. A restart is
needed otherwise because without it, automatic syncs would stop to
run.
This patch implements the shutdown part. Restart still needs to be
implemented. A 10 second delay is chosen between "file modified" and
"shut down". This is meant to ensure that a future restart does not
occur too soon (before all file changes are done). It's still a bit
racy, but a better solution would depend integration into
distro-specific hooks ("package update complete"), which is hard or
impossible (installation via "make install" or "tar xf").
This new feature is tested by test-dbus.py, including several corner
cases:
- testShutdown: files modified in regular intervals for a while
- testSession: a running session prevents the shutdown
- testSession2: same as testSession, with different timing
The Boost::intrusive_ptr works with GObject's own reference counting.
Defines in the global name space are needed for each GObject type to
tell Boost that the corresponding C struct is indeed a GObject.
The file notification class couples the C callbacks with
boost::function.
gcc 4.5.2 complained that a "case" value is not among those valid for
the "switch" enum. That's correct, the enum might be smaller than the
value. The root cause is an incorrect use of the enum type in the
callback: the signal itself is defined with gint.
SyncEvolution used to insist on having a template when creating a
config from scratch. This was meant to address typos like "--configure
sheduleworld". But when the command line contains enough information,
no template is needed.
The same logic was applied to configuring a source: if a source was not
among those defined by the template, configuring it was rejected as a
typo.
With this patch, there are two ways around these checks:
- --template none and/or
- specifying required properties on the command line
This patch also removes the special cases for "your SyncML server
account name/password". Username/password aren't always needed
(previous patch), nor are they always for a SyncML server anymore.
The value is a QtContacts URI for an address book, composed of
qtcontacts:<backend>:<URL encoded parameters>
Example values for "database":
qtcontacts:tracker:
qtcontacts:eds:source=local:/system
Fedora now includes a shared libgdbus under /usr/lib. This leads to a
conflict based on the *name* of the library, not just because of
*symbols* (which was resolved earlier).
Therefore this patch renames the version that SyncEvolution depends on
to libgdbusyncevo. All SyncEvolution patches were already sent
upstream, but it is uncertain whether libgdbus will be see further
releases. Patches to make SyncEvolution use a suitable system libgdbus
library are welcome.
The intention is that backends shall be able to bind additional
parameters to their implementation of the callbacks. The default
implementation used by most (all?) backends was adapted, so no
changes there are necessary.
This reverts commit ad608ca221.
vCard 3.0 has problems with other peers, for example Mobical. Better
stay conservative and use vCard 2.1 by default, as before. MemoToo
is dealt with by the previous commit which explicitly sets vCard 3.0
for MemoToo.
This patch fixes a problem that occured if:
* a peer was configured with a ifferently named template (--configure --template foo bar)
* and there was a config foo@xyz.
Then the nonsense error message "ERROR: template foo must not specify a context."
was printed (nonsense because "foo" does not specify a context) and the operation
failed.
The root cause was that SyncConfig::normalizeConfigString() expanded
the "foo" string to "foo@xyz" (desired when "foo" is a shorthand for
an existing config). This behavior now can be turned off with a
flag. The patch does that by replacing the bool parameter with a bit
field.
When SyncEvolution prefers vCard 3.0, additional properties like
NICKNAME work with Memotoo. This patch a) makes that the default
syncFormat in the Memotoo template and b) automatically sets it
in the "addressbook" source of all migrated configs which sync
with Memotoo (detected based on the syncURL).
The automatic config rewriting is possible because users upgrading
from the previous stable SyncEvolution release need to migrate their
configs anyway. Normally SyncEvolution does not modify configs like
this.
The reason for this change is that MemoToo works better
when vCard 3.0 is announced as the preferred format by
the client. Only then does it support the NICKNAME property,
for example.
The reason for keeping vCard 2.1 as the preferred format
where legacy servers. Not sure whether that still is an
issue. If it is, then these servers must have their
"syncFormat" property set explicitly in their templates
and existing configs.
Asking for .well-known wasn't leading to a redirect with Yahoo.
The RFC recommends trying the root path instead, which happens
to work.
A slight tweak was necessary to handle Yahoo's returning of
"/dav" properties when "/" is queried. With that it is possible
to unify the caldav/carddav code paths and remove the hard-coded
Yahoo paths.
The automatic service discovery was incomplete resp. faulty. Changes
necessary to make it work with Apple Calendar Server (both CalDAV and
CardDAV):
- properly detect redirects in PROPFIND
- send credentials as Basic authorization for PROPFIND, to
get "current-user-principal" properly set by server (<unauthenticated>
is returned by Apple Calendar Server otherwise)
- use the "current-user-principal" href to find calendar/addressbook-home-set
- extracting that href must be more intelligent to cope with
<DAV:href xmlns=...>
- avoid shared collections (we want the user's personal collection at this time)
Sending invalid credentials over HTTP was considered, but fails with
Apple Calendar Server if only Basic authentication is enabled, without
causing the server to send a challenge. Falling back to sending valid
credenticals would make the code more complex for a sitation which is
already insecure (plain text transmission of PIM data!) and thus
wasn't implemented.
See also:
http://tools.ietf.org/html/rfc4918#appendix-Ehttp://lists.w3.org/Archives/Public/w3c-dist-auth/2005OctDec/0243.htmlhttp://thread.gmane.org/gmane.comp.web.webdav.neon.general/717/focus=719
ne_request_create() copies all parameters, but due to a bug in neon,
Digest authentication ended up using the original parameter. With
SyncEvolution freeing it before running the request, the ensuing
read-after-use error caused authentication problems with servers using
Digest authentication (Apple Calendar Server) or might even have
crashed.
Patch submitted and accepted upstream, but better work around it by
keeping the affected "method" string valid throughout the lifetime of
the request.
libneon 0.27 doesn't have ne_options2() and
ne_session_system_proxy(). ne_options2() is purely for debugging,
disabled in builds on old distros. ne_session_system_proxy() is needed
to enable the use of the system proxy settings. If a recent enough
libneon.so.27 is found at runtime (= one which has the function), then
it is called.
libical was contained in libecal in Ubuntu Hardy, at least the way how
SyncEvolution is compiled there. Thus use libecal-1.2 when libical is
not found. Also handle the different ical string allocation semantic
via the icalstrdup() wrapper.
Use a more specific error code instead STATUS_FATAL because otherwise
the UI cannot inform the user.
CalDAV sync attempt while offline => DNS SRV fails => STATUS_TRANSPORT_FAILURE.
No sync URL (as with Yahoo) + no domain in user name => DNS SRV impossible => STATUS_UNAUTHORIZED.
Yahoo CardDAV and CalDAV config only works with full domain (needed
for DNS SRV lookup). Add it if not present. If the resulting
username is invalid, we'll get a credentials error, as desired.
The RetryDuration value now defines the time after which the Neon
library is told to give up. Previously some Neon-internal default
was used.
Resending a pending request earlier still needs to be implemented.
As with SyncML over HTTP, a loss of network connectivity can lead
to a half-open TCP connection which the client cannot detect when
relying on TCP alone.
Non-ASCII characters and pictures get lost. X- Extensions can be
uploaded and are shown on the server, but are not returned by it.
These issues are all on the server side and it is unclear how to
work around them.
Therefore this patch disables contact support in the Yahoo! profile.
Can be enabled manually for testing purposes.
SyncEvolution itself is careful to not send a VEVENT with
RECURRENCE-ID unless the parent item is also in the Google Calendar.
If that is done, then the Google CalDAV server reports the item
in a REPORT, but cannot GET or DELETE it (404 error).
Because other clients might get the server into such a state (Android
client was mentioned), we must deal with it nevertheless. Retrieving
such an item uses a REPORT with UID filter as fallback. A
calendar-multiget had the same issue as a single GET. That code is in
the patch, but not enabled.
Deleting has no fallback. The item will simply remain on the server
and no error will be shown to users. Note that the same 404 error
might also occur in case of a concurrent delete by some other client,
so it makes sense to ignore this.
Just to be sure, flush stderr after running a neon operation and give
the OS a chance to process output redirection, if neon prints output
and redirection is active. It is uncertain whether this is really
necessary, but because it is off most of the time, it doesn't hurt
either.
The "error line" debug output from neon was recognized as potential
error by LogRedirect and shown to users at higher log levels. This patch
uses the new LogRedirect::addIgnoreError() to suppress this line.
A GET of a vCard returns data that seems to have gone through an
encoding step for transmission as part of HTML or XML: special
characters are replaced by entity codes. Worse, that step seems to be
done multiple times, leading to:
backslash &#92; single quote ' double quote "
instead of:
backslash \ single quote ' double quote "
Note the double encoding of \ as &#92; - really broken.
This patch works around that by unconditionally and repeatedly
replacing XML entities with the corresponding characters until no
entities are left in the item data. It would be nice to only do that
when such a broken server is detected, but detecting that is hard and
not implemented.
The downside obviously is that entities that the user (for whatever
reason) wants to have in his contact data will be replaced.
Enabled support for Yahoo! Contacts, by adding it to the configuration
created for Yahoo. Sending data to it requires suppressing empty
properties, which it doesn't seem to handle in all cases (empty
X-GENDER, for example).
When the syncURL setting is empty, the username is checked for
a domain part and that is used to find the CalDAV or CardDAV service.
The actual DNS lookup is done by invoking a shell script and wrapping
some common tools (host, dnslookup, adnshost). This approach avoids a)
keeps the feature optional (no hard library dependency) and b) is more
portable across platforms (availability of DNS resolver libraries varies,
sometimes the API for SRV lookup is poor).
No attempt is made to also find the right URL path via a TXT lookup,
because the services which this patch was tested with didn't seem to
support it.
Instead of the TXT lookup, .well-known paths are tried. But even that
didn't work with the services, so the search code in WebDAV still
needs to hard-code some well-known paths and tries them in a (now)
depth-first search. Services with full support for a redirect of these
known paths should work (but untested).
Previously, the credentials were read directly when instantiating
the ButeoBridge. That happened before the SSO patch in msyncd
looked up the real credentials. Now the credentials are retrieved
in startSync(), where they are available.
When backends are compiled as modules, then the SyncSource
ScannedModules instance must be initialized before the backends
are available. This can be achieved by accessing information
obtained from that class via SyncSource::backendsInfo().
Not doing so resulted in "no backend" errors when
trying to init the configs.
This patch reduces the number of header files needed from core
SyncEvolution. TransportException is expected to be in util.h
instead of TransportAgent.h.
Compile flags are taken from SYNCEVOLUTION_CFLAGS/LIBS instead of
hard-cording paths as they are when compiling together with
SyncEvolution.
Check whether configs already exist instead of overwriting them
each time. Allows local modifications of the properties which
are set (useful for debugging) and is more efficient (always
useful).
Interfacing with the Cmdline class turned out to be the easiest
way of implementing the functionality. It is a bit indirect, but
replicating the logic for setting up and updating the config
would have required more code.
Currently the config is updated each time the plugin runs. Should
change that so that an existing config is left unchanged.
Error handling is incomplete.
When parsing data received from the server, we may end up with
X-LIC-ERROR error messages from libical in the internal
representation. This will get added to the server when combining
multiple items, which is undesirable (waste of bandwith, breaks
comparison).
This patch solves this by filtering out such errors.
Google Calendar adds a VALARM when a VEVENT is created for the first
time on the server via CalDAV. This is not what we want, if the VEVENT
had no VALARM, that's how it should be stored.
As a workaround we detect this special case (new VEVENT without
VALARM) and send the original data again. Having to resend with higher
SEQUENCE and LAST-MODIFIED/DTSTAMP values makes this a bit complex and
slow, because we need to get the actual data from the server (cannot
guess what time stamps were assigned).
This leads to an interesting question: if the clock on the local side
is unsynchronized, its LAST-MODIFIED time stamps might end up being
lower than the values on the server, which prevents sending local
updates. There's currently no solution for this in the backend.
Starting a neon Session is slow, in particular when libproxy is
involved (seems to search for proxy servers, which involves a DNS
timeout when none are around). Another delay is opening the TCP
connection and determining which kind of autentication is needed.
For a normal program run there's not much that can be done to speed
this up, but in client-test it helps a lot to reuse an existing
Session instance between different tests. This patch enables that
by keeping one Session instance around and returning that if the new
one has the same parameters.
The Settings pointer is a boost::shared_ptr, but this patch avoids
using it unless the Session is in active use, just in case that a user
wants to merge the settings into a class with a different life time.
It was observed once (but could no longer be reproduced after reducing the
number of requests) that Google Calendar started to reject requests
with valid credentials with a 401 credential error. Presumably that
was a counter-measure for excessive number of requests.
This patch adds a retry mechanism for this particular case, with
exponentially increasing delays before retrying the same request until it
succeeds or one minute has passed. A credential error on the very first
request in a session still fails immediately. This is implemented as
part of the normal neon "get credentials" callback, so the logic
on top of neon is unchanged.
Both CardDAV and CalDAV allow collections inside the collection of
items which are to be synchronized. They may not contain other contact
resp. calendar data and simply can be ignored.
This patch is only relevant for CardDAV; CalDAV uses its own method
for finding calendar items which already ignores non-calendar items.
This patch moves some of the CalDAV-specific code out of WebDAVSource
into CalDAVSource and adds the corresponding CardDAV code in the new
CardDAVSource. It improves searching for the right collection.
A Yahoo Contacts specific problem is that vCard UID and resource name
have to match, with ".vcf" used as suffix in the resource name. The
new createResourceName() and setResourceName() virtual methods ensure
that. They are nops in CalDAV and use simple string manipulation for
vCard.
Another Yahoo Contacts odity is merging of contacts in PUT, without
telling the client. This can be detected by checking the expected
resource name with PROPFIND: the server then replies with the real
resource name. This check is done only if no href is returned in the
response to PUT, so it should only be triggered for Yahoo Contacts.
CardDAV support is still experimental. It was tested with Yahoo
Contacts, using https://carddav.address.yahoo.com/dav/%u as sync
URL. This is different from the CalDAV URL, which currently prevents
using the same @yahoo context for both calendar and contacts.
Working on CardDAV showed that resource name comparisons had issues
due to different encoding of special characters. Added
canonicalization of paths to fix this problem. This also includes
appending the trailing slash for collections
(Neon::URI::normalizePath()).
Note that Buteo only serializes sync sessions based on their type, so
as soon as we allow CardDAV in addition to CalDAV, it might happen
that SyncEvolution is invokved multiple times in parallel => need to
remove global variables first.
If proxy settings are turned off in the config, the default system
proxy settings are used. This depends on ne_session_system_proxy(),
introduced in neon 0.29. The version is checked at configure time. A
fallback if only an older version is found is intentionally not
present yet, the feature is meant to work in all cases.
The username might contain special characters. It must be escaped
before adding it to the URL. Found when the code was used with an
the default "Your SyncML Username" username string.
Yahoo works without any of the Google specific hacks. Because these
have side effects (performance, change data), they shouldn't be used
when talking to other CalDAV servers.
This patch adds a "SyncEvolution" URL parameter with a comma separated
list of flags:
"UpdateHack" = increase SEQUENCE and LAST-MODIFIED
"ChildHack" = replace RECURRENCE-ID in stand-alone child events
This needs to be appended to the syncURL, like this:
syncURL=https://www.google.com/calendar/dav/%u/user/?SyncEvolution=UpdateHack,ChildHack
The reason for using "SyncEvolution" here is that other parameters
might one day be passed through to the underlying server; at the
moment, anything other than "SyncEvolution" causes an error.
They need to specified explicitly. Hard-coding a check for www.google.com
did not feel right.
Content-Type must be set, otherwise Yahoo CalDAV rejects the
items with "Bad Request".
The resource name must match the UID + .ics. It is okay to
escape special characters like @ in the PUT, even though the
server doesn't do the same in PROPFIND.
For change tracking we need a canonical form for DAV luids. Always
unescaping the relative path component is safe. For absolute paths, we
use the value as suggested by the server. There's no risk of mismatch
here, because we never create such a luid ourself.
This patch implements a dynamic search for the "default" calendar
of a user, starting with the user's principal URL:
https://caldav.calendar.yahoo.com/dav/<user>/
https://www.google.com/calendar/dav/<user>/user/
Stops when:
- current path is calendar collection (= contains VEVENTs)
Gives up:
- when running in circles
- nothing else to try out
- tried 10 times
Follows:
- CalDAV calendar-home-set
- collections
This implies that a specific calendar can be selected by using its URL
directly. The search for "default" calendar is a bit random, because
it depends on the naming of calendars on the server: calendars with
lexicographically lower name are tested (and thus found) first.
Without any kind odf <filter>, Yahoo rejected the request with 400
"Bad Request". Now pass a filter for VEVENT without actually
filtering on any property.
Yahoo requests a client certificate as part of setting up the SSL
connection. Without such a certificate, gnutls aborts setting up
the connection.
As a workaround this patch reads "client.p12" in the current
directory and uses that as certificate. Cannot be the final
solution - replace by dynamically generated certificate or use
ne_ssl_provide_clicert(), with empty list of certificates?
When a child is added without the corresponding parent, Google gets
confused (child is added, but cannot be retrieved). This patch turns a
child event into a normal event by replacing RECURRENCE-ID with
X-SYNCEVOLUTION-RECURRENCE-ID. The inverse operation is applied when
importing data back from Google.
Google requires that all items sent to it have a higher SEQUENCE
number than the copy on the server. Calendar applications are allowed
to make changes to an item even when they are not the organizer, in
which case they are *not* meant to increase the SEQUENCE.
This patch works around the issue by increasing the SEQUENCE number
artifically before sending to the CalDAV server (regardless whether
it is Google or something else).
It does that by keeping track of UID and SEQUENCE number in the
cache. The UID is used to find the resource before sending in case that
the caller didn't know, the SEQUENCE to know what the outgoing item
must have.
The sequence number is shared among all sub items. That is an
intentional simplification of the implementation which may or may not
be necessary to work around the issue. If we have to touch the
sequence number, we might as well do it for all sub items.
This patch enhances error handling so that a more specific
TransportStatusException with the right SyncMLStatus inside is
thrown. This is necessary to report back authentication errors.
libical needs to be checked for explicitly, instead of assuming
that it is pulled in via the ecal backend.
The neon package check can be deferred until the backend really
is needed.
Relying on the MapSyncSource and our own caching hides problems in
both. A backup mechanism taking data directly from the CalDAV server
is more reliable.
This patch implements that by running a REPORT and dumping the
returned data. It stores each VCALENDAR as one file, which may contain
multiple VEVENTs with the same UID. synccompare handles that.
The patch refactors the REPORT issuing because the new backupData()
and listAllSubItems() can share most of that code.
Restore is not implemented yet. The start of a sync could be optimized
by adding all backed up items to a revision list and reusing it in
listAllSubItems(), like SyncSourceRevisions does.
This implements logging like in EvolutionCalendarSource: SUMMARY
plus optional LOCATION. If possible, the item is taken from the cache.
A better solution would move the common "work with icalcomponent"
code into a shared part of libsyncevolution... left for later.
Implemented with a single REPORT request, instead of one query for
resource ID/etag pairs followed by one GET per item.
In theory also less data needs to be transmitted, because we only ask
for the minimum number of iCalendar 2.0 properties required at that
point. In practice, Google delivers full iCalendar 2.0 items. Because
we cannot rely on that, we do not populate the cache with these and
have to fetch those anew which are later involved in operations.
Overall this should still be more efficient.
When a single sub item in a merged item is updated, we modify the
calendar and sent the complete item to the server. Google didn't
like that and rejected the update because of "409 - Can only store an
event with a newer DTSTAMP." Found in testLinkedItemsInsertBothUpdateParent.
The solution is to update SEQUENCE and DTSTAMP/LAST-MODIFIED to that of
the recently updated sub item, then Google accepts the update. Updating
just one or the other was not enough.
This change should work even if more than one item is updated, as long
as a) item changes are reported in the right order (oldest change
first) or b) all sub items get updated.
a) is not guaranteed, b) might be. To be tested...
This patch implements the SubSyncSource interface in CalDAVSource,
then instantiates that class as backend for a MapSyncSource which
bridges between the core engine (one VEVENT per item) and CalDAV (all
VEVENTs with same UID in the same item).
The tests must run with a modified ical20.ics.google.tem which
does not contain RECURRENCE-ID, because that is not yet supported.
Also, our complex 1234567890!@#$%^&*()<>@dummy must be avoided
because Google has an issue with it (doesn't find resulting resource):
CLIENT_TEST_SERVER=google CLIENT_TEST_SIMPLE_UID=1 ./client-test Client::Source::caldav_ical20::testImport
This patch fixes some (but not all) issues found in testImport:
- readItem() did not clear the item string before appending
the new item.
- Detection of "item existed already" cannot use the alternative name
returned by the server, because Google does that also for items
which are really new.
- The test registered itself as "ical20", which conflicts with the EDS
backend. Now use "caldav_ical20".
This replaces the WEBDAV_* env variables with access to the
normal per-peer properties, passed into the source via the new
SyncSourceParams context member.
When used on the command line, @foobar context configs are not
enough to use the WebDAV sources. A properly configured peer
context (foo@bar) is needed. There is no explicit check for this,
the resulting error is about not being able to access the empty
URL `' (Neon uses these quotation marks).
Mapping removeItem() to DELETE is straight-forward. Except that CalDAV
will require additional support for removing individual VEVENTs from
a resource that contains more than one VEVENT...
This works for updating and adding an item. Storing a new item under a
different resouce URI is also handled, which can happen with CalDAV
when there is already an item with the same UID.
It would be nice if the server returned a ETag in response to a PUT,
but Google Calendar doesn't do that. The header is checked neverthless
(untested!), with an explicit PROPGET as fallback.
This patch also clarifies LUID (relative if possible, absolute if not)
and revision string handling (without W/ and quotation marks). We
don't care about weak ETags at the moment, Google doesn't use
them. Even if a server did, what would we do with the information that
the item wasn't stored verbatim?
The current implementation is not yet good enough for CalDAV:
- only one VEVENT per UID supported: must merge and split
related VEVENTs into one CalDAV calendar resource
- adding or updating VEVENT without UID probably fails:
must insert new or old UID
Listing items in WebDAV consists of getting the ETag for each URI
directly underneath the collection. Checking for "empty source"
is done by reading all pairs and then checking for non-empty map.
It would be nice to abort the processing once an error is encountered
or once "not empty" has been observed, but Neon does not support that.
Probably for a good reason, because it would render the TCP connection
useless.
This wraps the "simple" Neon propfind method which handles plain
strings as values. For more complex values, the value is an XML
fragment which should better be parsed using the "complex" Neon
propfind method with XML handlers.
The callbacks are defined by boost::function, which allows attaching
additional parameters to each callback. Exceptions must not cross Neon
library calls.
The Neon::Session wraps most of the relevant calls. It is
parameterized by Neon::Settings. It is uncertain where all of these
are meant to come from, because there is no peer configuration in many
cases. Perhaps we can enforce that a WebDAV source may only be created
in a context which has one and exactly one peer config?
The current intermediate solution in WebDAVSourceRegister.cpp grabs
all settings from WEBDAV_* env variables.
Disabling SSL verification and Neon debug logging are implemented.
Opening the source runs a few checks on the URL. Disabling SSL
certificate checking turned out to be necessary, probably because of
the known issue of gnutls not trusting the weak Google certificate
chain.
This patch maps QContactDetails which have no mapping to vCard to
X-SYNCEVO-QTCONTACTS. The value of that extension property is a dump
of the complete QContactDetail. Reading it again later can restore
that detail.
The goal is to make this additional information available to the
Synthesis engine such that it can preserve the additional information.
Some peers might even be able to store them.
One of the features of the Synthesis engine is that it can preserve
properties locally which are not supported by the
peer. "testExtensions" covers that, by updating all items on the
server (via client B) and reimporting them as updates into client A.
The data comparison then is done without the usual "allow server to
drop data" simplifications in synccompare.
This test is not enabled yet. "config.update" needs to be set for it,
but doing so then trips over other changes introduced by servers, like
for example re-encoding photos. Needs some more thoughts and testing...
The QVersitContactHandlerFactory::ProfileBackup profile enables a
vCard flavor which contains all information stored in a QtContact. It
might make sense to use that format for raw access (--import/export).
Not currently enabled though because the format is pretty verbose.
clang 2.8 compiles SyncEvolution + Synthesis faster than g++ 4.4.5
(3:40min instead of 4:10min on my laptop) and produces more useful
error reports. This patch fixes the code so that it compiles cleanly
with clang when using "-Wall -Werror -Wno-unknown-pragmas". Note that
clang 2.6 (Debian Squeeze) goes into an infinite recursion compiling
code using gdbus-cxx-bridge.h and dies eventually with a stack
overflow - can't be used.
Changes necessary for clang:
- eptr pointer referencing ambiguous, use *x.get() instead
- boost::intrusive_ptr* must be defined before code using it
- two-phase template checking requires explicitly specifying
members in base classes
- name clashes with plain C structs (DBusServer, DBusWatch) are
an error and need to be avoided (done with namespaces GDBusCXX and
SyncEvo)
- floats cannot be inline constants
- unused methods in local classes are warned about (left() in SyncML.cpp)
A lot of the code was roled out explicitly. This patch replaces
that with an approach similar to iostreams: remember context, then
work on one parameter at a time.
Because the decision what to do with a certain method parameter
depends on the type of the parameter, that type must be used to pick
between Get/Set classes which skip resp. work on the parameter.
Allows comparison, substraction and addition. Reads monotonic and system
time. This is a better alternative to time(NULL) + time_t, because it
has sub-second resolution and (for timeouts) is resilient against manually
setting system time.
When the glib event loop is left because the D-Bus client has
requested an abort, the LocalTransportAgent should simply return a
"failed" status and let the caller handle the abort. The return code
of write/readMessage() must be able to convey that - extended from
boolean to an enum.
SyncContext did not do that correctly in server mode: the check for
abort must be done before giving up by throwing an exception.
The D-Bus test now checks that the right status is recorded (wasn't
the case earlier).
The final sync status is reported by the Synthesis engine as a
progress event. SyncEvolution ignored that overall status. In most
cases, it ended up using either one of its own errors (like transport
problem) or the status of a specific source as the final status of
sync.
When synchronization was aborted by the user before the first message
(as in an artificial test), then the source did not encounter a
problem and thus no error was set for the session, although it
failed. Fixed by remembering the Synthesis status code.
Local sync inside syncevo-dbus-server must keep the main event
loop running, otherwise the D-Bus server would stop responding
to D-Bus calls while a sync runs.
This is tested as part of new test-dbus.py tests. They set up local
sync between two directories and then test syncing, timeout detection
when using the glib support code, and aborting via the D-Bus API
while a sync runs.
This testing depends on delaying the sync inside the child
process. Setting the SYNCEVOLUTION_LOCAL_CHILD_DELAY env variable
achieves that (might also be useful for interactive debugging). It is
set for those tests which need it via additional function properties,
similar to the older @timeout() mechanism, which now uses the
more general @property function decorator.
LocalTransportAgent needs to wait temporarily for message transfers.
Without other activity, that is done with select(). But inside syncevo-dbus-server,
the main glib event loop must be served while waiting. GLibSelect()
does that by setting up a source which returns a) after the timeout
or b) when the file descriptor is ready for the next read or write.
As a special case, other returns from the event loop must also be reported.
Doing setSourceType() unconditionally unintentionally also removed a
configured syncFormat and forceSyncFormat setting for an existing
configuration. Only setBackend() (as with this patch) is necessary
and valid in all cases (never run a test with the wrong backend!).
If the context "type" was wrong and only the peer "type" was right,
the old config was usable for syncing. It seems that the "Memotoo"
template created such broken configs. After migration, the wrong
backend was selected based on the context's broken "type", thus
breaking syncing.
This patch fixes that problem by overriding the context "type" with
the corresponding values from one peer which has the source enabled.
The rationale is that this peer used to work and should be kept
working.
There's still a chance that different peers had different backends
configured, which is neither checked nor detected at the moment.
The new call is meant to be used after forking: in such a situation,
redirected stdout/stderr output can be read by both the parent and the
child, which leads to a race condition and makes it non-deterministic
where that output will be handled. redoRedirect() closes the child's
connection to the parent's redirection and sets up its own instead.
Timeouts for the SyncML message are detected on the server side.
No resending is possible, nor needed: this is a reliable local
transport after all, not HTTP.
The implementation uses select() and non-blocking sockets to time out
at the right time. While at it, it also sets CLOEXEC to ensure that
the pipes are not accidentally inherited by other child programs.
Instead of allowing users of the API to register a callback
which can choose between aborting and time out, only accept
the timeout duration and always treat that as a timeout.
The advanced functionality wasn't used and the simplification
makes implementing the API easier.
In some cases (aborted sync, source status was set to a remote error),
the child process has more detailed and accurate information about a failure
related to a source. Copy that to the parent's reports.
Patch depends on still having the LocalTransportAgent instance (moved
m_agent.reset()) and updating the source should be reflectd in final
report (moved updateSyncReport()). The later probably also was broken
for the "partial success case".
In contrast to getSyncSourceReport(), findSyncSourceReport() can be
called on a const SyncReport and returns a pointer to an existing
SyncSourceReport or NULL if not present. Useful when
getSyncSourceReport() would unintentionally create the result.
After the removal of the "type" property, listing of sources broke
because the command kind code kept setting that to select a backend.
Fixed by parsing the type and setting "backend" and "databaseFormat"
instead.
SyncEvolution 1.1.1 did not build without this patch on S360, because
the source used an integer typedef which turned out to be incompatible
with size_t, although it should have the same size. Apparently the
compiler distinguishes between "unsigned int" and "unsigned long",
even if both have the same size.
One of them ("We don't know what this device...") was already
translated elsewhere. It might be that it is already translated in the
UI. The other ("Use these settings") was not and thus really showed up
as untranslated.
Running gen-autotools.sh as part of "make distcheck" fails because the
source directory is read-only. It shouldn't be needed and wouldn't
replicate the same autotools files (due to the missing "git describe"
output), so disable running it unless the source directory contains a
.git repo.
"gen-autotools.sh" (invoked by "autogen.sh") is needed to create some
of the autotools .am and .in files. This is necessary to avoid
hard-coding certain lists of dirs (backends) and files (templates).
The generated makefiles updated themselves when these files were
modified, but not when files were added or removed.
This patch fixes that. It also avoids some code duplication. Instead
of adding explicit rules to the makefiles for the file dependencies,
now gen-autotools.sh is run on each "make all" or "make Makefile"
invocation. It modifies the generated files only if their content has
changed. The dependency rules for "Makefile.am" and "configure.in" are
constructed so that make re-checks their time stamps after running the
gen-autotools.sh script.
A nightly test run showed that process() was freeing memory
when called indirectly from restore(). The protection against
recursive processing wasn't working because m_processing wasn't
set by restore(). Added it to that and some other high-level
calls which might end up with a recursive call into process().
The test failed only because the test's own introduction text on
stdout ended up being redirected. Flushing it before running the
test avoids that.
The flushing was added to both the client runner (because some other
tests also accidentally redirected the output, and it should be
made visible right away) and to the "glib" test (because it shouldn't
depend on flushing in the runner).
An incomplete source config might already exist. Better set the source
type in all cases, to ensure that we really run with the right backend
and data format.
Just saying "foo = bar" is enough now. --sync-property and
--source-property are still accepted, in case that "foo" ever becomes
ambiguous (not the case right now).
The meaning of "type" was horribly complex and had effects on the
backend and the peer. It was impossible to specify the sync format to
be used for a specific peer independently of the local backend and its
format, so adding a peer to a context broke the context configuration
(BMC #1023).
This is now fixed by splitting "type" into four independent properties:
- backend = plugin which interfaces with the data
- databaseFormat = data format used inside backend, only relevant for file backend
- syncFormat = data format preferred when talking to peer
- forceSyncFormat = disable format auto-negotiation, use preferred format
With that split, it is now possible to specify the format in which the
file backend stores items independently of the format in which they
are exchanged with the peer.
Old configurations with "type" can still be read. The values specified
inside it are transparently mapped to the new properties. Command line
and D-Bus API users will only see the new properties.
The command line tool still accepts "type" as an alias for the four new
properties. Using that has the same disadvantage as before: it will modify
the context even if only modifying the peer was intended.
The D-Bus API accepts only the new properties. Clients using "type"
must be adapted to the new property names. Clients not using that
continue to run unchanged.
Writing into the configuration requires a migration of the peer config
*and* the context in which it is defined. That is necessary because
the new semantic (independent database format) cannot be stored in the
old format. The migration is handled by rewriting first the context,
then all peers defined inside it.
Other user-visible changes:
- updated help texts
- the canonical "backend" value for the file backend is just "file"
instead of the long "Files in one directory", which is now an alias
(used to be the other way around); done because "type = file"
was expanded to the long name, which was a bit unexpected and showed
how unintuitive the long name is
Internal changes:
- getMimeVersion() is still present, although it hasn't been used
for a long time; FileSyncSource::getMimeVersion() now derives
the version from the supported Mime types, in case that the
function will be needed again in the future
- setSourceType() with string as argument was replaced with one
taking a SourceType instance; to emulate the old behavior if
desired, construct SourceType from an old-style string
- ConfigProperty methods need to be virtual so that derived classes
like SourceBackendConfigProperty can generate content at runtime
(a recent commit broke that feature)
- File templates were stripped down to the essential properties,
with "type" replaced by the per-peer "syncFormat". "type" would
still have been accepted (so it is not necessary to adapt
syncevo-phone-config right away), but has the original
disadvantage of modifying "backend" and "databaseFormat".
First step towards removing "type". The parsing of the type string was
moved into SourceType. m_localType and comments were added to
illustrate how SourceType will be used in the future (not enabled
yet). getSourceTypeString() was removed because it was only needed for
error messages and will no longer be correct once "type" property is
gone.
Because of the shared source property name changes
(evolutionsource/user/password -> database*), mixing old and new
configs inside the same context is not possible. Bumped
CONFIG_CONTEXT_MIN_VERSION and CONFIG_CONTEXT_CUR_VERSION to reflect
that and added the necessary code for migrating a context and all
peers inside it.
This is triggered by a) an automatic config migration in release mode
and b) an explicit --migrate of *any peer inside the context* or of
the context itself.
There was a getContextName(), but no getPeerName(). Added. Required
renaming the existing getPeerName() to getUserPeerName(), with
setUserPeerName() renamed accordingly. The former is the internal
part of the config name, the latter the user-visible chosen name.
The list of read-only properties which can be ignored in SetConfig()
was incomplete. Also need to take into account the ones added for
Bluetooth templates.
When creating a config from scratch, the command line source properties
applying to the default source set in the template were ignored. That's
because createFilters() needs to know about all active sources, otherwise
it won't create the filters for them.
When running a command line failed, an assertion was raised without
ever showing the error output of the failed command line. Now it is
included as message text in the CPPUNIT_ASSERT_MESSAGE() exception.
This config extension makes it possible to send BDAY and REV in
extended format (with hyphens) by including subrule
EXTENDED-DATE-FORMAT. Not done by default.
Calling the parent logger may invoke flushing in LogRedirect. This
needs to be done before writing the log message which causes that
flushing, otherwise the order of lines in the -log.html file is
slightly wrong.
An additional parameter allows the caller to decide whether
LogRedirect writes into stdout (original behavior) or opens
a specific file. Useful for client-test, which can avoid
one layer of loggers that way.
It was already possible to keep known errors out of stdout with
the SYNCEVOLUTION_SUPPRESS_ERRORS env variable. Now the same is
also possible at runtime via addIgnoreError(). This is useful
for backends which know about some of their internal errors that
they might produce indirectly (tools or libs being called, etc.).
If --source-property was set while syncing, the intention was to only
apply (and thus potentially enable) the filter to active sources. If
no sources where specified on the command line, then the code looked
at the config to build that list.
That code must have been broken for a long time, because it had a ==
"disabled" check instead of != "disabled", thus enabling exactly the
wrong sources. Fixed.
Probably not relevant for old releases because it only applied to cases
where no source filter was set. That additional shortcut check was
removed when rewriting the property filters, so now it was found.
Now the @context or @peer@context suffix in sync and source properties
on the command line can be used to configure both sides of the sync
differently.
The new format of the property name in --sync-property is:
<name>[@<context>|@<peer>@<context>]
--source-property also allows a source name:
[<source>/]<name>[@<context>|@<peer>@<context>]
This allows to set source properties differently for different
sources in the same command line invocation. The @<context> or
@<peer>@<context> will be used to set properties differently for
main and target context in a local sync (not used yet).
The advantage of this grammar is that a string can be split purely based
on the syntax in PropertySpecifier::StringToPropSpec().
The patch itself is based on the idea of first collecting all of these
config property filters in a new case-insensitive hash structure,
FullProps in ConfigFilter.cpp/h, as part of parsing command line
parameters.
Then once specific filters for sync or sources are needed, they are
generated from FullProps by collecting all that apply, starting with
the ones with lowest priority and overwriting them with more important
(= more specific) ones. This also covers additional filters, like the
shared properties of the target context when printing a template.
Currently FullProps may contain arbitrary source and config
names. Typos are not detected, which is both hard to implement (which
names and configs are valid in the current invocation?) and also
forces users to be very specific (can't apply one set of filters to
different configs) - this is the same conflict of interest as in
"configure", which allows unknown --enable/disable parameters because
they might be relevant in a sub-configure script.
SyncConfig itself still only stores the filters which apply to it, not
the full set of overrides that the Cmdline has in its m_props. The
advantage is that the API remains the same (no change needed or done
in the syncevo-dbus-server). The disadvantage is that in a local
sync, no information is available about the properties applying to the
target context - probably needs to change.
The traditional normal form of a peer config in the @default context
did not contain that suffix. This is more readable for the user,
but has disadvantages:
- name is only unambigious if that config exists, otherwise
the config name "foo" might also be mapped to "foo@someothercontext"
- if both a context and a config name are allowed, a config
can only be detected if it has an explicit @ sign in its name
Therefore this patch adds an alternative normal form which always
has a context specifier.
Renaming "evolutionsource/user/password" to "database", "databaseUser",
"databasePassword" affects D-Bus API clients which depend on the old
names. No known clients use these properties, so making the change
is acceptable, but some caution is needed nevertheless.
This patch adds a validation step to SetConfig() that detects unknown
properties and top-level entries. "" for sync properties and
"source/<name>" for source properties are currently accepted,
everything else is rejected.
Property values are also checked in advance, compared to checking them
later as part of copying them. The exception raised now for invalid
values is a more specific org.syncevolution.InvalidCall.
While adding the validation step it was noticed that the description
of temporary settings in SetConfig() was incorrect: temporary changes
*do* affect the result of GetConfig(). Documentation fixed.
Temporary settings can be reset now with SetConfig(update=False,
temporary=True), something which wasn't allowed earlier.
The SetConfig() call did not prepare the config for writing. Therefore
the obligatory version check was missing for old configs and versions
weren't set in new configs.
This renames the legacy "evolutionsource/user/password" properties to
"database" and "databaseUser/Password". The justification is that
these properties are also used outside of the Evolution backends.
The change can be done now because the on-disk format has to be
changed anyway in a backward-incompatible way.
The CmdlineTest::testOldConfigure test covers reading an old-style
config.
The goal is to allow multiple names for properties. This will be used
to replace legacy names like "evolutionsource", but it could also be
used to allow abbreviations on the command line.
First the patch replaces the single ConfigProperty::m_name with a list
of names, of which the first one is the current name of the
property.
The name that is to be used depends on the node which is to be
manipulated: if it already has a property under an alias, that alias
is read and also written, so existing configuration continue to use
the old config name (avoids the problem of having to remove the old
name and insert the new one at the same spot in a .ini file). Old
configs continue to parse okay. Writing into node which has none of
the names set, as in migrating into a fresh config, uses the new
names.
Most of the patch deals with the removal of a single, unique name,
either by picking the name so that it matches a node, using the
default name or simply hard-coding a property name ("sync").
The command line accepts all aliases, then uses the main name for
overriding configuration options.
SyncConfig inherited "const char *" from the Funambol C++ API and some
other methods used the same approach for efficient access to plain
strings. However, this has the disadvantage that dynamically generated
strings cannot be returned. SyncConfig had to use an awkward
workaround with a local string cache.
This patch converts most of that code to a normal std::string return
value and removes the string cache.
Out-of-tree backends must be adapted, otherwise they won't compile.
SyncEvolution < 1.2 considered all existing configurations
user-visible in UIs, therefore the ConsumerReady flags was not always
set (done by sync-ui, not done by command line).
In order to keep such old configs visible even when the updated
sync-ui (and soon Genesis) check the flag, the syncevo-dbus-server
transparently adds the flag when it finds a config written by
SyncEvolution < 1.2.
The Cmdline.cpp migration code does the same when rewriting such and
old config for read/write use.
KCalExtended does not support the operation needed to obtain changes since
the last sync. That is because it relies on the anchor, which is inaccessible
in the client binfile and only available in a real beginSync().
The command line output contained no line for the KCalExtended backend
because it got no information. It is better to explicitly mark unknown
entries with -1.
endSync() creates an anchor wich represents UTC system
time. beginSync() must be explicit about the UTC time zone when
restoring KDateTime, otherwise it'll end up using local time
and produce the wrong time for comparison against the DB.
Also added some debug code to record change tracking results.
Previously, existing configs did not need the ConsumerReady flag to be
shown to users in the UI. This was (intentionally?) inconsistent with
templates, perhaps reasoning that users configuring unsupported peers
shouldn't have to bother with setting the flag.
This patch makes templates and configs behave alike: they are ignored
by the GTK sync-ui unless the ConsumerReady flag is explicitly set to
1. The reason is that removing that flag is used to hide backup copies
(*.old) of migrated configs.
It might also be a worthwhile feature by itself: I have a lot of
experimental configs, but only very few that actually need to show up
in the GUI.
Users who configured peers without that flag must add it manually if
the sync UI is meant to show them.
The "*.old" configs get their ConsumerReady flag removed so that UIs
can decide whether they still need to be shown to the user. The
command line will show them, the GTK sync-ui is meant to hide them
(but doesn't to it yet).
Changed SyncConfig::getConfigs() such that foo@bar comes before foo.old@bar.
With a pure lexical sort (as before), '@' > '.' and thus the config
with the longer string was found first by syncevo-dbus-server after
migrating foo@bar.
Migrating a config in a context wasn't necessary so far, but also
wasn't explicitly forbidden or prevented. It did not work correctly
when the config was identified without explicit context on the command
line, because that short-hand was only mapped correctly before
renaming the config, but not anymore afterward. As a result, the
config was moved into the @default context or possibly some other
context which had a peer with the same name.
Cleaned up the m_server parsing (duplicate code?!) and fixed this
problem by setting m_server to the fully-qualified configuration name,
including a @default suffix. This is user-visible in error messages,
which actually may be a good thing.
The goal is to release SyncEvolution 1.2 with a more recent
libsynthesis, which rewrites its on-disk binfiles such that
older releases are forced to do slow syncs.
This versioning change forces old configs to be migrated. This
affects both the old .sync4j style config (as represented by the
testOldConfigure test) as well as unshared
~/.config/syncevolution/<peer> configs.
gen-autotools.sh now detects pre-releases: have "99" in the version or
the source isn't cleanly tagged. "syncevolution --version" includes
"(pre-release)" if in pre-release mode, otherwise just the version (as
before).
This patch adds version numbers to the on-disk config and files. The
goals are:
1. refuse to use a config which was written by a SyncEvolution release
in a format that is too recent to be handled correctly by the
current release ("detect invalid downgrades")
2. refuse to modify a config in such a way that the previous release
using that config will not be able to use it anymore ("prevent
unintentional upgrade of config")
In the first case the user is told:
SyncEvolution <version> is too old to read configuration '<config>',
please upgrade SyncEvolution.
In the second case the user is told to migrate the configuration manually:
Proceeding would modify config '<config>' such that the
previous SyncEvolution release will not be able to use it.
Stopping now. Please explicitly acknowledge this step by
running the following command on the command line:
syncevolution --migrate '<config>'
These are printed as [ERROR] messages on the command line and shown as
error codes 22004 resp. 22005 in front-ends which don't know about these
scenarios.
The first problem should be rare, so presenting a nicer error messages
in UIs is not essential. But the second problem will occur. The plan
(not implemented yet) is to automatically migrate in stable releases
without asking.
"--migrate @<context name>" would have happily migrated the non-peer
properties, but would not copy the peers inside that context. Wasn't
needed so far, but shouldn't have been allowed either.
Reloading properties of a node or a complete tree is necessary
when some other operation modified the background storage. One
use cases is rewriting .ini files during a migration while the
a SyncConfig has an active config tree for that config.
This is not a general solution to the problem of detecting
changes to the background store. Reloading must be triggered by
the code which made the change there.
Between MeeGo 1.1 and 1.2, the ConnMan D-Bus API has changed from
org.moblin to net.connman as identifier for the service.
This patch adapts to that change with a string replace. No attempt
is made to stay compatible with previous ConnMan releases because
it is expected that the main target, MeeGo 1.2, will release with
updated SyncEvolution and ConnMan versions.
Compared to the existing Resend tests, the new ResendProxy tests have
the advantage that they cover one additional error case: loss of
connection before the server has a chance to send the full reply. This
happened to be one case that the syncevo-http-server wasn't handling
properly.
This failure cannot be simulated inside client-test because the
TransportAgent API doesn't provide enough control. The implementation
uses a twisted HTTP proxy which intercepts data transfer at three
different points (before sending, after sending and before receiving,
after receiving) and drops the client connection. The script listens
on four different ports, numbered consecutively, with the first one
not introducing errors, and all others representing one failure.
The test must set and unset proxy information in the underlying transport;
this patch makes that possible in SoupTransportAgent::setProxy().
To use the new tests, run "proxy.py <port>", then set
CLIENT_TEST_RESEND_PROXY=<port>.
The version check for the 0.7.0 API change incorrectly switched to the
new API with libnotify 0.5.2 and thus fails to compile, because 0.5.2
only has NOTIFY_CHECK_VERSION(), but not the new API.
A more complex version check is part of this patch.
When sending a calendar event or task to a peer, the DESCRIPTION is now
only set to the SUMMARY in cases where the peer is known to not support
the SUMMARY property. This is an improvement over the original logic
which always duplicated the text.
When receiving such an item, a DESCRIPTION which is identical to the
SUMMARY is removed. This restores the original logic from upstream
Synthesis configs. Since commit
ca22c2d4c4, SyncEvolution avoided empty
DESCRIPTION fields by setting the DESCRIPTION to the SUMMARY because
libical was unhappy about it; but that was already successfully
avoided by another change in that commit, so it makes sense to avoid
storing duplicate data, should the peer send it (from a previous sync
or because it always sends the same text in both properties).
Conflicts:
configure-pre.in
src/syncevo/SyncContext.cpp
src/syncevo/configs/datatypes/10calendar-fieldlist.xml
test/ClientTest.cpp
test/client-test-main.cpp
test/synccompare.pl
Conflicts all resolved by using the code from "master" branch. Caused
by similar changes on release branch.
Running a sync with EDS CalDAV backend as storage to be synchronized
ignored changes made before the sync. Running the sync again then found
the changes.
It seems that the EDS CalDAV backend triggers a refresh from the
CalDAV server after the client is done, instead of doing that in
e_cal_open().
As a workaround, this patch destroys the newly opened calendar
instance and opens it again. This is done for *all* backends, because
it is cheap for local storage and other remote storage backends might
have the same issue.
The ScheduleWorld service shut down end of November 2010.
Removing it completely from SyncEvolution is a bit
intrusive (it is used as reference configuration and in
unit tests), so this patch only makes it invisible to
sync-ui users and replaces it in the examples with
Memotoo.
