apple, googlecalendar and yahoo include now also Client::Sync
tests. For "apple", this is the full set of tests. For the
others, only testItems is run.
The "source-config" peer in the target context used to be required for
local sync. Now it is optional. The reasoning is:
- For WebDAV, copying username/password from the main config may
already be enough to find the server.
- For local file sources, no sync properties ared needed
at all.
In both cases, "source-config" is used if available. That can be useful
to set loglevel separately, for example.
For SyncEvolution<->SyncEvolution synchronization there should be
no need to set the uri property of each source explicitly, because
typically it is the same as the source name.
With this patch, the source name is used as fallback. Because some
code needs to check whether the URI is set, getURI() returns the
setting directly (as before) whereas getURINonEmpty() always returns a
non-empty result with the name as fallback.
Specifying an invalid template name continued with an empty template
if enough properties were specified. That's not correct and leads
to unexpected results:
- peerIsClient might have been part of the desired template.
- Set of sources to be created is defined by template.
Now an error is printed for this case.
Printing of templates was changed to use underscores instead of
spaces, but retrieving a template was not. Turned out that
it doesn't compare against the list that it builds up, but instead
does a matching against the fingerprint.
Now the fingerprint comparison treats spaces like underscores,
so that "SyncEvolution_Client" matches against the "SyncEvolution Client"
fingerprint. Better for backwards compatibility, too.
The device templates contained spaces ("Nokia N900", for example),
which was hard to use on the command line and in scripts. Now
an underscore is used instead of spaces.
This is a better fix for the webDAVCredentialsOkay property
registration: always do it, regardless whether the backend is enabled,
because then the config migration code will always know about it.
Also fixed automated testing to include that property.
Logging calls might end up calling functions which modify errno.
Seen once where logging reported a "pipe error" and the following
exception a "resource temporarily unavailable".
testOneWayFromClient was still failing in server mode because of a
similar issue as with slow sync as fallback for refresh-from-server:
the client can ask for a two-way sync and then withhold all of its own
changes to achieve the same effect.
Detect this based on the requested sync mode.
Only the "evolution" test allowed selecting which client-test
tests to run. Now this code is in SyncEvolutionTest class itself.
The parameter to it with the default test list was turned into
a simple string, to match the value of the --enable parameter.
This is needed for local CalDAV/CardDAV tests to succeed.
With http_proxy set, all traffic would be attempted via
these proxies, which doesn't work.
All tests which need proxies have to be configured such
that proxyHost/useProxy is set correctly in the server configs.
recv() may return -1 and EAGAIN/EWOULDBLOCK, according to the man
page. That was treated as a fatal error instead of retrying, which
was the original intention.
For writev() it is less clear whether EAGAIN/EWOULDBLOCK need to be
handled. Let's catch them anyway.
The following code crashed if m_length was read after executing
the release() calls:
realloc(buffer.m_message.release(), buffer.m_message->m_length)
That's because release() resets the m_message pointer. It seems that
clang orders execution like that whereas gcc doesn't.
Some of our extensions are optional, for example X-EVOLUTION-UI-SLOT
(because Nokia phones had problems with that). CalDAV/CardDAV servers
are meant to preserve all extensions, so let's send them everything we
have.
For calendar, this is not yet relevant in practice (rule doesn't do
anything).
Apple Calendar server + CardDAV really preserves the slot parameter.
Yahoo CardDAV does not, but at least accepts it.
Apple Calendar server does not seem to support unlimited
recurrences. It adds a UNTIL clause. Ignore that by removing
the test case from the Apple specific test data.
Due to a bug in libical, the Apple Calendar server's data
is nor parse correctly and ORGANIZER gets lost. This patch
temporarily removes the property from the test data to get
the testItem test to pass.
When deleting items on the server due to "refresh-from-client" sync,
we want to have these items counted in the statistics, just as on
a client in a "refresh-from-server".
The code doing this had to be updated to distinguish between client and
server mode.
The main bug fix is for partial writes: the base address wasn't getting
updated, so the same data would have been sent multiple times. Not sure
whether this really occurred in practice.
fcntl() was called for random file descriptors due to a buffer
overflow. If a valid file descriptor was hit, the real user of it
might have gotten confused.
writev() was passed unitialized memory. Okay in this case (hole in struct),
but ugly in valgrind and easily fixed with memset().
Creating test data in SyncTests::doVarSizes() took a long time
because it created a TestingSyncSource multiple times inside
a loop. Each destruction of that instance caused a 5 second delay,
necessary to ensure correct time-based change tracking.
Adding a second version of insertManyItems() which uses a single
instance of TestingSyncSource created by doVarSizes() itself
makes it possible to sleep only once.
This patch relies on libsynthesis reporting a stable sync mode
of each source in the PEV_ALERTED progress event. This was added
recently to libsynthesis in server mode.
This patch deals with a client that falls back to "delete data + slow
sync" as a way of doing a refresh-from-server (as Synthesis engine
itself does) at least in one case: server-alerted syncs know the originally
requested sync mode and assume that a slow sync is the intended mode,
which then gets translated back into refresh-from-server.
A session with such a sync initiated by a client is still recorded as
"slow sync".
An empty "database" property for a virtual source lead to a useless
error about instantiating a source with an empty name. Now the error
is reported for the virtual source and includes the "database" value.
A sanity check that the source really had exactly two sub-sources
was missing. Added.
The file backend refuses to create a source if backendFormat is unset.
This was reported as "<source>: backend 'file' not supported",
which was confusing because the backend was in fact supported.
Now the error message is "<source>: backend 'file' not supported or
not fully configured (format '<format>')".
Exceptions encountered in the child process were printed as [ERROR],
but not reported to the parent process and thus did not end up in the
final sync report.
Now they are, which makes finding the root cause of a failed local
sync a bit easier.
The error message "@<context>/<source>: source not configured" becames
confusing when <source> is empty. Now "missing URI for one of the
sources" is printed instead.
The command line instantiated sources with the name "list" (cut-and-paste
error?) instead of using the proper name. As a result, error messages
created by these sources didn't refer back to the configuration. Now
the real name is used.
The Akonadi source reused the EDS test names (vcard30, ical20, ...).
This has the effect that the configuration of these tests was
overwritten each time client-test was started. Each backend must
use its own names here.
For Akonadi, kde_events/tasks/memos were chosen. "calendar" was avoided
because some people include tasks in that category.
This re-adds the code from a3d7412972
which was temporarily removed in revert commit
4ff3777c70.
This time, the changes are done in such a way that other, non-KDE
backends are not affected. The properties fall into three categories:
1. The KDE extension maps to a field which has already an property mapping
that is understood by SyncML peers (example: X-messaging/xmpp-All KDE
property maps to X-JABBER). In these cases, the external
property is not used with KDE (empty property with rule="KDE").
A second property entry exclusively for KDE (rule="KDE") defines
the mapping to the KDE property.
2. The KDE extension has no corresponding mapping (X-KADDRESSBOOK-CRYPTO*,
X-messaging/irc-All for IRC chat handle). In these cases the KDE
property name is used also with SyncML peers.
3. An Evolution extension has no corresponding extension in KDE (X-EVOLUTION-VIDEO-URL).
Here the Evolution extension is passed to KDE.
Whether KDE extensions like X-messaging/irc-All should be exposed as
they are to other peers is uncertain. It might be more natural to follow
the X-AIM/JABBER/MSM/... pattern and introduce X-IRC. Or follow RFC 4770,
which defines the IMPP property... at the end of the day, this depends on
the peers which we want to send these extensions to.
Same rationale as with KWallet: don't surprise traditional
users by enableing it without their consent. Also the automatism
enabled it on Ubuntu Hardy where it didn't compile.
Nightly testing shows that the changes in
a3d7412972 affect other backends. This
patch removes the changes temporarily to have more time for
investigating that.
The OPTIONS request doesn't use the retry mechanismn and thus
failed hard when Google started the 401 throttling. Now such
any error is simply ignored.
The outer retry loops seems to handle the 401 Google throttling
fine, no need anymore for the identical code which was invoked
by Neon inside the credentials callback.
The REPORT request which retrieved calendar data was only sent
once. If it failed, creating the backup failed. Now this request
is tried a few times similar to the other requests. Before each
retry, the potentially incomplete backup is removed.
The retry mechanism for Google's request throttling with incorrect
401 responses broke when introducing forced authentication, because
Neon never asked for credentials in that case, which bypassed the
retry logic.
Moved the 401 handling into the outer retry loop of run() + retry.
Cleaned up operation handling: the deadline is now set once
before entering the retry logic. This is also a convenient place
to record the name of the upcoming operation (used for debugging
and error messages).
Made the information that credentials were valid before persistent as
"WebDAVCredentialsOkay" in the .internal.ini file of the
"source-config" peer. This turned out to be necessary for unit testing,
because running the next test had no knowledge that retrying is
okay and gave up too early.
The inner retry loop in the credentials callback is probably obsolete
now; will be removed.
The intention always was that backends may add their own
properties. For sync properties this seemed less relevant,
so sources only got read access to it.
But the WebDAV backends run with their own "source-config" context
and now need read/write access to store information that is
shared between all sources.
Therefore this patch removes the "const" qualifier from SyncConfig.
The logic for detecting a KDE session was reversed: if KDE_FULL_SESSION
is *not* set, then the session is not KDE. As a result, KWallet was
used inside a GNOME session instead of GNOME keyring if both were
enabled.
The problem with eGroupware is that it doesn't have one fixed syncURL.
Users of the template must remember to change it. To make this more
obvious, it is set to http://set.your.domain.here/rpc.php
The failure to fix the URL is leading to fairly obvious network
errors.
Various people have reported success with eGroupware, the latest
one here: http://www.ruinelli.ch/how-to-sync-egroupware-with-a-tablet-n900-with-syncevolution
Therefore it makes sense to also mark this template as "consumerReady"
and have it shown in the sync UI.
When Google calendar has temporary problems, it redirects to
an HTML error page. That defeated our "retry on error" logic.
Now redirects are checked inside Session::check(), with a special
case in place for this Google redirect.
When capturing lines from stdout or stderr, strip the trailing newline
before calling message() because that introduces an additional newline
by itself.
It is useful to know how long certain operations take. Instead of
adding random time stamps at the log calls, this should better be
done consistently for all lines.
In a normal sync log, such a time stamp is added by the Synthesis
logging code. For stdout logging, this patch adds it if the log
level is >= DEBUG. That way it won't distract normal users but will
be available automatically when producing a log for developers.
Because a full time dump can be long and because the delta between
events is more interesting, only relative time since the start of
logging is printed.
This is motivated by the observation that the Yahoo server becomes
unresponsive (various 5xx errors) quite often, which used to abort
the sync. Now there are better chances that it'll complete eventually,
although the root cause (server not responsive enough) remains.
PROPFIND, PUT and GET are all tried again. For a failed PUT this is
problematic because it is simply not known whether the command was
already executed. A check for "item does not exist" therefore can only
be done on the first attempt. Likewise, any future "eTag matches"
check (not yet implemented) will be also impossible.
Because of the retry, a PUT is no longer marked as not
idem-potent. The advantage is that an existing connection will be
reused for it.
The implementation is based around the idea that the higher levels in
the stack define a deadline until which the operation must succeed,
and then the lower levels (individual PROPFIND/PUT/GET) retry inside
that allowed time interval. That ensures that the initial service
discovery never takes longer that the configured timeout, which
wouldn't be the case if the timeout was applied to every single
PROPFIND inside the semantic operation.
