The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google-calendar" will pick the "Google_Calendar"
template automatically because it knows that the intention is to
configure the target side of a local sync. It does not know that when
using some other name for the config, in which case the template (if
needed) must be specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
The first progress signal gets emitted after sleeping for 10 seconds
at the start of the sync and then killing syncevo-dbus-server races
with completing the sync. What we want is to kill during the 10 second
wait, so we better wait for the debug output directly before it and
then kill directly.
Funambol turned of the URL redirect from my.funambol.com to
onemedia.com. The Funambol template now uses the current URL. Users
with existing Funambol configs must updated the syncURL property
manually to https://onemediahub.com/sync
Kudos to Daniel Clement for reporting the change.
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.
In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.
Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.
During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:
X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE
Any IMPP property declared as X-SERVICE-TYPE=AIM will get
mapped to X-AIM. Same for others. Some IMPP service types
have no known X- property extension; they are stored in
EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
(for example, GaduGadu and Groupwise) are stored with
X-SERVICE-TYPE values chosen by SyncEvolution so that
Google CardDAV preserves them (GroupWise with mixed case
got translated by Google into Groupwise, so the latter is used).
Google always sends an X-ABLabel:Other for IMPP. This is ignored
because the service type overrides it.
The value itself also gets transformed during the mapping. IMPP uses
an URI as value, with a chat protocol (like "aim" or "xmpp") and
some protocol specific identifier. For each X- extension the
protocol is determined by the property name and the value is the
protocol specific identifier without URL encoding.
X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel
The mapping is based on the X-ABLabel property attached to
the X-ABRELATEDNAMES property. This depends on the English
words "Spouse", "Manager", "Assistant" that Google CardDAV
and Apple devices seem to use regardless of the configured
language.
As with IMPP, only the subset of related names which have
a corresponding X- property extension get mapped. The rest
is stored in EDS using the X-ABRELATEDNAMES property.
X-ANNIVERSARY <-> X-ABDATE
Same here, with X-ABLabel:Anniversary as the special case
which gets mapped.
X-ABLabel parameter <-> property
CardDAV vCards have labels attached to arbitrary other properties
(TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
item1.X-ABDATE:2010-01-01
item1.X-ABLabel:Anniversary
The advantage is that property values can contain arbitrary
characters, including line breaks and double quotation marks,
which is not possible in property parameters.
Neither EDS nor KDE (judging from the lack of responses on the
KDE-PIM mailing list) support custom labels. SyncEvolution could
have used grouping as it is done in CardDAV, but grouping is not
used much (not at all?) by the UIs working with the vCards in EDS
and KDE. It seemed easier to use a new X-ABLabel parameter.
Characters which cannot be stored in a parameter get converted
(double space to single space, line break to space, etc.) during
syncing. In practice, these characters don't appear in X-ABLabel
properties anyway because neither Apple nor Google UIs allow entering
them for custom labels.
The "Other" label is used by Google even in case where it adds no
information. For example, all XMPP properties have an associated
X-ABLabel=Other although the Web UI does not provide a means to edit
or show such a label. Editing the text before the value in the UI
changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
other fields.
Therefore the "Other" label is ignored by removing it during syncing.
X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.
CardDAV needs to use test data with the new CardDAV vCard flavor.
Most CardDAV servers can store EDS vCards and thus
Client::Source::*::testImport passed (with minor tweaks in
synccompare) when using the default eds_contact.vcf, but
Client::Sync::*::testItems fails when comparing synced data with test
cases when the synced data uses the native format and the test cases
are still the ones from EDS.
A libsynthesis with URLENCODE/DECODE() and sharedfield parameter for
<property> is needed.
The test assumed that it can rename the main syncevo-dbus-server
executable to trigger the file watch mechanism. That's not correct:
- It might be the system's /usr/libexec/syncevo-dbus-server,
which a normal user cannot rename.
- The binary might be also active in some other, parallel tests.
Renaming it interferes with those other tests.
The latter happened in the nightly testing: HTTP server tests with
a long-running syncevo-dbus-server failed because the daemon terminated
during the tests.
This adds GetPeerStatus() and "progress" events.
To detect DB changes as they happen, the SyncSource operations are
monitored. Upon entry, a counter gets increased and transmitted
through to the PIM manager in syncevo-dbus-server using extended
SourceProgress structs (only visible internally - public API must not
be extended!). This will count operations which fail and count those
twice which get resumed, so the counts will be too high
occasionally. That is in line with the API definition; because it is
not exact, the API only exposes a "modified" flag.
Progress is reported based on the "item received" Synthesis event and
the total item count. A modified libsynthesis is needed where the
SyncML binfile client on the target side of the local sync actually
sends the total item count (via NumberOfChanges). This cannot be done
yet right at the start of the sync, only the second SyncML message
will have it. That is acceptable, because completion is reached very
quickly anyway for syncs involving only one message.
At the moment, SyncContext::displaySourceProgress() holds back "item
received" events until a different event needs to be emitted. Progress
reporting might get more fine-grained when adding allowing held back
events to be emitted at a fixed rate, every 0.1s. This is not done yet
because it seems to work well enough already.
For testing and demonstration purposes, sync.py gets command line
arguments for setting progress frequency and showing progress either
via listening to signals or polling.
In testAutoSyncNoNetworkManager, syncs keep happening while the test
terminates. The processes get killed correctly, but that ocassionally triggers
notifictions which the post-test D-Bus log checking treats as test failures.
To avoid this, truncate the D-Bus log before shutting down. Then all extra
events are available for debugging, but will not be seen by the check code.
Check for an already running instance. Otherwise the new one
will fail to start up, which will be hard to diagnose. Happened
recently when a test was broken and failed to kill its daemon
instance.
When spawning the dlt daemon under "unbuffer", a lingering tclsh process
exists before the test starts and the test ended up waiting for that to
stop.
We need to be more specific with identifying sync processes: only those
processes started as part of syncing matter.
The test randomly failed under load because the client's GetVersion
call did not make it to the server in time. It seems to work better
with a smaller delay.
The test relied on fixed timeouts for a) killing syncevo-dbus-server
and b) completion of the sync. The second timeout sometimes occurred
to soon, causing the testNoParent itself to fail and also the following
test (because log files from testNoParent were not removed).
Now be more careful about really killing in the middle of the sync (thanks
to delaying the sync and not the helper startup) and really wait for process
termination (with getChildren() and os.kill() checking).
Picking a port dynamically was broken: if 9999 was in use, the test script
connected to the wrong process and then incorrectly continued, instead of
detecting that syncevo-http-server failed. Use the auto-alloc feature
of syncevo-http-server (port == 0) instead and get the actual port
from the output.
Now that we redirect that output into a file, it makes sense to also include
that log in test failure reports.
Only the "client-test" executable and its test data were installed so far, in
/usr/bin resp. the doc dir. Now also test-dbus.py and testpim.py plus their
data files get installed, using a new /usr/lib/syncevolution/test directory
for all test files and commands.
"runtests.py --prebuilt" can take one or more .deb files including these
tests, will install them as the "compile" operation and then will use the
installed SyncEvolution, without overriding any paths. This relies on having
suitable rights for "dpkg" and /usr/lib/syncevolution/test; the way how that
is done in the nightly testing is via chroots where the entire file system
is writable by the normal test user.
Normal users without such write access to /usr/lib/syncevolution/test need
to copy that directory into their home directory first.
This adds hooks into doAutoSyncLocalSuccess() which are used by
testAutoSyncLocalMultiple() to start a session while waiting for auto-sync,
access the config which will get auto-synced, then continue waiting.
While at it, also measure the actual auto sync interval and roughly
compare against the configured one.
The actual timeout duration was too small by a factor of 1000 (ms
versus s mismatch). Did not affect existing tests which just had their
(anyway small) timeouts triggered too soon.
When TEST_DBUS_DLT is set, start the DLT daemon and tell syncevo-dbus-server
to use it. This relies on dlt-daemon being in the PATH, which requires
some changes in setting that env variable via runtests.py.
unbuffer from expect-dev (on Debian) is needed to get dlt-daemon to
print each message immediately.
This only adds the infrastructure for testing DLT logging. The actual
tests still need to be added.
The hyphen introduced with the alternate EDS backends wasn't expected
by the regex. Now simply assume that backend names are non-whitespace
characters.
In testSyncOutput, valgrind really chewed more than one minute
on syncevo-dbus-server during process shutdown. We need to give
it enough time, otherwise the test failed with "process had to be
killed".
Diagnostic Log and Trace (DLT) manages a sequence of log messages,
with remote controllable level of detail. SyncEvolution optionally
(can be chosen at compile time and again at runtime) uses DLT instead
of its own syncevolution-log.html files.
The backends must be compiled differently for EDS < 3.6 (using the old
API before EBookClient, ECalClient and ESource, ideally in
compatibility mode) and for EDS >= 3.6 (using the new API, with hard
linking against libebook-1.2.so.14, libecal-1.2.so.15,
libedataserver-1.2.so.17).
With these changes, a SyncEvolution binary compiled for the older EDS
API will be able to load and use backends compiled against the current
one. Both backends can be installed in the same
lib/syncevolution/backends dir under different names. The newer ones
must have an additional -<version> appendix before the .so suffix.
Then loading will attempt to use those backends first and if
successful, skip the older ones. This is necessary because loading
both into the same process will fail due to symbol clashes. If
unsuccessful (as on systems with only the older EDS), the dynamic
linker will fail and SyncEvolution proceeds with the backends for the
old API.
Packaging of binaries as .dev/.rpm/.tar.gz includes these additional
backends if given in the EXTRA_BACKENDS variable when invoking "make
deb rpm distbin".
It is highly annoying that D-Bus method calls (synchronous and
asynchronous!) time out fairly quickly (a few minutes) when doing
interactive debugging of syncevo-dbus-server.
Traditionally, every single method call had to be made with
timeout=<some high value>, which made the test code unnecessarily
larger and often was forgotten.
Now all method calls via the session bus connection are done with a
high timeout if none was set by the caller. This is achieved by
wrapping the default (internal?) method send methods on the bus object
with versions which add that timeout. This should be safe, because
these methods are part of the public D-Bus Python API.
The "D-Bus testing: don't depend on server output during startup, truely quiet
TEST_DBUS_QUIET" change broke TestFileNotify.testSession3: because the test
started after syncevo-dbus-server obtained the bus name and before the server
could finish its startup (which test-dbus.py previously waited for by waiting
for the server's output), the executable was touched too soon and the server
often didn't notice that it had to shut down.
This inherent race condition can't be fixed, but in reality it should be a lot
rarer (not happening at all?) than in testing. Therefore fix testing by having
test-dbus.py wait for a response from syncevo-dbus-server (which implies that
it is done with its startup) before starting the test.
This actually applies to all three ways of starting syncevo-dbus-server (with
gdb, with and without logging), so now all of them use the same code.
We used to kill it when it showed the first sign of life via D-Bus log output
- any output! Depending on timing, it may or may not have been able to send
the "target side ready" INFO message. If it did, our strict output check
failed.
Fix that by waiting for that message, which should be the only INFO message
and thus the only one which will appear in the output text, before killing the
process.
For each incoming change, one INFO line with "received x[/out of y]"
was printed, immediately followed by another line with total counts
"added x, updated y, removed z". For each outgoing change, a "sent
x[/out of y]" was printed.
In addition, these changes were forwarded to the D-Bus server where a
"percent complete" was calculated and broadcasted to clients. All of
that caused a very high overhead for every single change, even if the
actual logging was off. The syncevo-dbus-server was constantly
consuming CPU time during a sync when it should have been mostly idle.
To avoid this overhead, the updated received/sent numbers that come
from the Synthesis engine are now cached and only processed when done
with a SyncML message or some other event happens (whatever happens
first).
To keep the implementation simple, the "added x, updated y, removed z"
information is ignored completely and no longer appears in the output.
As a result, syncevo-dbus-server is now almost completely idle during
a running sync with no log output. Such a sync involving 10000 contacts
was sped up from 37s to 26s total runtime.
When asked to run quietly via a non-empty TEST_DBUS_QUIET,
test-dbus.py still enabled INFO messages to determine that the server
is ready. Waiting for bus name (as already done when starting with a
debugger) avoids that and allows us to disable all LogOutput signals.
More recent GNOME Python bindings are provided by gobject
introspection. The traditional gobject/glib modules no
longer exist.
The API is similar enough that we just need to adapt importing: if
importing the normal modules fails, try importing from gi.repository
instead.
It is useful to let tests add (or more likely, overwrite) files to an
XDG directory shared between tests. Those tests must be aware that
there may be other files in the XDG dir from other tests.
It can happen that the background thread terminates before libsynthesis checks
for termination. In that case our helper code does not get called and thus
cannot log the message that the test was checking for. Instead check for
proper continuation of the file source open.
Use type checking to determine whether a timeout was done via
glib or signals. Without removal of a glib timeout, a periodic
timeout keeps firing in other tests.
Ensure that server is currently waiting for background thread, then
abort via Session.Abort(). Needs to be detected by backend and the
lead to normal session shutdown.
test-dbus.py now knows how to start syncevo-http-server. Therefore
it can test normal HTTP-based syncing as well as several scenarios
which fail or succeed with a slow server depending on the server's
ability to send SyncML messages while still initializing the storage.
To make the server slow, env variables are checked by the file
backend. It may matter whether open() or listAll() are slow, so test
both.
The tests expecting the 2 minute default must check whether the
feature is enabled at all in the binary that they are testing. If not,
the test cannot run. All other tests work, albeit somewhat unsafely
because they force the engine to run multithreaded when the engine was
compiled without mutex locking of global data structures.
Refactored the code into a new utility base class for use in other
tests.
Replace pipes with temporary files, using the same base name as the
traditional .syncevo.log and .dbus.log. They are numbered (because the
command line might be run multiple times per test) and use .out, .err,
or .outerr as suffix depending on what kind of output they
contain. The advantage is that the output gets recorded permanently.
Use that when waiting for command completion times out: in that case,
the content of the output file(s) gets added to the exception.
The subprocess handle returned by startCmdline() is extended with
information about output redirection that is then used by
finishCmdline(). This makes one parameter of finishCmdline()
redundant.
For interactive debugging it is better to have an infinite timeout of
the D-Bus method call. The overall test will still time out as quickly
as it did before.
Patrick Ohly