Recent shells filter out environment variables that are not valid shell
variables, so for example,
SYNCEVOLUTION_FILE_SOURCE_DELAY_OPEN_addressbook-slow-server did not get
passed through to syncevo-dbus-server because of the hyphen. As a result, the
tests became unreliable (non-deterministic timing) or outright failed.
Now we ensure that these variables are valid also in a shell and in addition,
make the test stricter such that it detects when the file backend did not
wait.
Recent distros like Debian Stretch have newer dbus-monitor with
extended content that no longer matched the regex used before. The
relaxed regex works with old and new dbus-monitor.
As reported by Canonical, syncing fails if data items contain
text which is not correct UTF-8 in one of the fields that
SyncEvolution logs in the command line output (like SUMMARY of
a calendar event).
That is because the byte string coming from the item is passed
unchecked to the D-Bus implementation for transmission via D-Bus. But
D-Bus strings must be correct UTF-8, so depending on the D-Bus library
in use, one gets a segfault (GIO D-Bus, due to an unchecked NULL
pointer access) or an "out of memory" error (libdbus, which checks for
NULL).
What the D-Bus bindings now do is checking the string in advance (to
avoid error messages inside the D-Bus implementations) and then
replacing all invalid bytes with question marks. The rest of the
string is preserved.
Handling this inside the D-Bus binding layer is not the "correct"
solution (which would be to check for UTF-8 in the higher layers were
such bad data might get into SyncEvolution), but it is the less
intrusive and more complete one. Changing the bindings such that all
strings must be declared explicitly as "UTF-8 string" would have been
a way to find all places where such checks are missing, but it turned
out to be too complex and requiring too many changes.
Extracting a meaningful description of each item from the Synthesis
engine when updating and adding items is easy to do for items of
certain known types (contacts and calendar items) and arguably an
improvement; in particular it makes tests like
TestCmdline.testSyncOutput more realistic.
The issue with malloc being called in niam() in syncevo-dbus-server
and deadlocking was only found after adding gdb's output of the
syncevo-dbus-server stack backtrace.
During TestCmdline.testSyncOutput also verify that syncs which don't
change the database also don't update the ~/.config meta data.
The listall() method from testpim.py is used and extended for that
and therefore moved to test-dbus.py.
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
The log prefix seems to be unused (except for some debug message) at
the moment but will be soon, so make sure it gets embedded in the
string sent to the syncevo-dbus-server.
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google-calendar" will pick the "Google_Calendar"
template automatically because it knows that the intention is to
configure the target side of a local sync. It does not know that when
using some other name for the config, in which case the template (if
needed) must be specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
The first progress signal gets emitted after sleeping for 10 seconds
at the start of the sync and then killing syncevo-dbus-server races
with completing the sync. What we want is to kill during the 10 second
wait, so we better wait for the debug output directly before it and
then kill directly.
Funambol turned of the URL redirect from my.funambol.com to
onemedia.com. The Funambol template now uses the current URL. Users
with existing Funambol configs must updated the syncURL property
manually to https://onemediahub.com/sync
Kudos to Daniel Clement for reporting the change.
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.
In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.
Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.
During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:
X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE
Any IMPP property declared as X-SERVICE-TYPE=AIM will get
mapped to X-AIM. Same for others. Some IMPP service types
have no known X- property extension; they are stored in
EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
(for example, GaduGadu and Groupwise) are stored with
X-SERVICE-TYPE values chosen by SyncEvolution so that
Google CardDAV preserves them (GroupWise with mixed case
got translated by Google into Groupwise, so the latter is used).
Google always sends an X-ABLabel:Other for IMPP. This is ignored
because the service type overrides it.
The value itself also gets transformed during the mapping. IMPP uses
an URI as value, with a chat protocol (like "aim" or "xmpp") and
some protocol specific identifier. For each X- extension the
protocol is determined by the property name and the value is the
protocol specific identifier without URL encoding.
X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel
The mapping is based on the X-ABLabel property attached to
the X-ABRELATEDNAMES property. This depends on the English
words "Spouse", "Manager", "Assistant" that Google CardDAV
and Apple devices seem to use regardless of the configured
language.
As with IMPP, only the subset of related names which have
a corresponding X- property extension get mapped. The rest
is stored in EDS using the X-ABRELATEDNAMES property.
X-ANNIVERSARY <-> X-ABDATE
Same here, with X-ABLabel:Anniversary as the special case
which gets mapped.
X-ABLabel parameter <-> property
CardDAV vCards have labels attached to arbitrary other properties
(TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
item1.X-ABDATE:2010-01-01
item1.X-ABLabel:Anniversary
The advantage is that property values can contain arbitrary
characters, including line breaks and double quotation marks,
which is not possible in property parameters.
Neither EDS nor KDE (judging from the lack of responses on the
KDE-PIM mailing list) support custom labels. SyncEvolution could
have used grouping as it is done in CardDAV, but grouping is not
used much (not at all?) by the UIs working with the vCards in EDS
and KDE. It seemed easier to use a new X-ABLabel parameter.
Characters which cannot be stored in a parameter get converted
(double space to single space, line break to space, etc.) during
syncing. In practice, these characters don't appear in X-ABLabel
properties anyway because neither Apple nor Google UIs allow entering
them for custom labels.
The "Other" label is used by Google even in case where it adds no
information. For example, all XMPP properties have an associated
X-ABLabel=Other although the Web UI does not provide a means to edit
or show such a label. Editing the text before the value in the UI
changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
other fields.
Therefore the "Other" label is ignored by removing it during syncing.
X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.
CardDAV needs to use test data with the new CardDAV vCard flavor.
Most CardDAV servers can store EDS vCards and thus
Client::Source::*::testImport passed (with minor tweaks in
synccompare) when using the default eds_contact.vcf, but
Client::Sync::*::testItems fails when comparing synced data with test
cases when the synced data uses the native format and the test cases
are still the ones from EDS.
A libsynthesis with URLENCODE/DECODE() and sharedfield parameter for
<property> is needed.
The test assumed that it can rename the main syncevo-dbus-server
executable to trigger the file watch mechanism. That's not correct:
- It might be the system's /usr/libexec/syncevo-dbus-server,
which a normal user cannot rename.
- The binary might be also active in some other, parallel tests.
Renaming it interferes with those other tests.
The latter happened in the nightly testing: HTTP server tests with
a long-running syncevo-dbus-server failed because the daemon terminated
during the tests.
This adds GetPeerStatus() and "progress" events.
To detect DB changes as they happen, the SyncSource operations are
monitored. Upon entry, a counter gets increased and transmitted
through to the PIM manager in syncevo-dbus-server using extended
SourceProgress structs (only visible internally - public API must not
be extended!). This will count operations which fail and count those
twice which get resumed, so the counts will be too high
occasionally. That is in line with the API definition; because it is
not exact, the API only exposes a "modified" flag.
Progress is reported based on the "item received" Synthesis event and
the total item count. A modified libsynthesis is needed where the
SyncML binfile client on the target side of the local sync actually
sends the total item count (via NumberOfChanges). This cannot be done
yet right at the start of the sync, only the second SyncML message
will have it. That is acceptable, because completion is reached very
quickly anyway for syncs involving only one message.
At the moment, SyncContext::displaySourceProgress() holds back "item
received" events until a different event needs to be emitted. Progress
reporting might get more fine-grained when adding allowing held back
events to be emitted at a fixed rate, every 0.1s. This is not done yet
because it seems to work well enough already.
For testing and demonstration purposes, sync.py gets command line
arguments for setting progress frequency and showing progress either
via listening to signals or polling.
In testAutoSyncNoNetworkManager, syncs keep happening while the test
terminates. The processes get killed correctly, but that ocassionally triggers
notifictions which the post-test D-Bus log checking treats as test failures.
To avoid this, truncate the D-Bus log before shutting down. Then all extra
events are available for debugging, but will not be seen by the check code.
Check for an already running instance. Otherwise the new one
will fail to start up, which will be hard to diagnose. Happened
recently when a test was broken and failed to kill its daemon
instance.
When spawning the dlt daemon under "unbuffer", a lingering tclsh process
exists before the test starts and the test ended up waiting for that to
stop.
We need to be more specific with identifying sync processes: only those
processes started as part of syncing matter.
The test randomly failed under load because the client's GetVersion
call did not make it to the server in time. It seems to work better
with a smaller delay.
The test relied on fixed timeouts for a) killing syncevo-dbus-server
and b) completion of the sync. The second timeout sometimes occurred
to soon, causing the testNoParent itself to fail and also the following
test (because log files from testNoParent were not removed).
Now be more careful about really killing in the middle of the sync (thanks
to delaying the sync and not the helper startup) and really wait for process
termination (with getChildren() and os.kill() checking).
Picking a port dynamically was broken: if 9999 was in use, the test script
connected to the wrong process and then incorrectly continued, instead of
detecting that syncevo-http-server failed. Use the auto-alloc feature
of syncevo-http-server (port == 0) instead and get the actual port
from the output.
Now that we redirect that output into a file, it makes sense to also include
that log in test failure reports.
Only the "client-test" executable and its test data were installed so far, in
/usr/bin resp. the doc dir. Now also test-dbus.py and testpim.py plus their
data files get installed, using a new /usr/lib/syncevolution/test directory
for all test files and commands.
"runtests.py --prebuilt" can take one or more .deb files including these
tests, will install them as the "compile" operation and then will use the
installed SyncEvolution, without overriding any paths. This relies on having
suitable rights for "dpkg" and /usr/lib/syncevolution/test; the way how that
is done in the nightly testing is via chroots where the entire file system
is writable by the normal test user.
Normal users without such write access to /usr/lib/syncevolution/test need
to copy that directory into their home directory first.
This adds hooks into doAutoSyncLocalSuccess() which are used by
testAutoSyncLocalMultiple() to start a session while waiting for auto-sync,
access the config which will get auto-synced, then continue waiting.
While at it, also measure the actual auto sync interval and roughly
compare against the configured one.
The actual timeout duration was too small by a factor of 1000 (ms
versus s mismatch). Did not affect existing tests which just had their
(anyway small) timeouts triggered too soon.
When TEST_DBUS_DLT is set, start the DLT daemon and tell syncevo-dbus-server
to use it. This relies on dlt-daemon being in the PATH, which requires
some changes in setting that env variable via runtests.py.
unbuffer from expect-dev (on Debian) is needed to get dlt-daemon to
print each message immediately.
This only adds the infrastructure for testing DLT logging. The actual
tests still need to be added.
The hyphen introduced with the alternate EDS backends wasn't expected
by the regex. Now simply assume that backend names are non-whitespace
characters.
In testSyncOutput, valgrind really chewed more than one minute
on syncevo-dbus-server during process shutdown. We need to give
it enough time, otherwise the test failed with "process had to be
killed".
Diagnostic Log and Trace (DLT) manages a sequence of log messages,
with remote controllable level of detail. SyncEvolution optionally
(can be chosen at compile time and again at runtime) uses DLT instead
of its own syncevolution-log.html files.
The backends must be compiled differently for EDS < 3.6 (using the old
API before EBookClient, ECalClient and ESource, ideally in
compatibility mode) and for EDS >= 3.6 (using the new API, with hard
linking against libebook-1.2.so.14, libecal-1.2.so.15,
libedataserver-1.2.so.17).
With these changes, a SyncEvolution binary compiled for the older EDS
API will be able to load and use backends compiled against the current
one. Both backends can be installed in the same
lib/syncevolution/backends dir under different names. The newer ones
must have an additional -<version> appendix before the .so suffix.
Then loading will attempt to use those backends first and if
successful, skip the older ones. This is necessary because loading
both into the same process will fail due to symbol clashes. If
unsuccessful (as on systems with only the older EDS), the dynamic
linker will fail and SyncEvolution proceeds with the backends for the
old API.
Packaging of binaries as .dev/.rpm/.tar.gz includes these additional
backends if given in the EXTRA_BACKENDS variable when invoking "make
deb rpm distbin".
It is highly annoying that D-Bus method calls (synchronous and
asynchronous!) time out fairly quickly (a few minutes) when doing
interactive debugging of syncevo-dbus-server.
Traditionally, every single method call had to be made with
timeout=<some high value>, which made the test code unnecessarily
larger and often was forgotten.
Now all method calls via the session bus connection are done with a
high timeout if none was set by the caller. This is achieved by
wrapping the default (internal?) method send methods on the bus object
with versions which add that timeout. This should be safe, because
these methods are part of the public D-Bus Python API.
The "D-Bus testing: don't depend on server output during startup, truely quiet
TEST_DBUS_QUIET" change broke TestFileNotify.testSession3: because the test
started after syncevo-dbus-server obtained the bus name and before the server
could finish its startup (which test-dbus.py previously waited for by waiting
for the server's output), the executable was touched too soon and the server
often didn't notice that it had to shut down.
This inherent race condition can't be fixed, but in reality it should be a lot
rarer (not happening at all?) than in testing. Therefore fix testing by having
test-dbus.py wait for a response from syncevo-dbus-server (which implies that
it is done with its startup) before starting the test.
This actually applies to all three ways of starting syncevo-dbus-server (with
gdb, with and without logging), so now all of them use the same code.
We used to kill it when it showed the first sign of life via D-Bus log output
- any output! Depending on timing, it may or may not have been able to send
the "target side ready" INFO message. If it did, our strict output check
failed.
Fix that by waiting for that message, which should be the only INFO message
and thus the only one which will appear in the output text, before killing the
process.
For each incoming change, one INFO line with "received x[/out of y]"
was printed, immediately followed by another line with total counts
"added x, updated y, removed z". For each outgoing change, a "sent
x[/out of y]" was printed.
In addition, these changes were forwarded to the D-Bus server where a
"percent complete" was calculated and broadcasted to clients. All of
that caused a very high overhead for every single change, even if the
actual logging was off. The syncevo-dbus-server was constantly
consuming CPU time during a sync when it should have been mostly idle.
To avoid this overhead, the updated received/sent numbers that come
from the Synthesis engine are now cached and only processed when done
with a SyncML message or some other event happens (whatever happens
first).
To keep the implementation simple, the "added x, updated y, removed z"
information is ignored completely and no longer appears in the output.
As a result, syncevo-dbus-server is now almost completely idle during
a running sync with no log output. Such a sync involving 10000 contacts
was sped up from 37s to 26s total runtime.
When asked to run quietly via a non-empty TEST_DBUS_QUIET,
test-dbus.py still enabled INFO messages to determine that the server
is ready. Waiting for bus name (as already done when starting with a
debugger) avoids that and allows us to disable all LogOutput signals.
More recent GNOME Python bindings are provided by gobject
introspection. The traditional gobject/glib modules no
longer exist.
The API is similar enough that we just need to adapt importing: if
importing the normal modules fails, try importing from gi.repository
instead.