Switching the testing to Exchange 2016 revealed a slightly
different time zone mangling than before:
- the "US & C" time zone name already gets truncated before the C
- due to recent rules on the server (?), several TZNAMEs get
changed
Updating the data again...
dbus-launch is considered deprecated because of the X11 dependency.
See https://lists.debian.org/debian-devel/2016/08/msg00554.html "Mass bug
filing: use and misuse of dbus-launch (dbus-x11)"
The script still needs to start the D-Bus daemon when used in the nightly
testing, so the code now does it as in
6cc8062cfb
syncevo-http-server still has some usage of dbus-launch left, but that's
strictly for systems which don't have the more modern D-Bus.
DAViCal on Debian Stretch incorrectly re-formats contacts which do not have a
REV property (see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837154).
Until this gets fixed we work around the problem by including a REV in all
test cases, which implies refreshing the Google CardDAV patch.
Recent shells filter out environment variables that are not valid shell
variables, so for example,
SYNCEVOLUTION_FILE_SOURCE_DELAY_OPEN_addressbook-slow-server did not get
passed through to syncevo-dbus-server because of the hyphen. As a result, the
tests became unreliable (non-deterministic timing) or outright failed.
Now we ensure that these variables are valid also in a shell and in addition,
make the test stricter such that it detects when the file backend did not
wait.
Recent distros like Debian Stretch have newer dbus-monitor with
extended content that no longer matched the regex used before. The
relaxed regex works with old and new dbus-monitor.
When building on the new syncevolution.org reference platform, Ubuntu Trusty,
building rpm packages with checkinstall fails for unknown reasons. As
providing those rpms is of questionable value (libraries are more likely to be
different on rpm-based distros than on Debian/Ubuntu), building rpm simply
gets disabled entirely.
As a fallback for users there still are the plain .tar.gz archives containing
the same files.
Previously, when --enabled was not given, all tests were run. That was meant
to simplify running all tests, but that is rare and more likely to happen by
mistake when not selecting tests correctly in the automated testing, so now we
revert the logic and don't select a test if no --enable filter is set.
The XDG symlinks were created as fallback in case that the XDG env variables
somehow were ignored. In practice, the symlinks were not needed, so it didn't
matter that they were set incorrectly to the path name outside of the chroot.
But it's confusing when debugging the setup and just plain wrong, so better
fix it...
client-test makes new directories user-readable because that is
the default for all SyncEvolution directories. But after publishing
the test results, they need to be world-readable, because they are
going to be accessed via some kind of web server running under
some different user.
These linker flags are added to those normally used for linking against
libsyncevolution. The intended usage is to extend linking of syncevolution.org
binaries such that static versions of libcppunit, libpcrecpp and libopenobex
are used, because the ABI of those libs have changed such that binaries linked
on Ubuntu Trusty would not run on more recent distros like Ubuntu Xenial.
For example, on Ubuntu Trusty i386 one can configure with:
PCRECPP_LIBS=-lpcre \
LIBOPENOBEX_LIBS=-lpcre \
'--with-extra-core-ldadd=/usr/lib/i386-linux-gnu/libpcrecpp.a -lpcre /usr/lib/libopenobex.a /usr/lib/i386-linux-gnu/libusb.a' \
CPPUNIT_LIBS=/usr/lib/i386-linux-gnu/libcppunit.a
Now cppcheck is found via the normal PKG_CHECK_MODULES. The advantage
is that CPPUNIT_CFLAGS and CPPUNIT_LIBS can be overridden with
configure parameters, which will be used to link cppcheck statically
into syncevolution.org binaries (libcppunit ABI has changed).
Building client-test fails because a const pointer looses the
const attribute while passing it through boost::bind.
Fixes:
boost/lambda/detail/function_adaptors.hpp:357:16: error: invalid conversion
from 'const SyncEvo::SyncSource*' to 'SyncEvo::SyncSource*' [-fpermissive]
return func(a1);
twisted.web.error.NoResource was replaced by
twisted.web.resource.NoResource. This has become a real problem for
example on Fedora 22 where the old name is no longer available.
As reported by Canonical, syncing fails if data items contain
text which is not correct UTF-8 in one of the fields that
SyncEvolution logs in the command line output (like SUMMARY of
a calendar event).
That is because the byte string coming from the item is passed
unchecked to the D-Bus implementation for transmission via D-Bus. But
D-Bus strings must be correct UTF-8, so depending on the D-Bus library
in use, one gets a segfault (GIO D-Bus, due to an unchecked NULL
pointer access) or an "out of memory" error (libdbus, which checks for
NULL).
What the D-Bus bindings now do is checking the string in advance (to
avoid error messages inside the D-Bus implementations) and then
replacing all invalid bytes with question marks. The rest of the
string is preserved.
Handling this inside the D-Bus binding layer is not the "correct"
solution (which would be to check for UTF-8 in the higher layers were
such bad data might get into SyncEvolution), but it is the less
intrusive and more complete one. Changing the bindings such that all
strings must be declared explicitly as "UTF-8 string" would have been
a way to find all places where such checks are missing, but it turned
out to be too complex and requiring too many changes.
Extracting a meaningful description of each item from the Synthesis
engine when updating and adding items is easy to do for items of
certain known types (contacts and calendar items) and arguably an
improvement; in particular it makes tests like
TestCmdline.testSyncOutput more realistic.
Occasionally the script shutdown got stuck on Ubuntu Vivid because killing
the background daemon failed although it was still runnning, thus causing the
wait to hang forever.
Not exactly sure what caused this. The enhancement tries to fall back to
killing the process instead of the process group (in case that there is
a race condition, which shouldn't be the case when waiting for the daemon),
preserves stderr from the kill commands and adds ps output when there is
an unexpected failure.
The test started to fail in 2015 because the VTIMEZONE generated by
the code copied from libical has a minor dependency on the current
time: it finds the transitions for the current year, then pretends
that the same rule applied since 1970. While doing that, it uses this
years transition date and just replaces the year. Because the exact
date varies between years, the result depends on the current year.
The test now simply ignores the day of the month, while still checking
year and month. Those should always be the same.
Useful for correlating events in the daemon with events in testing.
We need to use a process group to deliver SIGINT/SIGTERM, otherwise
we cannot be sure that we catch all processes created by the daemon.
The return code of the daemon was not checked before (accidentally?!)
and this patch does not change that. Might be fixed in the future.
The daemon log gets reused when starting daemons multiple times.
When starting a second time, the simple grep used to find the
output of the previous and did not wait properly. We need to count
matching lines and continue once a new one appears.
Multiple different wrappercheck instances are running concurrently
in the nightly testing, so better include a prefix with unique PID
in the "set -x" output to identify where it came from.
The issue with malloc being called in niam() in syncevo-dbus-server
and deadlocking was only found after adding gdb's output of the
syncevo-dbus-server stack backtrace.
"primary 80" ends up being recognized as phone number, adding the E.164
parameter if (and only if) EDS was compiled with libphonenumber support
(Debian Jessie!). Adding that breaks the comparison, so avoid the situation
by updating to a string which is not a phone number.
Akonadi uses different quoting for special characters. Probably would have to
be looked into and reported, but for now ignore it by not testing these
special cases.
When syncing Akonadi with file source, neither side recognizes duplicates
based on UID/RECURRENCE-ID. Should be added to Akonadi source. For now ignore
it.
When client-test starts, it determines all tests that would get
run, then runs all of them one-by-one in new instances. A single
test gets run directly.
The output changes slightly: the CppUnit summary with number of
failures or errors is no longer available and there are additional
blank lines between tests.
The advantage is that each single test is properly isolated from the
other, which is closer to how real syncs run. It also helps when
running under valgrind, because a leak can be attributed exactly to
one test and because it avoids permanently growing memory consumption
in long-running client-test runs (seen in the nightly testing even
when there were no leaks, perhaps because of memory fragmentation).
A potential downside of this change is that unexpected and undesirable
side effects of modules might no longer show up in testing, only when
combining them in real syncs. This should still be covered by
Client::Sync tests involving multiple modules.
When sent
SUMMARY:Simple
DESCRIPTION:Simple
Memotoo returns
SUMMARY:Simple
DESCRIPTION:Simple\nSimple
Remove this particular test case until the problem in the server
is fixed.
During TestCmdline.testSyncOutput also verify that syncs which don't
change the database also don't update the ~/.config meta data.
The listall() method from testpim.py is used and extended for that
and therefore moved to test-dbus.py.
This allows signal handlers to affect the operation execution and
result without having to throw an exception, which would get logged as
error and is not desirable when the operation gets skipped
intentionally. Needed for skipping SaveAdminData in the next patch.
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
The log prefix seems to be unused (except for some debug message) at
the moment but will be soon, so make sure it gets embedded in the
string sent to the syncevo-dbus-server.
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google-calendar" will pick the "Google_Calendar"
template automatically because it knows that the intention is to
configure the target side of a local sync. It does not know that when
using some other name for the config, in which case the template (if
needed) must be specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
The first progress signal gets emitted after sleeping for 10 seconds
at the start of the sync and then killing syncevo-dbus-server races
with completing the sync. What we want is to kill during the 10 second
wait, so we better wait for the debug output directly before it and
then kill directly.
Patrick Ohly
frdsktp
Tino Mettler