When neither gSSO nor UOA were enabled, the provideruoa.so was
still enabled and failed to link because its LDFLAGS were not
set. It shouldn't have been enabled at all.
While we are at it, allow both to be enabled in the build rules,
even though the code doesn't support it (symbol clashes).
Add a new configuration flag (--enable-uoa) to enable building the
signon backend for Ubuntu Online Accounts. The work done for gSSO is
reused and just slightly adapted (using preprocessor conditions) to work
with UOA as well.
Because of an uninitialized memory read introduced for DLT after
1.3.99.5, output from the syncevolution command line tool was not
shown randomly.
Other usages of the second constructor for MessageOptions may also
have been affected.
That this was not caught by nightly testing points towards a problem
which will have to be solve separately: test-dbus.py needs to run
syncevolution under valgrind.
Change the EDS backend to not batch by default and only enable it
for PIM Manager PBAP syncs.
This avoids unnecessary overhead when running a normal SyncML sync
with a non-SyncEvolution peer, because then the batched operation
would be flushed immediately by the Synthesis engine, and more
importantly, it fixes a segfault in
Client::Sync::eds_event_eds_contact::testLargeObject EDS<->DAV tests.
Enabling batching during general syncs obviously needs further work...
libical 1.0 changed the library name to libical.so.1, causing dependency
problems becaus syncevolution.org binaries required libical.so.0.
It turned out that the relevant parts of the API and ABI had not changed,
so simply allowing both versions of the library is sufficient.
In addition, a detection of libical.so.1 gets added, just in case.
Most glib warnings are too technical for normal users. Better restrict
their logging to "for developers", which keeps them out of stderr of normal
invocations.
Triggered by a new warning showing up with GNOME 3.8 which didn't seem
to cause any problems.
The code wasn't used and incorrectly made it look like SyncEvolution used an
enum which changed its value between libical.so.0 and libical.so.1. Better
remove the code...
obexd 0.48 is almost the same as obexd 0.47, except that it dropped
the SetFilter and SetFormat methods in favor of passing a Bluex 5-style
filter parameter to PullAll.
SyncEvolution now supports 4, in words, four different obexd
APIs. Sigh.
The PIM Manager should be able to start normally and the default sort
order should be used instead of the invalid one from the config.
Invalid address books are also tested, without checking for any
specific value from GetActiveAddressBooks().
Failure to set the sort order from pim-manager.ini should not
prevent the startup of the PIM Manager because the client
cannot really diagnose and fix the problem. It is better
to try again with the default sort order.
The content of pim-manager.ini should be monitored to find issues
like the one reported in FDO #70772. For that particular problem
to appear we also need to use SetActiveAddressBooks().
Removing a peer accidentally wrote the updated list of active address
books into the "sort" property of pim-manager.ini, which then
prevented starting the PIM Manager (a different problem which will
be addressed separately).
The normalization of phone numbers depends on the locale. This commit
adds a test for that and code which works without relying on EDS to
re-calculate those normalized numbers.
This is not the normal mode of operation and thus this can't be the
final solution. The right solution would be for EDS to notice the
locale change, re-check all precomputed phone numbers
(X-EVOLUTION-E164 parameter) and emit change notifications. Those then
would get picked up by folks and from there, update the PIM views.
The tests rely on running dbus-session.sh with
DBUS_SESSION_SH_SYSTEM_BUS set.
Listen to signals from localed D-Bus system service and update all
internal state which depends on the current locale. This state includes:
- pre-computed data in all loaded contacts
- filtering (for example, case sensitivity is locale dependent)
- the sort order
In the current implementation, the entire LocaleFactory gets replaced
when the environment changes. The new instance gets installed in
FullView, then the sort comparison gets created anew and the existing
setSortOrder() updates the main view, which includes re-computing all
pre-computed data.
As a last step, the search filter is re-recreated and filtering gets
updated in all active filter views.
There is a minor risk that unnecessary changes get emitted because
first filtered views react to modified contacts and/or reshuffling
them, then later their filter gets replaced.
The listening functionality is meant to be optional, which includes
running inside an environment which doesn't even have a system D-Bus,
like the nightly testing.
This is the client side for the org.freedesktop.locale1 "Locale"
property. It ensures that new values of that property are made
available locally.
Optionally it can also update the current process' environment and
send a simpler "locale environment changed" if it made some changes.
This needs to be enabled by connecting the setLocale() method to the
m_localeValues signal.
After testing more with EDS 3.8 on Debian Testing, random hangs
were observed inside the glib main loop while waiting for the
registry creation callback. It's uncertain why that happens,
and due to lack of time at the moment it is easier to switch
back to the synchronous method. Needs to be investigated later.
Accent-insensitive search ignores accents, using the same code as in
EDS. Transliterated search ignores foreign scripts by transliterating
search term and contact properties to Latin first. That one is using
ICU directly in the same way as EDS, but doesn't use the EDS
ETransliterator class to avoid extra string copying.
This commit changes the default behavior such that searching is by
default most permissive (case- and accent-insensitive, does
transliteration). Flags exist to restore more restrictive matching.
By default, syncevo-dbus-server logs to syslog. This can be changed
with the new parameter of --enable-dbus-service. For example, this
changes the logging to DLT (must be available, or course):
configure "--enable-dbus-server=--dlt --no-syslog"
Diagnostic Log and Trace (DLT) manages a sequence of log messages,
with remote controllable level of detail. SyncEvolution optionally
(can be chosen at compile time and again at runtime) uses DLT instead
of its own syncevolution-log.html files.
Adding link flags to the libsyncevlution.la dependencies only works
for .la file. This started to break on Debian Testing (?) because
a -L<something> flag became a dependency.
A proper fix would be to set libs and dependencies separately,
but for now it is easier to rely on GNU make and filter for the
desired .la files.
We unintentionally and unnecessarily included boost/signals.hpp instead of
boost/signals2.hpp, which started to trigger warnings with later versions of
boost.
This fixes compilation with EDS 3.6. libebook-contacts was needed
for a while when configuring address books directly. It is not needed
anymore because now we simply clone the system database.
The backends must be compiled differently for EDS < 3.6 (using the old
API before EBookClient, ECalClient and ESource, ideally in
compatibility mode) and for EDS >= 3.6 (using the new API, with hard
linking against libebook-1.2.so.14, libecal-1.2.so.15,
libedataserver-1.2.so.17).
With these changes, a SyncEvolution binary compiled for the older EDS
API will be able to load and use backends compiled against the current
one. Both backends can be installed in the same
lib/syncevolution/backends dir under different names. The newer ones
must have an additional -<version> appendix before the .so suffix.
Then loading will attempt to use those backends first and if
successful, skip the older ones. This is necessary because loading
both into the same process will fail due to symbol clashes. If
unsuccessful (as on systems with only the older EDS), the dynamic
linker will fail and SyncEvolution proceeds with the backends for the
old API.
Packaging of binaries as .dev/.rpm/.tar.gz includes these additional
backends if given in the EXTRA_BACKENDS variable when invoking "make
deb rpm distbin".
The EDS backends for >= 3.6 failed to work property in SyncEvolution
binaries compiled for EDS < 3.6, because the EDS compatibility
layer reported "EDS not found" to the backend.
A backend that gets compiled without compatibility hacks doesn't
need the runtime check, because merely loading already guarantees
that the necessary libs were found. Therefore make the check
a compile-time define when compatibility mode is disabled.
A backend compiled with compatibility mode enabled then fails to load
in a binary compiled without because libsyncevolution.so does not
export the necessary symbols. That's actually desirable, because the
backend would not work anyway.
Setting the SYNCEVOLUTION_EBOOK_QUERY env variable to a valid EBook
query string limits the results to contacts matching that
query. Useful only in combination with --print-items or --export. Only
implemented for EDS >= 3.6.
Previously, the current default country was used to turn phone numbers
without an explicit country code into full E164 numbers, which then
had to match the search term when doing a caller ID lookup.
This was inconsistent with EDS, where a weaker
EQUALS_NATIONAL_PHONE_NUMBER was done. The difference is that a
comparison between a number with country code matches one without if
the national number of the same, regardless of the current default
country. This is better because it reduces the influence of the hard
to guess default country on matching.
SyncEvolution also differed from EDS by doing a prefix comparison,
which in theory might have also ignored differences caused by
extensions. It is uncertain whether that was useful, so for the sake
of consistency, a full number comparison of the national number is now
done.
Another advantage of this change is the lower memory consumption and
faster comparison, because strings are now stored in 4 + 8 byte
numbers instead of strings of varying length.
The array operator happens to work on some platforms, but not others
(see previous commit). Make it private without an implementation to
catch the undesired usage of it on platforms whether the code would
happen to work otherwise.
This fixes the following problem, seen with Boost 1.53.0 on altlinux
when compiling for EDS >= 3.6:
/usr/include/boost/smart_ptr/shared_ptr.hpp: In instantiation of 'typename boost::detail::sp_array_access<T>::type boost::shared_ptr<T>::operator[](std::ptrdiff_t) const [with T = char*; typename boost::detail::sp_array_access<T>::type = void; std::ptrdiff_t = long int]':
src/backends/evolution/EvolutionSyncSource.cpp:163:38: required from here
/usr/include/boost/smart_ptr/shared_ptr.hpp:663:22: error: return-statement with a value, in function returning 'void' [-fpermissive]
make[2]: *** [src/backends/evolution/src_backends_evolution_syncecal_la-EvolutionSyncSource.lo]
The "void" type above is wrong, so it looks like a missing type trait
for the pointer type used in the smart_ptr. PlainGStrArray already had
an at() method to work around such issues, so use it. Not sure why this
one usage of [] slipped through.
While running a sync with a binary compiled with -fPIE -pie, a crash
in strlen() occured because a 64 bit string pointer coming from D-Bus
was incorrectly passed through a 32 bit unsigned variable.
These special compile flags merely caused the problem to occur
reliably, it may also have crashed under other circumstances.
Kudos to Tino Keitel for reporting the problem and identifying the
relation to the compile flags.
Running a sync while the UI had no service selected caused a crash in
find_updated_source_progress() because the code dereferences the NULL
prog_data->data->current_service pointer. Affected both the GTK2 and
GTK3 UI.
Fix it by checking for NULL and not doing anything if NULL.
When compiling source files of client-test, use -g as default CXXFLAGS
instead of the "-g -O2" that autotools normally picks. That speeds up
compilation significantly (on some platforms, gcc can't deal with the
many templates in ClientTest.cpp well) and leads to more useful
executables (suitable for interactive debugging) even when the rest of
the code gets optimized.
Explicitly specifying CXXFLAGS still overrides this per-target
default.
This feature depends on GNU make. A configure check is in place
to disable it when not using GNU make.
It seems that sometimes setting up a session with GNOME keyring fails such
that all further communication leads to decoding problem.
There is an internal method to reset the session, but it cannot be called
directly. As a workaround, fake the death of the GNOME keyring daemon
and thus trigger a reconnect when retrying the GNOME keyring access.
This drops the support for libgnome-keyring < 2.20, because older
versions did not have the error->text conversion method which is now
used in revised error and log messages.
This code also adds a retry loop around reading/writing passwords.
This was meant to work around these intermittent Gkr errors:
Gkr: received an invalid, unencryptable, or non-utf8 secret
Gkr: call to daemon returned an invalid response: (null).(null)()
These lead to an "Error communicating with gnome-keyring-daemon" status
code from libgnome-keyring.
However, once the error occurred in a process, it persists for at least
two seconds, possibly forever. Therefore the retry loop is not enabled.
Google CardDAV has one peculiarity: it renames new contacts during PUT without
returning the new path to the client. See also
http://lists.calconnect.org/pipermail/caldeveloper-l/2013-July/000524.html
SyncEvolution already had a workaround for that (PROPGET on old path, extract
new path from response) which happened to work. This workaround was originally
added for Yahoo, which sometimes merges contacts into existing ones. In
contrast to Yahoo, Google really seems to create new items.
Without some server specific hacks, the client cannot tell what happened.
Because Google is currently supported and Yahoo is not, let's change the
hard-coded behavior to "renamed items are new".
The nightly testing configures some platforms such that
XDG_CONFIG/DATA/CACHE_HOME are inside the build dir. It also populates these
dirs with files (for example, GNOME Online Accounts) which must survive all
cleaning of these directories.
Long term it would be better to separate test files from build files,
but that's a task for some other time...
When running a local sync, the syncURL/username/password are not meant
for the sync and cannot be used if they refer to an AuthProvider which
cannot return plain username/password.
In all other cases, this may or may not work, so at least try it instead
of hard-coding the IdentityProviderCredentials.
"username = goa:..." selects an account in GOA and retrieves the
OAuth2 token from that.
The implementation uses the GOA D-Bus API directly, because our C++
D-Bus bindings are easier to use and this avoids an additional library
dependency.
When clients like the GTK sync-ui stored a password, it was always
stored as plain text in the config.ini file by the
syncevo-dbus-server. The necessary code for redirecting the password
storage in a keyring (GNOME or KWallet) simply wasn't called in that
case.
The command line tool, even when using the D-Bus server to run the
operation, had the necessary code active and thus was not affected.
The client template is also used in cases where passwords are not
needed (local sync) and where passwords cannot be stored in a keyring
due to the missing syncURL/remoteDeviceID. Therefore don't set dummy
username/password values in the template.
When configuring a new peer and looking for databases, we need the
database password of an already existing source config, otherwise the
lookup will fail if that password is hidden in a keyring.
The command line tool in --daemon=no mode did not use the GNOME
keyring or KWallet even if the syncevo-dbus-server did, leading
to failing test cases when actually starting to use it by default
there.
Now all components use the same default: use safe password storage if
any was enabled during compilation, don't use if not.
This also makes SyncEvolution work without user intervention on
systems without a password storage.
Figuring out where credentials come from became harder. These debug
messages help. Perhaps they should even be logged as INFO messages
such that normal users can see them?
The code works with gSSO (https://01.org/gsso). With some tweaks to
the configure check and some ifdefs it probably could be made to work
with Ubuntu Online Accounts.
The code depends on an account accessible via libaccounts-glib which
has a provider and and (optionally) services enabled for that
provider. It is not necessary that the account already has a signon
identity ID, the backend will create that for the provider (and thus
shared between all services) if necessary.
Therefore it is possible to use the ag-tool to create and enable the
account and services. Provider and service templates are in the next
commit.
If given an AuthProvider which can handle OAuth2, then OAuth2 is
used instead of plain username/password authentication.
Obtaining the OAuth2 token must be done at a point where we can still
abort the request. If obtaining the token fails, then this should be
considered a fatal error which aborts scanning for resources. Other
errors cause the current URL to be skipped while scanning continues.
This commit moves the "execute request" functionality back into the
Neon::Session class, because that is where most of the logic (retry
request?) and state is (access tokens which persist across requests).
The real username is only relevant when running a sync. When looking
at a config with a D-Bus client like the GTK UI, the username should
always be "id:<config>", to avoid accidentally removing the
indirection, while the password should be the real one, to allow the
user to edit like he normally would with passwords stored in a
keyring.
To achive this, overriding the username must be suppressed when
resolving as part of the D-Bus config API. While at it, move the
entire "iterate over properties" into a common utility function in
PasswordConfigProperty.
Storing a password with just "user=foo" as lookup attributes is problematic
because it is too unspecific. Different services or configs with the same
user, but different passwords end up overwriting each other's passwords. In
practice, the config with "user=foo" even had the effect of removing the entry
for "user=foo server=bar".
The situation can be avoided by using the remotePeerId as fallback when the
syncURL is empty. There is a (minor?) risk that some configs were stored
in the past without that additional key and now won't be found anymore in the
keyring. Users need to re-set such passwords.
If an attempt is made to store a password with insufficient lookup attributes,
GNOME keyring will now reject the attempt.
When instantiating multiple SyncConfig instances, it is important that
they share filter nodes and the underlying tree, because the nodes
store copies of already retrieved credentials (lookup shall only be
done once!) and the trees represent the current content of the config
(which must be consistent when making changes).
Currently the new code is not thread-safe, but nor are nodes and trees,
so a lot more work would be needed to make this safe. Instead we avoid
concurrency.
SyncConfigTest::normalize() only passed because FileConfigTree accidentally
created the "peers" directory inside the peer. That will change, so don't rely
on that. Instead ensure that the config.ini file of the peers gets written
because it contains something.
Instantiating LogDirTest used to create a SyncContext and use that as logger
for the entire duration of testing inside client-test, even when not running
LogDirTest tests at all. This is undesirable and together with caching of the
config tree while in use, broke some other tests (EvolutionCalendarTest)
because obsolete DB names were shared.
It is better to create the context during setUp() and remove it in tearDown().
The previous approach made FileConfigTree more complex than necessary.
Having an abstract ConfigTree::getRootPath() with undefined behavior
is bad design.
The code was had undesiredable side effects: inside a peer config,
another "peers" directory was created because FileConfigTree didn't
know whether creating that dir was required or not.
Now all of the complexity is in SyncConfig, which arguably knows
better what the tree stands for and how to use
it.
In practice, the methods are always called for a specific SyncConfig.
Passing that allows removing several other parameters and, more
importantly, also grants access to the config and through that other
configs. This will be needed for the indirect credential lookup.
"username", "proxyUsername" and "databaseUser" used to be simply a
string containing the name of the respective user or (in the case of
the ActiveSync backend) the account ID in gconf.
Now it is also possible to provide credentials (username + password)
indirectly: when any of these properties is set to "id:<config name>",
then the "username/password" properties in that config are used
instead. This is useful in particular with WebDAV, where credentials
had to be repeated several times (target config, in each database when
used as part of SyncML) or when using a service which requires several
configs (Google via SyncML and CalDAV).
For user names which contain colons, the new "user:<user name>" format
must be used. Strings without colons are assumed to be normal user
names.
This commit changes the SyncConfig APIs for this extension. More work
is needed to make the indirect lookup via "id" functional.
Passwords are cached after the initial check as temporary property
values. The explicit string members are obsolete and can be removed
together with the code using them.
Using asynchronous method calls did not eliminate the default timeout,
as expected. In particular, SyncPeer() still timed out when syncing many
contacts.
Instead set up all necessary parameters (= callbacks and now also a
long timeout) in a hash and pass that to all D-Bus method calls. It's
less code duplication, too.
Using the underscore in the UID has been wrong all along, it only
happened to work because UID sanity checking was missing. After adding
it, the example broke.
Now simply remove the colon. It makes the UID less readable, but it
doesn't have to be, and ensures that file names and database names
contain the UID as-is.
Only saw this once in nightly testing and couldn't reproduce it:
$ make -j 16
perl /data/runtests/work/sources/syncevolution/src/syncevo/readme2c.pl
/data/runtests/work/sources/syncevolution/README.rst
>src/syncevo/CmdlineHelp.c
/usr/bin/xsltproc -o src/dbus/interfaces/syncevo-server-doc.xml
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/spec-to-docbook.xsl
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/syncevo-server-full.xml
/usr/bin/xsltproc -o src/dbus/interfaces/syncevo-connection-doc.xml
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/spec-to-docbook.xsl
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/syncevo-connection-full.xml
/usr/bin/xsltproc -o src/dbus/interfaces/syncevo-session-doc.xml
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/spec-to-docbook.xsl
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/syncevo-session-full.xml
/usr/bin/glib-genmarshal
/data/runtests/work/sources/syncevolution/src/dbus/glib/syncevo-marshal.list
--header --prefix=syncevo_marshal > src/dbus/glib/syncevo-marshal.h
runtime error
xsltApplyStylesheet: saving to src/dbus/interfaces/syncevo-session-doc.xml may
not be possible
/usr/bin/xsltproc -o src/dbus/glib/syncevo-server.xml
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/spec-strip-docs.xsl
/data/runtests/work/sources/syncevolution/src/dbus/interfaces/syncevo-server-full.xml
runtime error
xsltApplyStylesheet: saving to src/dbus/interfaces/syncevo-server-doc.xml may
not be possible
make: *** [src/dbus/interfaces/syncevo-server-doc.xml] Error 9
make: *** Deleting file `src/dbus/interfaces/syncevo-server-doc.xml'
make: *** Waiting for unfinished jobs....
make: *** [src/dbus/interfaces/syncevo-session-doc.xml] Error 9
make: *** Deleting file `src/dbus/interfaces/syncevo-session-doc.xml'
Looks like multiple xsltproc commands ran in parallel and then stepped on each
others toes while creating the src/dbus/interfaces directory, which does not
exist after an out-of-tree configure.
To address the issue, serialize creating that directory by having make create
it as a prerequisite.
While there are sessions pending or active, the server should not shut down.
It did that while executing a long-running PIM Manager SyncPeer() operations,
by default after 10 minutes.
This was not a problem elsewhere because other operations are associated with
a client, whose presence also prevents shutdowns. Perhaps PIM Manager should
also track the caller and treat it like a client.
Like everything else that waits for a certain event on the main loop,
SYNCEVO_GLIB_CALL_SYNC() should also better use GRunWhile(). This is
necessary to be usable in threads.
obexd 0.48 is almost the same as obexd 0.47, except that it dropped
the SetFilter and SetFormat methods in favor of passing a Bluex 5-style
filter parameter to PullAll.
SyncEvolution now supports 4, in words, four different obexd
APIs. Sigh.
The code works with gSSO (https://01.org/gsso). With some tweaks to
the configure check and some ifdefs it probably could be made to work
Ubuntu Online Accounts.
The code depends on an account accessible via libaccounts-glib which
has a provider and and (optionally) services enabled for that
provider. It is not necessary that the account already has a signon
identity ID, the backend will create that for the provider (and thus
shared between all services) if necessary.
Therefore it is possible to use the ag-tool to create and enable the
account and services. Provider and service templates are in the next
commit.
If given an AuthProvider which can handle OAuth2, then OAuth2 is
used instead of plain username/password authentication.
Obtaining the OAuth2 token must be done at a point where we can still
abort the request. If obtaining the token fails, then this should be
considered a fatal error which aborts scanning for resources. Other
errors cause the current URL to be skipped while scanning continues.
This commit moves the "execute request" functionality back into the
Neon::Session class, because that is where most of the logic (retry
request?) and state is (access tokens which persist across requests).
The real username is only relevant when running a sync. When looking
at a config with a D-Bus client like the GTK UI, the username should
always be "id:<config>", to avoid accidentally removing the
indirection, while the password should be the real one, to allow the
user to edit like he normally would with passwords stored in a
keyring.
To achive this, overriding the username must be suppressed when
resolving as part of the D-Bus config API. While at it, move the
entire "iterate over properties" into a common utility function in
PasswordConfigProperty.
A SHARED_LAYOUT config tree caches config nodes. Allow a second config
to use those same nodes as an already existing config. This will be
useful in combination with indirect password lookup, because then the
credentials can be stored as temporary property values and be reused
when used multiple times in a process (for example, by CardDAV and by
CalDAV).
In practice, the methods are always called for a specific SyncConfig.
Passing that allows removing several other parameters and, more
importantly, also grants access to the config and through that other
configs. This will be needed for the indirect credential lookup.
"username", "proxyUsername" and "databaseUser" used to be simply a
string containing the name of the respective user or (in the case of
the ActiveSync backend) the account ID in gconf.
Now it is also possible to provide credentials (username + password)
indirectly: when any of these properties is set to "id:<config name>",
then the "username/password" properties in that config are used
instead. This is useful in particular with WebDAV, where credentials
had to be repeated several times (target config, in each database when
used as part of SyncML) or when using a service which requires several
configs (Google via SyncML and CalDAV).
For user names which contain colons, the new "user:<user name>" format
must be used. Strings without colons are assumed to be normal user
names.
This commit changes the SyncConfig APIs for this extension. More work
is needed to make the indirect lookup via "id" functional.
Passwords are cached after the initial check as temporary property
values. The explicit string members are obsolete and can be removed
together with the code using them.
Like everything else that waits for a certain event on the main loop,
SYNCEVO_GLIB_CALL_SYNC() should also better use GRunWhile(). This is
necessary to be usable in threads.
When clients like the GTK sync-ui stored a password, it was always
stored as plain text in the config.ini file by the
syncevo-dbus-server. The necessary code for redirecting the password
storage in a keyring (GNOME or KWallet) simply wasn't called in that
case.
The command line tool, even when using the D-Bus server to run the
operation, had the necessary code active and thus was not affected.
uint16 happened to work when compiling with a recent GNOME stack, but
without that uint16 is not defined. The right approach is to use
stdint.h and uint16_t.
Both maintaining the map items inside the Synthesis engine and storing
them in .ini hash config nodes inside SyncEvolution are fairly heavy
operations which are not needed at all during an ephemeral sync (= no
meta data stored, done by the PIM Manager when triggering a pure PBAP
sync).
Using the new Synthesis CA_ResumeSupported DB capability it is
possible to suppress these operations without having to fiddle with
the Synthesis DB API that SyncEvolution provides. To make it possible
at the DB layer to detect that the meta data is not needed, the
ConfigNode passed to it must be marked as volatile.
This change sped up a sync with 10000 unmodified, known items from 38s
to 23s.
The synthesisID value is required for each Synthesis source progress
event, which can be fairly frequent (more than one per item). Instead
of going down to the underlying .ini config node each time, cache the
value in the SyncSourceConfig layer.
Syncing was slowed down by fowarding all log messages from the local
sync helper to its parent and from the D-Bus helper to
syncevo-dbus-server. Quite often, the log messages then were simply
discarded by the recipient. To speed up syncing, better filter at the
source.
The syncevo-dbus-helper is told which messages are relevant by
forwarding the syncevo-dbus-server "D-Bus log level" command line
setting to the helper process as part of its argv parameters.
The synevo-local-sync helper applies its own log level now also to the
messages sent to the parent. This ensures that messages stored in the
client log also show up in the parent log (unless the parent has more
restrictive settints, which is uncommon) and that INFO/SHOW messages
still reach the user.
For each incoming change, one INFO line with "received x[/out of y]"
was printed, immediately followed by another line with total counts
"added x, updated y, removed z". For each outgoing change, a "sent
x[/out of y]" was printed.
In addition, these changes were forwarded to the D-Bus server where a
"percent complete" was calculated and broadcasted to clients. All of
that caused a very high overhead for every single change, even if the
actual logging was off. The syncevo-dbus-server was constantly
consuming CPU time during a sync when it should have been mostly idle.
To avoid this overhead, the updated received/sent numbers that come
from the Synthesis engine are now cached and only processed when done
with a SyncML message or some other event happens (whatever happens
first).
To keep the implementation simple, the "added x, updated y, removed z"
information is ignored completely and no longer appears in the output.
As a result, syncevo-dbus-server is now almost completely idle during
a running sync with no log output. Such a sync involving 10000 contacts
was sped up from 37s to 26s total runtime.
Set SYNCEVOLUTION_DBUS_HELPER_VGDB=1, add --vgdb-error=1 --vgdb=yes
to VALGRIND_ARGS, run test, wait for vgdb message in valgrind*.out files,
attach as instructed.
With --vgdb-error=0, all processes block during startup, waiting for
the debugger to attach.
The previous attempt with concurrent reading while listing IDs did not
work, that listing must complete before the SyncML client contacts the
server. What works is transfering and parsing after the engine starts
to ask for the actual data.
For that we need to list IDs in advance. We use GetSize() for that.
If contacts get deleted while we read, getting the data for the
contacts at the end of the range will fail with 404, which is
understood by the Synthesis engine and leads to ignoring the ID, as
intended.
If contacts get added while we read, we will ignore them even if they
happen to be in the result of PullAll. The next sync will include
them.
When doing a PBAP sync, PIM manager asks the D-Bus sync helper to set
its SYNCEVOLUTION_PBAP_SYNC to "incremental". If the env variable
is already set, it does not get overwritten, which allows overriding
this default.
Depending on the SYNCEVOLUTION_PBAP_SYNC env variable, syncing reads
all properties as configured ("all"), excludes photos ("text") or
first text, then all ("incremental").
When excluding photos, only known properties get requested. This
avoids issues with phones which reject the request when enabling
properties via the bit flags. This also helps with
"databaseFormat=^PHOTO".
When excluding photos, the vcard merge script as used by EDS ensures
that existing photo data is preserved. This only works during a slow
sync (merge script not called otherwise, okay for PBAP because it
always syncs in slow sync) and EDS (other backends do not use the
merge script, okay at the moment because PIM Manager is hard-coded to
use EDS).
The PBAP backend must be aware of the PBAP sync mode and request a
second cycle, which again must be a slow sync. This only works because
the sync engine is aware of the special mode and sets a new session
variable "keepPhotoData". It would be better to have the PBAP backend
send CTCap with PHOTO marked as not supported for text-only syncs and
enabled when sending PHOTO data, but that is considerably harder to
implement (CTCap cannot be adjusted at runtime).
beginSync() may only ask for a slow sync when not already called
for one. That's what the command line tool does when accessing
items. It fails when getting the 508 status.
The original goal of overlapping syncing with download has not been
achieved yet. It turned out that all item IDs get requested before
syncing starts, which thus depends on downloading all items in the current
implementation. Can be fixed by making up IDs based on the number of
existing items (see GetSize() in PBAP) and then downloading later when
the data is needed.
If PHOTO and/or GEO were the only modified properties during a slow
sync, the updated item was not written into local storage because
they were marked as compare="never" = "not relevant".
For PHOTO this was intentional in the sample config, with the
rationale that local storages often don't store the data exactly as
requested. When that happens, comparing the data would lead to
unnecessary writes. But EDS and probably all other local SyncEvolution
storages (KDE, file) store the photo exactly as requested, so not
considering changes had the undesirable effect of not always writing
new photo data.
For GEO, ignoring it was accidental.
A special merge script handles EDS file:// photo URIs. When the
loosing item has the data in a file and the winning item has binary
data, the data in the file may still be up-to-date, so before allowing
MERGEFIELDS() to overwrite the file reference with binary data and
thus forcing EDS to write a new file, check the content. If it
matches, use the file reference in both items.
Performance is improved by requesting multiple contacts at once and
overlapping reading with processing. On a fast system (SSD, CPU fast
enough to not be the limiting factor), testpim.py's testSync takes 8
seconds for a "match" sync where 1000 contacts get loaded and compared
against the same set of contacts. Read-ahead with only 1 contact per
query speeds that up to 6.7s due to overlapping IO and
processing. Read-ahead with the default 50 contacts per query takes
5.5s. It does not get much faster with larger queries.
While returning items from one cache populated with a single
e_book_client_get_contacts() call, another query is started to overlap
processing and loading.
To achieve efficient read-ahead, the backend relies on the hint given
to it via setReadAheadOrder(). As soon as it detects that a contact is
not the next one according to that order, it switches back to reading
one contact at a time. This happens during the write phase of a sync
where the Synthesis engine needs to read, update, and write back
changes based on updates sent by the peer.
Cache statistics show that this works well for --print-items, --export
and slow syncs.
Writing into the database must invalidate the corresponding cached
contact. Otherwise the backup operation after a sync may end up
reading stale data.
Trying to predict in the SyncSource which items will be needed is
hard. It depends what the item is retrieved for (sync or
backup/printing) and on the sync mode (during a sync).
Instead of guessing, better have the user of the source tell the
source in advance what it will read. In most cases this is cheap
because it does not involve copying of items luids ("read all items",
"read new or modified items"). The source then can use its own internal
mechanisms to figure out what that means.
Only extracting specific items specified on the command line and the
backup operation must explicitly pass an ordered list of luids. For
the sake of simplicity, do that in a std::vector even though it
involves copying.
The
while (<something>) g_main_context_iterate(NULL, true);
pattern only works in the main thread. As soon as multiple
threads are allowed to process events, race conditions occur,
as described before.
But the pattern is useful, so support it in all threads by
shifting the check into the main thread, which will then notify
other threads via a condition variable when something changes.
gcc complained about the "protected" m_mutex access. Apparently
it did not realize that the access was to a base class. An explicit
get() solves that. Another way to get the pointer is a type cast.
Derive from SyncSource and SyncSourceSession directly instead of going
through TrackingSyncSource. This allows removing several dummy methods
that we have to implement for TrackingSyncSource and allows
reporting existing items incrementally, instead of having to load all
of them at once for the old listAllItems().
Contacts are now already reported to the engine while their transfer
still runs. That depends on monitoring the temporary file, remapping
the larger file and continuing parsing where the previous parsing
stopped.
This only works with obexd 0.47 and obexd from Bluez 5, because it
depends on the temporary file interface. The older PullAll did not
return any data until it had downloaded everything.
Because it isn't known when the contact data will be needed, the backend
still maintains the mapping from ID to vCard data for all contacts seen
in the current session. Because that memory is backed by a temporary file
system, unused memory can be swapped out (and in) well by the OS.
If the file is in a ram-based temp file system, then it may also not
matter at all that the file gets mapped multiple times.
Instead of reading all item IDs, then iterating over them, process
each new ID as soon as it is available. With sources that support
incremental reading (only the PBAP source at the moment) that provides
output sooner and is a bit more memory efficient.
remove() is useful when want to continue using the file but also want
to ensure that it gets deleted when we crash.
moreData() can be used to determine if the file has grown since the
last time that map() was called. In other words, it supports
processing data while obexd still writes into the file.
The previous commit "PBAP: fix support for obexd >= 0.47 and < Bluez 5"
made the backend work with obexd 0.48 and broke it with 0.47. That's because
there was another API change between 0.47 and 0.48, which wasn't known
at the time of that commit.
SyncEvolution now works with 0.47 and does not work with 0.48. This choice
was made because 0.47 supports the file-based data transfer (same as in Bluez 5
and thus useful for testing when Bluez 5 is not available) and 0.47 still
compiles against older Bluez versions (which makes it easier to use than 0.48).
More recent GNOME Python bindings are provided by gobject
introspection. The traditional gobject/glib modules no
longer exist.
The API is similar enough that we just need to adapt importing: if
importing the normal modules fails, try importing from gi.repository
instead.
EDS 3.8 sets X-EVOLUTION-FILE-AS to "last, first" when a contact is
created without it. This leads again to unnecessary DB updates because
the incoming item that the engine works with doesn't have that field
set.
To mitigate that issue, set FILE_AS (renamed to make the field name
valid in a script) like EDS would do during writing.
The downside is that all incoming items now have FILE_AS set, which
will overwrite a locally stored copy of that property even if the peer
did not store X-EVOLUTION-FILE-AS. Previously, as tested by
testExtension, the local value was preserved. There is no good solution
that works for both use cases, so allow X-EVOLUTION-FILE-AS to get lost
and relax the test.
Traditionally, contacts were modified shortly before writing into EDS
to match with Evolution expectations (must have N, only one CELL TEL,
VOICE flag must be set). During a slow sync, the engine compare the
modified contacts with the unmodified, incoming one. This led to
mismatches and/or merge operations which end up not changing anything
in the DB because the only difference would be removed again before
writing.
To avoid this, define datatypes which include the EDS modifications in
its <incomingscript>. Then the engine works with an item as it would
be written to EDS and correctly determines that the incoming and DB
items are identical.
Found in testpim.py's testSync when applied to the test cases
generated by the Openismus test case generator.
If the sources are used in a mode which doesn't need revision strings,
then skip the retrieval of items from the EDS daemons when we would
normally do it to get the revision string.
Recording the revision of items during a caching sync is unnecessary
because the next sync will not use the information. For item
modifications, the information was not even recorded.
Now a "don't need changes" mode can be set in SyncSource, which is
done for caching and command line item operations. This is better than
second-guessing the mode in which a source is used.
SyncSourceRevision checks that flag and skips updating the
luid->revision mapping if set.
Individual backends can also check the flag and skip calculating the
revision to begin with.
Only works when compiled for EDS >= 3.6. Uses the new ITEM_AGAIN in
SyncSourceSerialize. Combines multiple add/update operations into one
asynchronous batch operation which can overlap with network IO if
the engine supports it.
To take full advantage of batch operations, we must enable out-of-order
execution of commands. Otherwise the engine will never issue more than
one change to us. Do it for SyncEvolution as peer.
TODO: do it for all peers or disable batched processing when it is not
enabled.
SyncSourceSerialize uses ITEM_AGAIN plus a callback in
InsertItemResult to indicate that and how an insert (aka add or
update) operation must be continued.
TrackingSyncSource merely passes that result through until it finally
completes.
Direct, raw access to items as done by the command line and restore
operations is not done asynchronously. The backend does not need to
know how it is used, the SyncSourceSerialize wrapper around the
backend will flush and wait.
Asynchronous deleting is not supported. It would require throwing an
exception or an API change (better - exceptions are for errors!)
because removeItem() has no return code which could be extended.
The wrapper around the actual operation checks if the operation
returned an error or result code (traditional behavior). If not, it
expects a ContinueOperation instance, remembers it and calls it when
the same operation gets called again for the same item.
For add/insert, "same item" is detected based on the KeyH address,
which must not change. For delete, the item local ID is used.
Pre- and post-signals are called exactly once, before the first call
and after the last call of the item.
ContinueOperation is a simple boost::function pointer for now. The
Synthesis engine itself is not able to force completion of the
operation, it just polls. This can lead to many empty messages with
just an Alert inside, thus triggering the "endless loop" protection,
which aborts the sync.
We overcome this limitation in the SyncEvolution layer above the
Synthesis engine: first, we flush pending operations before starting
network IO. This is a good place to batch together all pending
operations. Second, to overcome the "endless loop" problem, we force
a waiting for completion if the last message already was empty. If
that happened, we are done with items and should start sending our
responses.
Binding a function which returns the traditional TSyError still works
because it gets copied transparently into the boost::variant that the
wrapper expects, so no other code in SyncSource or backends needs to
be adapted. Enabling the use of LOCERR_AGAIN in the utility classes
and backends will follow in the next patches.
Check conditions each time they change as part of view updates. This
would have helped to find the invalid check earlier and catches
temporary violations of the invariants.
The assertEqual() expected 'quiesence' to be seen, but runUntil() was not
asked to run until that was seen. Therefore tests failed randomly depending on
timing: they passed if the quiescence signal was processed in the same event
loop run as the change that was waited for and failed if it arrived later.
Fixed by waiting for 'quiesence' explicitly. It should arrive last.
When "Abraham" falls out of the view, it is possible that we have temporarily
no contacts in the view, because first "Abraham" gets reviewed and then
"Benjamin" gets added. The check "at least one contact" therefore was too
strict.
So far, a "check" callback was passed to runUntil() which checked the state
after each main loop run and at regular time intervals. The downside was that
an incorrect state change inside a single main loop run (for example, removing
a contact and adding one in testMerge) was not detected.
To increase coverage of state changes, the check is now also invoked after
each method call in the ViewAgent. If a check fails, it gets recorded as
error in the view and then will get noticed outside of the event loop by the
check that is passed to runUntil().
These checks needs to be installed before running the command line which
changes the state, not just as part of runTime(), because running the command
line also processes glib events.
This insufficient coverage was found when changes in timing caused testMerge
to fail: the previous check for "always exactly one contact" turned out to be
too strict and merely passed when remove + add were processed together. In
practice, the contact's ID changes because folks ties the merged contact to
a different primary store than the unmerged, original contact.
TESTPIM_TEST_SYNC_TESTCASES can be set to the name of a file
containing contacts in vCard 3.0 format with an empty line as
separator. In this mode, testSync() measures the duration of cleaning
the local cache, importing all test cases, matching them against the
cache (which must not write into the cache!) and removing the cached
contacts.
Typically the peer configs get created from scratch, in particular
when testing with testpim.py. In that case the log level cannot be set
in advance and doing it via the D-Bus API is also not supported.
Therefore, for debugging, use SYNCEVOLUTION_LOGLEVEL=<level> to create
peers with a specific log level.
