The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Raising exceptions via throwError() looses the original source code
location information. Fixing that by explicitly passing that
information as additional parameter, created with the preprocessor
macro SE_HERE.
Several files included the complex SyncContext.h only because needed
throwError(). A better place for the revised implementation is the
Exception class which used to be in util.h, now Exception.h.
Simplifying the include statements exposed indirect include
dependencies and removed "using namespace std" from the compilation of
some source files which happened to rely on it. We want to get rid of
that name space polution, so fix the code instead of adding it back.
cppcheck complains about sub-optimal performance (std::string should be passed
via const reference, not by value). It is not critical here, but for the sake
of getting clean scan results let's fix the issues...
One of the offending methods was never implemented, nor called. Removed.
The Collection class did not initialize all its members. Even if the code
never relied on them being initialized (not checked), it is better to be
safe and predictable, which means initializing all members in the constructor.
"username", "proxyUsername" and "databaseUser" used to be simply a
string containing the name of the respective user or (in the case of
the ActiveSync backend) the account ID in gconf.
Now it is also possible to provide credentials (username + password)
indirectly: when any of these properties is set to "id:<config name>",
then the "username/password" properties in that config are used
instead. This is useful in particular with WebDAV, where credentials
had to be repeated several times (target config, in each database when
used as part of SyncML) or when using a service which requires several
configs (Google via SyncML and CalDAV).
For user names which contain colons, the new "user:<user name>" format
must be used. Strings without colons are assumed to be normal user
names.
This commit changes the SyncConfig APIs for this extension. More work
is needed to make the indirect lookup via "id" functional.
"username", "proxyUsername" and "databaseUser" used to be simply a
string containing the name of the respective user or (in the case of
the ActiveSync backend) the account ID in gconf.
Now it is also possible to provide credentials (username + password)
indirectly: when any of these properties is set to "id:<config name>",
then the "username/password" properties in that config are used
instead. This is useful in particular with WebDAV, where credentials
had to be repeated several times (target config, in each database when
used as part of SyncML) or when using a service which requires several
configs (Google via SyncML and CalDAV).
For user names which contain colons, the new "user:<user name>" format
must be used. Strings without colons are assumed to be normal user
names.
This commit changes the SyncConfig APIs for this extension. More work
is needed to make the indirect lookup via "id" functional.
Passing an explicit Logger instance to the SE_LOG* macros was hardly
ever used and only made the macros more complex.
The only usage of it was in some backends, which then added a prefix
string automatically. The same effect can be achieved by passing
getDisplayName(). Exception handling no longer needs a Logger instance,
the prefix alone is enough.
The other intended usage, avoiding global variables for logging by
passing a logger known to the caller, was not possible at all.
To make prefix handling more flexible, it is now passed as a "const
std::string *" instead of a "const char *".
Previously, the database field was interpreted as a Collection ID. This adds
logic to allow the database to be interpreted as a folder path. The logic is:
1) If the database is an empty string, pass it through (this is the most
common case as it is interpreted as "use the default folder for the
source type").
2) If the database matches a Collection ID, use the ID (this is the same as
the previous behaviour).
3) If the database matches a folder path name, with an optional leading "/",
use the Collection ID for the matching folder.
4) Otherwise, force a FolderSync to get the latest folder changes from the
server and repeat steps 2 and 3
5) If still no match, throw an error.
Steps 2 and 3 are in the new function lookupFolder. The remaining logic has
been added to the open function. Note that the result is that m_folder (and
hence getFolder()) are always either empty or a Collection ID -- that is as
before so the sync logic itself is unchanged.
A new method, findCollections, fetches the folder list from the server and
creates two maps:
m_collections - store all the information about each collection (folder),
indexed by server collection ID
m_folderPaths - map full folder paths to collection IDs
getDatabases uses this data to returns the folder path, collection ID and a
flag indicating if the folder is the default for that type.
Note 1: getDatabases always asks activesyncd to update the folder list from the
server in order to return up to date information.
Note 2: this depends on a new libeasclient routine:
eas_sync_handler_get_folder_list
This reverts commit 2735273ec60b289c5ec2c49f3eacb9d7d04d5ea1.
With this patch, setting up ActiveSync fails with Google as server.
Needs further investigation.
Also note the explicit g_object_unref() and EASFolderUnref - these
should get replace with SE_GOBJECT_TYPE smart pointers.
This reverts commit 7327b23a4dd31abdc9596916743892402bcffe0c.
Depends on 273527 "ActiveSync: added getDatabases support for fetching
folder list" which has to be reverted.
Previously, the database field was interpreted as a Collection ID. This adds
logic to allow the database to be interpreted as a folder path. The logic is:
1) If the database is an empty string, pass it through (this is the most
common case as it is interpreted as "use the default folder for the
source type").
2) If the database matches a Collection ID, use the ID (this is the same as
the previous behaviour).
3) If the database matches a folder path name, with an optional leading "/",
use the Collection ID for the matching folder.
4) Otherwise, force a FolderSync to get the latest folder changes from the
server and repeat steps 2 and 3
5) If still no match, throw an error.
Steps 2 and 3 are in the new function lookupFolder. The remaining logic has
been added to the open function. Note that the result is that m_folder (and
hence getFolder()) are always either empty or a Collection ID -- that is as
before so the sync logic itself is unchanged.
A new method, findCollections, fetches the folder list from the server and
creates two maps:
m_collections - store all the information about each collection (folder),
indexed by server collection ID
m_folderPaths - map full folder paths to collection IDs
getDatabases uses this data to returns the folder path, collection ID and a
flag indicating if the folder is the default for that type.
Note 1: getDatabases always asks activesyncd to update the folder list from the
server in order to return up to date information.
Note 2: this depends on a new libeasclient routine:
eas_sync_handler_get_folder_list
The backends had SYNCEVOLUTION_LIBS in their _DEPENDENCIES entries,
which is wrong because SYNCEVOLUTION_LIBS must include -lrt (which
can't be a dependency).
Fixed by depending on libsyncevolution.la directly.
Instantiating an ActiveSyncSource for testing must work with
empty database names now. Testing no longer forces the
database to be set.
While at it rewrote the code to avoid the explicit pointer.
A backend cannot know whether it'll be used together
with a client-test which supports integration testing.
Therefore the backend should always register its tests.
Updating failed when using Google because the Synthesis engine
tried to read the existing item in order to merge it with
the update. This failed because Google does not implement the
Fetch command.
Pretending to update the item intelligently avoids that. It
also helps to improve performance of updates with Exchange.
The downside is that syncing with local storages which do
not support all ActiveSync fields will cause data loss.
Need to check whether Exchange-only attributes get lost
also when the local storage supports everything, for
example because activesynd unintentionally removes data.
The error message for "Invalid synchronization key" changed in
activesyncd, now it contains the D-Bus error type as prefix. Fixed by
doing a substring search.
Also, not freeing the GError before trying again is a bug. Apparently
that was ignored earlier, now it triggers an assert.
As mentioned by Tino Keitel on the mailing list, some libs and
executables were only implicitly linked against libraries that they
called directly. This happened to work by chance because these libraries
ended up in the running executable anyway, due to indirect loading.
To catch such problems, the "make installcheck" was extended:
dpkg-shlibdeps is run, if available, and the error output is scanned
for the messages which indicate that a symbol is used without linking
to the right library (example output below).
Had to fix quite a few _LIBADD lines to pass the test.
Some exceptions are allowed:
- libsmltk depends on the caller providing SySync logging support.
- libneon is intentionally not linked explicitly for syncevolution.org
binaries, to make resulting binaries work with GNUTLS and OpenSSL.
dpkg-shlibdeps: warning: debian/syncevolution-libs/usr/lib/syncevolution/backends/syncdav.so contains an unresolvable reference to symbol icalparameter_new_from_value_string: it's probably a plugin.
dpkg-shlibdeps: warning: 51 other similar warnings have been skipped (use -v to see them all).
...
dpkg-shlibdeps: warning: symbol dlsym used by debian/libsyncevolution0/usr/lib/libsyncevolution.so.0.0.0 found in none of the libraries.
dpkg-shlibdeps: warning: symbol dlerror used by debian/libsyncevolution0/usr/lib/libsyncevolution.so.0.0.0 found in none of the libraries.
dpkg-shlibdeps: warning: symbol dlopen used by debian/libsyncevolution0/usr/lib/libsyncevolution.so.0.0.0 found in none of the libraries.
Google doesn't seem to support the Fetch operation, which is used
during testing to retrieve unchanged items. During syncing it will
only be needed when merging incoming data with an existing item, which
should not be necessary... except that testing shows that it is
necessary. The case where it is needed is:
- incremental sync (cache empty)
- calendar event series needs to be modified
- items from that series need to be fetched in preparation
for updating it
To allow testing, several workarounds are necessary:
- request *all* data before doing a data dump in testImport (and friends),
to ensure that the cache in the backend is fully populated
- use the cache in ActiveSyncCalendarSource instead of accessing
the base class directly, because that would trigger a Fetch
Accessing the base class is still useful for Exchange+calendar,
because testing then avoids the cache (and thus can expose bugs
in it).
While at it, adapted the README. "database" needs to be set
explicitly, "client-test" only does it when creating configs.
Both begin() methods grab the item node and use it for tracking.
That must only be done once, otherwise the second cycle uses an empty
Boost pointer (=> exception).
The main goal is to test CalDAV/CardDAV sources as part
of a SyncML client and/or server. A test involving syncevo-http-server
is now named "<client storage><server storage>":
- edsfile = EDS in client, file in server (used to be syncevohttp)
- davfile = CalDAV/CardDAV in client, file in server (new)
- edsdav = EDS in client, CalDAV/CardDAV in server (new)
For this, WebDAVSourceRegister.cpp must be able to create test sources
which match the client 1/2 sync configs. The client "1" or "2" strings
are passed through the abstract ClientTest into the source A/B create
callbacks. WebDAVSourceRegister.cpp cannot get this directly from
ClientTest because it lives in a plugin which is not necessarily
linked against ClientTest.
A conceptual change is that CLIENT_TEST_EVOLUTION_PREFIX/USER/PASSWORD
no longer override existing config properties. That is necessary
because the shared prefix is too simplistic for WebDAV (needs full URL
in "database"); also helps KDE (needs resource URI). The env variables
and the default "SyncEvolution_Test_" value for the database prefix are
still used if the config does not exist. That is useful to prevent
accidentally running client-test against the default databases.
The nightly setup script might (should!?) be made public to simplify
configuring a server.
Another change is the user-configurable part of client-test now lives
entirely in the _1/_2 client sync configs and contexts. From there the
source properties are copied into the Client::Source context each time
client-test runs.
The special semantic of the former RegisterSyncSource::InactiveSource
(invalid pointer of value 1) caused bugs, like using it in
--print-databases (=> segfault) or not being able to store the result
of a createSource() directly in a smart pointer (=> potential leak in
SyncSource::createSource()).
Obviously a bad idea to start with. Replaced with a
RegisterSyncSource::InactiveSource() method which creates a real,
inactive SyncSource instance which can and must be deleted by the
caller.
This is a SyncSource API change for backend developers.
Instead of RegisterSyncSource::InactiveSource, return
RegisterSyncSource::InactiveSource().
Comparisons against RegisterSyncSource::InactiveSource needs
to be replaced with a call to the new SyncSource::isInactive().
User visible fixes:
* --print-databases: no longer crashes when EDS or KDE backends
are not usable. Instead it prints "not enabled during compilation or
not usable in the current environment".
* --print-databases: continues with other backends even if
one backend throws an exception, as the KDE backend does
when it cannot find Akonadi. Error messages are printed.
The camel backend failed to compile with latest EDS from master
(a 3.3.99 pre-release). Disabling compilation of camel
backend and eplugin (which might still compile, but wasn't
tested anyway in the nightly builds), to keep testing of activesyncd
for contacts and calendar working.
ClientSourceRevisions needs to be reset() before adding new items
while doing change detection. CalDAVSource also didn't properly
populate its m_cache in following cycles because the cache was assumed
to be set already.
Re-insert the recurrence ID if not provided as part of the
data when updating an item. Necessary to support dumb local
storage which doesn't support UID/RECURRENCE-ID.
SyncEvolution now requires that "item not found" errors have
a 404 status code; the Synthesis engine depends on recognizing
such errors in some cases.
For calendars, the backend does all the checking itself and
doesn't event talk to the server.
For contacts, the same is done for deleting (because the
server accepts a delete request for a non-existent item
without complaining) and also detects the "not found error"
when reading (which would be nicer if it could be done
without string comparisons, but that doesn't seem to work yet).
The plain "char *" sync key used to be for a static buffer, now
is allocated dynamically. Transfer ownership of the buffer
to our eptr smart pointer, which will also throw an error if the
pointer is NULL.
activesyncd gets compiled completely (which should always succeed) and
then only the required pieces are installed (without the things which
are hard-coded for /usr, because installing those will fail).
Testing is done by running a similar set of client-test tests as for remove
CalDAV/CardDAV servers.
activesyncd is started anew each time client-test is run, done in
the new wrappercheck.sh script. Can be combined with valgrindcheck.sh:
wrappercheck.sh valgrindcheck.sh activesyncd -- valgrindcheck.sh client-test ...
The return code of wrappercheck.sh is the return code of the real command
or the daemon, if the real command succeeded. This way the special 100 return
code from valgrindcheck.sh is returned if the only problems were memory
issues.
The default implementation removes one VEVENT after the other, which
is slow for large merged events. Directly removing the entire event
series is faster.
Calendar and generic ActiveSync source now use the same logic in
beginSync(). New is a workaround for Google, which seems to require
that eas_sync_handler_get_items() gets called twice at the start of a
slow sync.
The unconditional change of calling StartDataRead (aka beginSync)
early enough so that ActiveSync can force a slow sync had negative
consequences, because now it was called before the peer was contacted
and credentials were accepted:
- broke the "sync started successfully" logic, resulting in
notifications for syncs which were supposed to be retried silently
(showed up in TestSessionAPIsDummy.testAutoSyncNetworkFailure)
- database dumps were done even if not needed because sync never
starts
Now all backends are called as before unless they explicitly ask for
the early call. The ActiveSync backend does that. The downsides of
that approach do not matter much because syncing will start okay and
dumping of data is typically disable on that side of a local sync.
The ActiveSync backend now detects the daemon's "Sync error: Invalid
synchronization key" and falls back to a slow sync. This is only done
if the sync key was already invalid when beginSync()
started. Otherwise something fishy must be going on and it seems
prudent to rather abort the sync with an error.
It would be nice if this special error could be detected without
having to resort to a string comparison, but this is not currently
supported by libeasclient because error codes are not yet part of the
API (BMC #23618).
Various backends (Evolution, ActiveSync, WebDAV) depend on
libical. This wasn't done correctly, with the result that
--enable-activesync without --enable-evolution and --enable-webdav
failed to compile because ENABLE_ICAL was unset by the WebDAV
configure.
Now backends can request libical support by setting need_ical="yes",
then later LIBICAL_LIBS/CFLAGS, ENABLE_ICAL define and condition
variable will be set accordingly. Similar to need_glib="yes".
Conflicts:
configure.ac
test/ClientTest.cpp
test/testcases/eds_event.ics.funambol.tem.patch
Conflicts because of version number and updated test cases resp. local
delete optimization.
ActiveSync backend had to be adapted to modified InsertItemResult: now
it requests a merge when it detects duplicates, like the CalDAV backend
already did on the 1.2 branch.
The target config cannot be shared between different sync configs
unless those configs select a different gconf account config in their
"username" property. The reason is that change tracking is tied to
that gconf account, not the sync config itself.
Related to BMC #22881 "Invalid synchronization key". Shows that
endSync() isn't called when beginSync() already fails. Need
a different way of reseting the sync key.
The memset/memcpy of the embedded boost::function instances inside the
old ClientTestConfig was causing segfaults at the end of a client-test
run if compiled with optimization.
Therefore this commit turns ClientTestConfig into a proper class
containing members which initialize themselves (Bool wrapper class,
std::string), thus memset is no longer needed and used. Also added the
standard m_ prefix.
m_numItems is gone, was never set by any backend anyway and even
expected to be consistent in one test. Now CLIENT_TEST_NUM_ITEMS is
read by defNumItems() each time it is needed.
Removed "const char *" strings from method parameters. This revealed
that config.itemType (a const char *) was incorrectly passed to
insert() where the boolean "relax" parameter should have been given.
Replaced by "false" (= strict checking) even though the old code
must have run with an implicit "true" (= relaxed checking). Let's see
whether any tests fail now.
Patrick Ohly
Graham R. Cobb
David Woodhouse