Derive from SyncSource and SyncSourceSession directly instead of going
through TrackingSyncSource. This allows removing several dummy methods
that we have to implement for TrackingSyncSource and allows
reporting existing items incrementally, instead of having to load all
of them at once for the old listAllItems().
Contacts are now already reported to the engine while their transfer
still runs. That depends on monitoring the temporary file, remapping
the larger file and continuing parsing where the previous parsing
stopped.
This only works with obexd 0.47 and obexd from Bluez 5, because it
depends on the temporary file interface. The older PullAll did not
return any data until it had downloaded everything.
Because it isn't known when the contact data will be needed, the backend
still maintains the mapping from ID to vCard data for all contacts seen
in the current session. Because that memory is backed by a temporary file
system, unused memory can be swapped out (and in) well by the OS.
If the file is in a ram-based temp file system, then it may also not
matter at all that the file gets mapped multiple times.
The previous commit "PBAP: fix support for obexd >= 0.47 and < Bluez 5"
made the backend work with obexd 0.48 and broke it with 0.47. That's because
there was another API change between 0.47 and 0.48, which wasn't known
at the time of that commit.
SyncEvolution now works with 0.47 and does not work with 0.48. This choice
was made because 0.47 supports the file-based data transfer (same as in Bluez 5
and thus useful for testing when Bluez 5 is not available) and 0.47 still
compiles against older Bluez versions (which makes it easier to use than 0.48).
Traditionally, contacts were modified shortly before writing into EDS
to match with Evolution expectations (must have N, only one CELL TEL,
VOICE flag must be set). During a slow sync, the engine compare the
modified contacts with the unmodified, incoming one. This led to
mismatches and/or merge operations which end up not changing anything
in the DB because the only difference would be removed again before
writing.
To avoid this, define datatypes which include the EDS modifications in
its <incomingscript>. Then the engine works with an item as it would
be written to EDS and correctly determines that the incoming and DB
items are identical.
Found in testpim.py's testSync when applied to the test cases
generated by the Openismus test case generator.
If the sources are used in a mode which doesn't need revision strings,
then skip the retrieval of items from the EDS daemons when we would
normally do it to get the revision string.
Only works when compiled for EDS >= 3.6. Uses the new ITEM_AGAIN in
SyncSourceSerialize. Combines multiple add/update operations into one
asynchronous batch operation which can overlap with network IO if
the engine supports it.
The localized DisplayName strings must be removed explicitly. To find them,
list all properties in the main section and then remove if the prefix
matches.
There is a difference in EDS between removing the database definition
from the ESourceRegistry (which makes the data unaccessible via EDS)
and removing the actual database. EDS itself only removes the definition
and leaves the data around to be garbage-collected eventually. This is
not what we want for the PIM Manager API; the API makes a stronger
guarantee that data is really gone.
Fixed by introducing a new mode flag for the deleteDatabase() method
and deleting the directory of the source directly in the EDS backend,
if requested by the caller.
The syncevolution command line tool will use the default mode and thus
keep the data around, while the PIM Manager forces the removal of
data.
Instead of hard-coding a specific "Backend Summary Setup" in
SyncEvolution, copy the config of the system database. That way
special flags (like the desired "Backend Summary Setup" for local
address books) can be set on a system-wide basis and without having to
modify or configure SyncEvolution.
Because EDS has no APIs to clone an ESource or turn a .source file
into a new ESource, SyncEvolution has to resort to manipulating and
creating the keyfile directly.
Following the boost::instrusive_ptr example and making "add_ref =
true" the default in our CXX GLib and GObject wrappers led to some
memory leaks because it didn't enforce thinking about whether the
plain pointer is already owned by us or not.
It is better to use a mandatory enum value, ADD_REF and TRANSFER_REF,
and force explicit construction. Doing that revealed that the
assignment operator was implemented as constructing a CXX instance
with increased ref count and/or that in some places, a real leak was
caused by increasing the ref count unnecessarily.
Running under valgrind gave a false sense of security. Some of the
real leaks only showed up randomly in tests.
Avoid referencing pathProps->second when the set of paths that
PROPFINDs returns is empty. Apparently this can happen in combination
with Calypso.
The stack backtrace sent via email looked like this:
Program received signal SIGSEGV, Segmentation fault.
0x4031a1a0 in std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::find(std::string const&) const () from /usr/lib/libsyncevolution.so.0
0x4031a1a0 <_ZNKSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE4findERS1_+60>: ldr r4, [r0, #-12]
(gdb) bt
from /usr/lib/syncevolution/backends/syncdav.so
from /usr/lib/syncevolution/backends/syncdav.so
from /usr/lib/libsyncevolution.so.0
test-dbus.py now knows how to start syncevo-http-server. Therefore
it can test normal HTTP-based syncing as well as several scenarios
which fail or succeed with a slow server depending on the server's
ability to send SyncML messages while still initializing the storage.
To make the server slow, env variables are checked by the file
backend. It may matter whether open() or listAll() are slow, so test
both.
The tests expecting the 2 minute default must check whether the
feature is enabled at all in the binary that they are testing. If not,
the test cannot run. All other tests work, albeit somewhat unsafely
because they force the engine to run multithreaded when the engine was
compiled without mutex locking of global data structures.
Only one thread may handle events in the default context at any point
in time. If a second thread calls g_main_context_iteration() or
g_main_loop_run(), it blocks until the other thread releases ownership
of the context. In that case, the first thread may wake up because of
an event that the second thread waits for, in which case the second
thread may never wake up. See
https://mail.gnome.org/archives/gtk-list/2013-April/msg00040.html
This means that SyncEvolution can no longer rely on these functions
outside of the main thread. This affects Sleep() and the EDS backend.
As an interim solution, take over permanent ownership of the default
context in the main thread. This prevents fights over the ownership
when the main thread enters and leaves the main loop
repeatedly. Utility code using the main context must check for
ownership first and fall back to some other means when not the owner.
The assumption for the fallback is that the main thread will drive the
event loop, so polling with small delays for the expected status
change (like "view complete" in the EDS backend) is going to succeed
eventually.
A better solution would be to have one thread running the event loop
permanently and push all event handling into that thread. There is C++
utility code for such things in:
http://cxx-gtk-utils.sourceforge.net/2.0/index.html
See in particular the TaskManager class and its
make_task_when()/make_task_compose()/make_task_when_full() functions
for executing asynchronous results via a glib main loop, also the
Future::when() method and a number of other similar things in the
library.
Passing an explicit Logger instance to the SE_LOG* macros was hardly
ever used and only made the macros more complex.
The only usage of it was in some backends, which then added a prefix
string automatically. The same effect can be achieved by passing
getDisplayName(). Exception handling no longer needs a Logger instance,
the prefix alone is enough.
The other intended usage, avoiding global variables for logging by
passing a logger known to the caller, was not possible at all.
To make prefix handling more flexible, it is now passed as a "const
std::string *" instead of a "const char *".
The call cannot succeed without an empty UID, so there's no point
even trying it. Worse, EDS 3.8 aborts the process when given
an empty UID:
[ERROR 00:00:00] GLib: g_variant_builder_end: assertion
`!GVSB(builder)->uniform_item_types || GVSB(builder)->prev_item_type != NULL
|| g_variant_type_is_definite (GVSB(builder)->type)' failed
[ERROR 00:00:00] GLib: g_variant_get_type: assertion `value != NULL' failed
[ERROR 00:00:00] GLib: g_variant_type_is_subtype_of: assertion `g_variant_type_check (type)' failed
[ERROR 00:00:00] GLib: g_variant_get_type_string: assertion `value != NULL' failed
[ERROR 00:00:00] GLib: g_variant_new: expected GVariant of type `a(ss)' but received value has type `(null)'
See https://bugzilla.gnome.org/show_bug.cgi?id=697705
In EDS 3.6.4, opening fails a lot more often with E_CLIENT_ERROR_BUSY.
We must retry until we succeed.
Seen in particular with testpim.py testFilterStartup. Clients are expected to
try the open call again. EDS >= 3.8 does that automatically.
All of the jobs were allocated dynamically without freeing them
explicitly, instead relying on auto-deletion (enabled by default).
This behavior is surprising for SyncEvolution developers and Klocwork,
which expects explicit delete calls for each new. Better use
traditional RAII.
The advantage also is that each job gets deleted right away, instead
of waiting for another round of event processing, which may or may not
happen soon.
Qt has never been happy with the old code, reporting:
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
This error message became visible again after some recent logging
changes and broke tests, so better avoid it by testing for D-Bus
with a private connection.
Previously, the database field was interpreted as a Collection ID. This adds
logic to allow the database to be interpreted as a folder path. The logic is:
1) If the database is an empty string, pass it through (this is the most
common case as it is interpreted as "use the default folder for the
source type").
2) If the database matches a Collection ID, use the ID (this is the same as
the previous behaviour).
3) If the database matches a folder path name, with an optional leading "/",
use the Collection ID for the matching folder.
4) Otherwise, force a FolderSync to get the latest folder changes from the
server and repeat steps 2 and 3
5) If still no match, throw an error.
Steps 2 and 3 are in the new function lookupFolder. The remaining logic has
been added to the open function. Note that the result is that m_folder (and
hence getFolder()) are always either empty or a Collection ID -- that is as
before so the sync logic itself is unchanged.
A new method, findCollections, fetches the folder list from the server and
creates two maps:
m_collections - store all the information about each collection (folder),
indexed by server collection ID
m_folderPaths - map full folder paths to collection IDs
getDatabases uses this data to returns the folder path, collection ID and a
flag indicating if the folder is the default for that type.
Note 1: getDatabases always asks activesyncd to update the folder list from the
server in order to return up to date information.
Note 2: this depends on a new libeasclient routine:
eas_sync_handler_get_folder_list
A quick-and-dirty solution for enabling phone number summaries when
creating contact databases in the PIM Manager: let the EDS backend
recognize the special UIDs used by the PIM Manager and then hard-code
the minimal set of summary fields and indexed fields which allow
executing the E_CONTACT_TEL, E_BOOK_QUERY_EQUALS_NATIONAL_PHONE_NUMBER
query quickly.
A proper solution would use a new EDS function for parsing ESource
defaults from a string and then providing these defaults to the
backend from the PIM Manager.
Also note that configuring the EDS system address book must be covered
elsewhere, because it wouldn't be correct for SyncEvolution as only
one of many clients to change the configuration of that.
To enable the special support, add the following section to
share/evolution-data-server-3.6/rw-sources/system-address-book.source:
[Backend Summary Setup]
SummaryFields=phone
IndexedFields=phone,phone
This patch adds new function calls to code shared by syncecal and syncebook,
so we have to add libebook-contacts to both to avoid link errors.
This reverts commit 2735273ec60b289c5ec2c49f3eacb9d7d04d5ea1.
With this patch, setting up ActiveSync fails with Google as server.
Needs further investigation.
Also note the explicit g_object_unref() and EASFolderUnref - these
should get replace with SE_GOBJECT_TYPE smart pointers.
This reverts commit 7327b23a4dd31abdc9596916743892402bcffe0c.
Depends on 273527 "ActiveSync: added getDatabases support for fetching
folder list" which has to be reverted.
Previously, the database field was interpreted as a Collection ID. This adds
logic to allow the database to be interpreted as a folder path. The logic is:
1) If the database is an empty string, pass it through (this is the most
common case as it is interpreted as "use the default folder for the
source type").
2) If the database matches a Collection ID, use the ID (this is the same as
the previous behaviour).
3) If the database matches a folder path name, with an optional leading "/",
use the Collection ID for the matching folder.
4) Otherwise, force a FolderSync to get the latest folder changes from the
server and repeat steps 2 and 3
5) If still no match, throw an error.
Steps 2 and 3 are in the new function lookupFolder. The remaining logic has
been added to the open function. Note that the result is that m_folder (and
hence getFolder()) are always either empty or a Collection ID -- that is as
before so the sync logic itself is unchanged.
A new method, findCollections, fetches the folder list from the server and
creates two maps:
m_collections - store all the information about each collection (folder),
indexed by server collection ID
m_folderPaths - map full folder paths to collection IDs
getDatabases uses this data to returns the folder path, collection ID and a
flag indicating if the folder is the default for that type.
Note 1: getDatabases always asks activesyncd to update the folder list from the
server in order to return up to date information.
Note 2: this depends on a new libeasclient routine:
eas_sync_handler_get_folder_list
The backends had SYNCEVOLUTION_LIBS in their _DEPENDENCIES entries,
which is wrong because SYNCEVOLUTION_LIBS must include -lrt (which
can't be a dependency).
Fixed by depending on libsyncevolution.la directly.
Use e_book_client_connect_direct_sync(), the official API, when
available. Support for e_book_client_new_direct() is still in the
code; it can be removed onces the 3.6 openismus-work branch adapts the
official API.
The new Bluez 5 API is the third supported API for doing PBAP
transfers. It gets checked first, then the PBAB backend falls back to
new-style obexd (file based, similar to Bluez 5, but not quite the
same) and finally old-style obexd (data transfer via D-Bus).
In contrast to previous APIs, Bluez 5 does not report the reason for a
failed PBAP transfer. SyncEvolution then throws a generic "transfer
failed" error with "reason unknown" as message.
Google CalDAV currently sends invalid XML back when asked to include
CardDAV properties in a PROPFIND. This gets rejected in the XML
parser, which prevents syncing calendar data:
Neon error code 1: XML parse error at line 55: undeclared namespace prefix
The incorrect XML is this:
<D:propstat>
<D:status>HTTP/1.1 404 Not Found</D:status>
<D:prop>
...
<caldav:max-attendees-per-instance/>
<ns1:addressbook-home-set xmlns:ns1="urn:ietf:params:xml:ns:carddav"/>
==> <ns1:principal-address/>
<ns1:addressbook-description/>
<ns1:supported-address-data/>
<ns1:max-resource-size/>
</D:prop>
</D:propstat>
This was introduced on the server side sometime after December 12nd
2012 (tests run at that time showed a different response) and does not
affect SyncEvolution 1.2 because it did not yet ask for CardDAV
properties.
The workaround on the client side is to ask for only the properties
which are really needed.
Check for e_book_client_new_direct() in configure and use it if
found. This enables direct reading from the EDS file backend in the
direct searching of EDS address books.
The PIM code and the EDS backend both need access to common classes
(like the smart pointers) and eventually will share one instance of
the ESourceRegistry.
--create-database was broken in combination with the final code in EDS
3.6 because it passed NULL for the UID to e_source_new_with_uid(),
which is considered an error by the implementation of that
method. Must use e_source_new() if we don't have a UID.
"--status target-config@..." cannot really report on the status of the target
side of the sync, because nothing in the command line identifies *which* sync
is meant to be reported on: the same target config can be used in multiple
sync configs, therefore the meta data which tracks changes is attached to the
sync config, not the target config.
It should not abort, though, which happened because the backend had not
been properly initialized in this particular code path. Now it reports
the total number of items and zero changes.
Long term a dedicated error message would be better (see FDO #57145).
Fetched the code and its history from the 1.3.1 archives at:
http://people.debian.org/~ovek/maemo/http://people.debian.org/~ovek/harmattan/
Merged almost everything, except for Maemo/Harmattan specific build files:
autogen-maemo.sh builddeb buildsrc debian
The following changes were also removed, because they are either local
workarounds or merge artifacts which probably also don't belong into
the Maemo/Harmattan branch:
diff --git a/configure.ac b/configure.ac
index cb66617..2c4403c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -44,7 +44,7 @@ if test "$enable_release_mode" = "yes"; then
AC_DEFINE(SYNCEVOLUTION_STABLE_RELEASE, 1, [binary is meant for end-users])
fi
-AM_INIT_AUTOMAKE([1.11.1 tar-ustar silent-rules subdir-objects -Wno-portability])
+AM_INIT_AUTOMAKE([subdir-objects -Wno-portability])
AM_PROG_CC_C_O
diff --git a/src/backends/webdav/CalDAVSource.cpp b/src/backends/webdav/CalDAVSource.cpp
index decd170..7d338ac 100644
--- a/src/backends/webdav/CalDAVSource.cpp
+++ b/src/backends/webdav/CalDAVSource.cpp
@@ -1282,6 +1282,7 @@ void CalDAVSource::Event::fixIncomingCalendar(icalcomponent *calendar)
// time.
bool ridInUTC = false;
const icaltimezone *zone = NULL;
+ icalcomponent *parent = NULL;
for (icalcomponent *comp = icalcomponent_get_first_component(calendar, ICAL_VEVENT_COMPONENT);
comp;
@@ -1295,6 +1296,7 @@ void CalDAVSource::Event::fixIncomingCalendar(icalcomponent *calendar)
// is parent event? -> remember time zone unless it is UTC
static const struct icaltimetype null = { 0 };
if (!memcmp(&rid, &null, sizeof(null))) {
+ parent = comp;
struct icaltimetype dtstart = icalcomponent_get_dtstart(comp);
if (!icaltime_is_utc(dtstart)) {
zone = icaltime_get_timezone(dtstart);
diff --git a/src/backends/webdav/CalDAVSource.h b/src/backends/webdav/CalDAVSource.h
index 517ac2f..fa7c2ca 100644
--- a/src/backends/webdav/CalDAVSource.h
+++ b/src/backends/webdav/CalDAVSource.h
@@ -45,6 +45,10 @@ class CalDAVSource : public WebDAVSource,
virtual void removeMergedItem(const std::string &luid);
virtual void flushItem(const string &uid);
virtual std::string getSubDescription(const string &uid, const string &subid);
+ virtual void updateSynthesisInfo(SynthesisInfo &info,
+ XMLConfigFragments &fragments) {
+ info.m_backendRule = "HAVE-SYNCEVOLUTION-EXDATE-DETACHED";
+ }
// implementation of SyncSourceLogging callback
virtual std::string getDescription(const string &luid);
Making SySync_ConsolePrintf a real instance inside SyncEvolution leads
to link errors in other configurations. It really has to be extern. Added
a comment to the master branch to make that more obvious:
-extern "C" { // without curly braces, g++ 4.2 thinks the variable is extern
- int (*SySync_ConsolePrintf)(FILE *stream, const char *format, ...);
-}
+// This is just the declaration. The actual function pointer instance
+// is inside libsynthesis, which, for historic purposes, doesn't define
+// it in its header files (yet).
+extern "C" int (*SySync_ConsolePrintf)(FILE *stream, const char *format, ...);
Depends on the EDS 3.6 ESourceRegistry API. Needs the very latest EDS
with the following commit. There's no configure check yet because EDS
3.6 is not released yet.
commit 6df76009318eac9dbe3dd49165394d389102764e
Author: Matthew Barnes <mbarnes@redhat.com>
Date: Tue Sep 11 22:56:08 2012 -0400
Bug 683785 - Add e_source_new_with_uid()
Variation of e_source_new() which allows a predetermined UID to be
specified for a scratch source. This changes the "uid" property from
read-only to read/write + construct-only, and eliminates the need for
EServerSideSource to override the property.
The synchronous version used to deadlock frequently, the asynchronous
one doesn't, so use that instead. The problem was also worked around
later in EDS (commit below), but let's stick to the asynchronous
version anyway to be closer to Evolution.
commit 6c4c1c6a1f40a6873be954a0b9d770e31726bd50
Author: Matthew Barnes <mbarnes@redhat.com>
Date: Fri Sep 7 07:40:09 2012 -0400
ESourceRegistry: Work around GType deadlock.
Work around http://bugzilla.gnome.org/show_bug.cgi?id=683519
until GObject's type initialization deadlock issue is fixed.
Apparently only the synchronous instantiation is affected.
Instead of manually written callback functions, use the C++ helper
code to get C++ code invoked from inside glib. Reduces the code
size and ensures that no exceptions escape into glib C code.
With Google Contact + CardDAV the auto-discovery failed after
finding the default address, without reporting that result.
The reason was that the search continued at the root of the server
where PROPFIND triggers an error when using the Google server. Because
of a missing check for "have result", that error was treated as
something which needs to be reported to the user.
Fixed by unifying the various checks in a singe class.
Fallback to old obexd API was broken because creating the
DBusRemoteObject does not verify whether the service really exists and
thus always succeeds. Fixed by checking for existence as part of the
actual CreateSession method call.
The new code needs glib. Include header file and declare dependency in
configure.
The backend must throw errors when something fatal happens. Logging
the error is not enough, because that can't be checked by the
caller. Throwing errors is best done via the utility methods in
SyncSource, because those include the source name in the exception.
Memory handling was broken: nothing owned the memory in the
StringPiece instances, munmap() was missing. Fixed by making
PbabSyncSource the owner of both.
Unified the parsing of the result. The new code based on pcrecpp is
used for both old and new obexd API.
File name and GError allocated by g_file_open_tmp() were leaked. The
file descriptor and the file were leaked in case of aborting via an
exception. Now these resources are owned by a class which will always
clean up when getting destructed.
A failed transfer was not checked for when using the new API. Probably
failed when trying to use the file (because obexd deletes it), but
better show the error message that we got for the failed transfer.
Remove the obsolete vcardParse().
The backend is not useful for most users, therefore it has to be enable
during compilation with --enable-pbap. The code for "PBAP disabled"
had to be adapted to a backend API change.
and allow this UID to be used for selecting a particular calendar.
Since all listed calendars are in the default storage anyway,
the UID is far more useful to have. (On the N9, knowing the
physical storage does not help the user at all anyway, as access
to it is restricted and needs to go through the API anyway.)
(cherry picked from commit a5c2939c1d)
The PBAP backend must link against pcrecpp explicitly. The header
files are part of the global search path, but the library is not.
Having to specify the library explicitly is right (avoids
adding it to modules which don't need it); perhaps the header
file flags also should have to be added explicitly.
It worked when linked into an executable which also uses pcrecpp, but
can fail to build depending on how strict the toolchain is. Found
by dist checks in the nightly testing.
Change tracking in the file backend used to be based on the
modification time in seconds. When running many syncs quickly (as in
testing), that can lead to changes not being detected when they happen
within a second.
Now the file backend also includes the sub-second part of the
modification time stamp, if available. The revision string
intentionally uses no leading zeros, because that would make it
unnecessarily larger.
This change is relevant when upgrading SyncEvolution: most of the
items will be considered "updated" once during the first sync after
the upgrade (or a downgrade) because the revision strings get
calculated differently.
With this patch, the databaseFormat can be used as follows:
[(2.1|3.0)][:][^]propname,propname,...
3.0 = download in vCard 3.0 instead of the default 2.1
3.0:^PHOTO = download in vCard 3.0 format, excluding PHOTO
PHOTO = download in vCard 2.1 format, only the PHOTO
Valid property names are pulled from obexd with ListFilterFields().
Binding a temporary std::string (the result of getDatabase() in this
case) to a const reference is broken, because the temporary instance
will get deleted before the reference.
Local IDs across sessions are only useful when we also have useful
revision strings. For debugging it is easier to just enumerate the
contacts. Would be nice to use the same number as in the PBAP session,
but that information is not currently available via obexd (see "PBAP +
two-step download" on Bluez mailing list).
As it stands now, the PBAP backend can only be used in slow syncs
where the engine does the matching. Perhaps that's the right way to do
it, instead of adding redundant code to the backends.
Some unnamed version of KDE crashes in KApplication when invoked
without a D-Bus session. The reporter ran into this when compiling
from source, because the SyncEvolution binary is invoked as part of
the build process, which ran outside of a D-Bus session.
Avoid the crash by checking for a D-Bus session bus with
QDBusConnection::sessionBus().isConnected() before instantiating
KApplication. The QDBusConnection API does not say explicitly when it
connects to the daemon, but testing shows that in practice this
detects missing env variables and an unreachable daemon right away as
expected, while passing when the daemon can be contacted.
Instantiating KApplication was added for KWallet support. Without
D-Bus, KWallet does not work either, therefore throw an explicit error
when the lack of D-Bus was detected.
and allow this UID to be used for selecting a particular calendar.
Since all listed calendars are in the default storage anyway,
the UID is far more useful to have. (On the N9, knowing the
physical storage does not help the user at all anyway, as access
to it is restricted and needs to go through the API anyway.)
When compiled against EDS 3.5.x or later, SyncEvolution now uses
the backend code originally written for the EClient API introduced
in EDS 3.2. That code was changed so that it works with the new
include file rules and ESourceRegistry in EDS 3.5.x. Support
for using the EClient API with EDS 3.4 was removed because maintaining
three different flavors of the EDS backend code would be too much
work and not gain much (just the possibility to test the EDSClient
code with 3.4).
At the moment, this is a compile time choice made automatically
by configure. syncevolution.org binaries are compiled against
an older EDS and thus do not work with EDS 3.5.x or later.
EDS 3.5.x handles authentication itself, using a standard system
prompt if necessary. SyncEvolution can no longer provide the password,
and thus the "databaseUser/Password" options have no effect when using
EDS 3.5.x.
The patch leaves code for older EDS almost completely unchanged and
therefore is considered safe for the stable release series leading to
1.3. Using EClient is an all-or-nothing choice now, because the common
EvolutionSyncSource needs to be compiled differently. Thanks to the
reorganized API, a lot more common code for ECal and EBook sources
could be moved into EvolutionSyncSource.
Instantiating an ActiveSyncSource for testing must work with
empty database names now. Testing no longer forces the
database to be set.
While at it rewrote the code to avoid the explicit pointer.
A backend cannot know whether it'll be used together
with a client-test which supports integration testing.
Therefore the backend should always register its tests.
Updating failed when using Google because the Synthesis engine
tried to read the existing item in order to merge it with
the update. This failed because Google does not implement the
Fetch command.
Pretending to update the item intelligently avoids that. It
also helps to improve performance of updates with Exchange.
The downside is that syncing with local storages which do
not support all ActiveSync fields will cause data loss.
Need to check whether Exchange-only attributes get lost
also when the local storage supports everything, for
example because activesynd unintentionally removes data.
The error message for "Invalid synchronization key" changed in
activesyncd, now it contains the D-Bus error type as prefix. Fixed by
doing a substring search.
Also, not freeing the GError before trying again is a bug. Apparently
that was ignored earlier, now it triggers an assert.
As mentioned by Tino Keitel on the mailing list, some libs and
executables were only implicitly linked against libraries that they
called directly. This happened to work by chance because these libraries
ended up in the running executable anyway, due to indirect loading.
To catch such problems, the "make installcheck" was extended:
dpkg-shlibdeps is run, if available, and the error output is scanned
for the messages which indicate that a symbol is used without linking
to the right library (example output below).
Had to fix quite a few _LIBADD lines to pass the test.
Some exceptions are allowed:
- libsmltk depends on the caller providing SySync logging support.
- libneon is intentionally not linked explicitly for syncevolution.org
binaries, to make resulting binaries work with GNUTLS and OpenSSL.
dpkg-shlibdeps: warning: debian/syncevolution-libs/usr/lib/syncevolution/backends/syncdav.so contains an unresolvable reference to symbol icalparameter_new_from_value_string: it's probably a plugin.
dpkg-shlibdeps: warning: 51 other similar warnings have been skipped (use -v to see them all).
...
dpkg-shlibdeps: warning: symbol dlsym used by debian/libsyncevolution0/usr/lib/libsyncevolution.so.0.0.0 found in none of the libraries.
dpkg-shlibdeps: warning: symbol dlerror used by debian/libsyncevolution0/usr/lib/libsyncevolution.so.0.0.0 found in none of the libraries.
dpkg-shlibdeps: warning: symbol dlopen used by debian/libsyncevolution0/usr/lib/libsyncevolution.so.0.0.0 found in none of the libraries.
When libical and libecal were not installed, trying to use the CalDAV
backend for VEVENTs segfaulted because it depends on libical and did
not check properly for it. Only affected syncevolution.org binaries.
The root cause is that libical functions were only looked up in
combination with libecal, when compiled with
--enable-evolution-compatibility. Now they are first checked via
libecal (for old libecal which embedded libical) and separately in
libical itself as fallback.
Google Calendar sometimes returns redirect requests to human-readable
web sites (an "unavailable" page, a login form). This is of course
bogus when the client is an automated CalDAV client.
The "unavailable.html" case was already handled. Made it a bit more
flexible to also catch possible variations of it (additional
parameters, https instead of http).
Added the https://accounts.google.com/ServiceLogin case. Not sure
whether retrying will help in that case, but there's not much else
that SyncEvolution can do.
When asked to insert a VJOURNAL which already existed (= same UID),
CalDAV servers respond with a 412 "Precondition failed" error. This
needs to be detected and translated into an "item needs to be merged"
result so that the engine can load the existing item, merge the data,
and then write back.
A test for this, testInsertTwice, will be committed separately. The
code was written so that it handles the same error when using CardDAV.
However, this was not tested because CardDAV test data does not have a
UID (wouldn't trigger the problem) and Radicale did not report 412 when
adding the UID.
Instead of relying on catching an exception, better use the new
"expected status codes" feature and check for 412 as part of the
normal switch statement.
Some WebDAV requests might fail with a non-okay status code that the
caller expects. This was reported via an exception. But
SyncEvolution's design uses exceptions only for non-normal
incidencences. Therefore better allow the caller to indicate which
status codes are expected and return normally from run() and
checkError() when those are encountered, without retrying and without
throwing an exception.
In some cases data with a very long UID wasn't handled correctly,
causing the out-going data to be malformed and probably causing a
rejection by the server.
The root-cause was two-fold:
- extractUID() didn't expect folding. Normally it deals only with
data encoded by libsynthesis, which does not use folding; the unexpected
exception was data which gets imported directly (--import).
- setResourceName() used std::string::replace() incorrectly: second
parameter is length, not end offset, of the data to be replaced.
The command line --status operation did not complete when applied to a
CalDAV/CardDAV source. Instead it aborted because the operation took a
code path where the backend was not fully initialized.
Sometimes the backend has to resend requests because of temporary
issues. If the problem turned out to be permanent, there was a long
period of time, retryDuration=5 minutes to be precice, in which no
visible progress happened.
Now SyncEvolution's WebDAV backend will print a message like this
before going to sleep until it is time to retry:
[INFO @client-test-googlecalendar] operation temporarily (?) failed, going to retry in 5.0s before giving up in 18.4s: PROPFIND: Neon error code 1: 401 Unauthorized
The uncertainty comes from several factors. In this example, the 401
might indicate a permanent problem (wrong credentials), or it could be
Google reporting a temporary authorization problem which is (probably)
meant to slow down the client while it asks the user to re-enter the
password. SyncEvolution only asks for passwords once, so it tries
again with the same password if it was successful with it in the
past. Otherwise it gives up immediately.
Another dubious example are name server lookup errors. They can be
ermanent (wrong host name) or temporary (name server
down). SyncEvolution errs on the side of retrying, to avoid
interrupting an operation which still has a chance to continue.
Currently this output is only visible in combination with --daemon=no,
because when using syncevo-dbus-server the output is not passed on
properly.
The message resend loop in the WebDAV backend did not check abort
requests. Now the error which caused the resend is reported
immediately in case of an abort. The actual send/receive still cannot
be interrupted, as it happens inside libneon with little chances to
influence the execution (?).
When waiting for resending a failed message, the sleeping couldn't be
interrupted when using the D-Bus server. It now uses the SuspendFlags
infrastructure and glib, which detects abort requests sent via D-Bus
in addition to those sent via signals (which already worked earlier).
CalDAV VJOURNAL and VTODO need similar UID handling as VCARD: when
creating or updating an item, ensure that it has a UID in the
payload.
It may also help to match that with the resource name and use
.ics/.vcf as suffix of the resource name. It was necessary for Yahoo's
CardDAV support and doesn't hurt elsewhere.
When updating, the merging with local data now ensures that any UID
different from the resource is properly preserved when the incoming
data has no UID.
All backends except for EDS replaced local data wholesale with an
incoming update, even if that update came from a peer which did not
store all properties. The EDS backend had already been configured
earlier to always merge remote and local data before writing it back.
Instead of updating each backend individually it makes more sense to
make the more intelligent (and more expensive) merging the default in
backends derived from SyncSourceSerialize/TrackingSyncSource. Backends
which do not want it can remove
"<updateallfields>true</updateallfields>" from
SynthesisInfo::m_datastoreOptions.
The new backend property values "CalDAVTodo" and "CalDAVJournal"
select tasks resp. memos stored in a CalDAV collection. "CalDAV"
continues to select events.
Events, tasks and journals can be mixed in the same resource (=
URL). However, this is less efficient than storing them separately.
A good CalDAV server allows filtering items by type, and SyncEvolution
uses that. However, it was found that Radicale 0.7 ignores this
filtering, which could have led to data loss (SyncEvolution asks for
all VTODOs in preparation for a "delete all items" operation in a
"CalDAVTodo" source, gets also VJOURNALs, then deletes them).
Therefore SyncEvolution plays it safe and downloads the VTODO and
VJOURNAL data to double-check that it is working on the right items.
This causes additional traffic for well-behaving servers; currently
it cannot be turned off.
What is missing for VJOURNAL is the conversion to plain text (see BMC
not possible yet.
Radicale reports 200 'Okay' instead of 204 when asked to delete an
existing item. Was triggering a status error. Found when running
Client::Source test against Radicale 0.7-7-g186c59c; 0.7 still
returned 204, see http://redmine.kozea.fr/issues/802.
Radicale reports 412 'Preconditiona Failed' instead of 404 when asked
to delete a non-existent item. Translate that into the 404 expected by
the Synthesis engine. Found when running Client::Source test against
Radicale 0.7-7-g186c59c.
Radicale reports 410 'Gone' instead of 404 when asked to read a
non-existent item. Translate that into the 404 expected by the
Synthesis engine. Found when running Client::Source test against
Radicale 0.7-7-g186c59c.
Due to the way how properties were handled internally, it wasn't
possible to explicitly set a property to its default value. Instead
the property was unset. For example, explicitly setting database= was
not possible.
This is necessary for client-test and ActiveSync, because client-test
needs to know that the testing is expected to run with the default
databases (something which normally is avoided by overwriting empty
database properties).
Now the "is set" state is tracked explicitly in the config storage and
command line property APIs. Unsetting a property via the command line
could be implemented with an explicit command line option, but is not
supported at the moment.
Tests were extended to cover the new functionality and adapted to the
change behavior for "type" migration: syncFormat was empty
already (because the empty string matched the default), but
forceSyncFormat was unnecessarily set explicitly. Now it is not.
Google doesn't seem to support the Fetch operation, which is used
during testing to retrieve unchanged items. During syncing it will
only be needed when merging incoming data with an existing item, which
should not be necessary... except that testing shows that it is
necessary. The case where it is needed is:
- incremental sync (cache empty)
- calendar event series needs to be modified
- items from that series need to be fetched in preparation
for updating it
To allow testing, several workarounds are necessary:
- request *all* data before doing a data dump in testImport (and friends),
to ensure that the cache in the backend is fully populated
- use the cache in ActiveSyncCalendarSource instead of accessing
the base class directly, because that would trigger a Fetch
Accessing the base class is still useful for Exchange+calendar,
because testing then avoids the cache (and thus can expose bugs
in it).
While at it, adapted the README. "database" needs to be set
explicitly, "client-test" only does it when creating configs.
Fixed ESourceList leak. Was never freed?! Now assigned to a new GObject
C++ wrapper as owner.
client API: Use EvolutionSyncSource::throwError() consistently, because it
relates the error to the source (nicer).
client API: fixed "only-if-exists" issue for contacts.
client API: fixed leak of uid after creating calendar object.
The Qt headers define "signal", which breaks GNOME header
files if those get included later. Rearranged header
files to avoid ugly "undefine" statements.
Found when using EDS client API header files, because those
include different GNOME headers.
e_cal_client_tzlookup() doesn't detect that lookup
fails because of non-existant time zone, because it gets
a non-descriptive error code from EDS ("Invalid object").
Work around that by ignoring all errors in our own lookup
function.
See "[Evolution-hackers] e_cal_client_check_timezones() + e_cal_client_tzlookup() + Could not retrieve calendar time zone: Invalid object"
KWallet support was broken: syncevo-dbus-server checked
KDE_FULL_SESSION to determine whether it should use KWallet instead of
GNOME Keyring. That did not work, because the env variable was not set
for D-Bus daemons.
Automatically detecting KDE users is not possible at the
moment. Instead KDE users have to manually set the new "keyring"
global config property to "KDE" (case insensitive) if the
SyncEvolution installation supports both, because GNOME Keyring is the
default to avoid surprises for traditional users. If only KWallet
support is enabled, then this is not necessary.
"GNOME" and "true/false/1/0/yes/no" can also be set. This has the
advantage that keyring usage can be enabled permanently for the
command line in --daemon=no mode; normally keyrings are not used in
that mode because accessing them can bring up UI dialogs.
It also becomes possible to disable keyring usage in syncevo-dbus-server,
something which couldn't be done before.
The --keyring command line option is still supported, as an alias for
"[--sync-property] keyring=<value>". The default value for --keyring
is true, to match the traditional behavior. In contrast to other sync
properties, setting "keyring" does not require an explicit --run
parameter. Again this is done to mirror traditional usage.
Reading a password also (unintentionally) checked all supported
storages while searching for the password. Now it uses exactly
one storage and falls back to asking for the password directly.
The commit itself also cleans up the code a bit (reformatted, fixed
comments). Choosing the right slot in the password signals is done via
a new InitStateTri parameter which contains the "keyring" setting.
Error checking (unsupported keyring string, --keyring=yes and no
keyring enabled) is done in additional slots which run after all the
regular ones.
Parameter parsing for --sync and --keyring were unified. However,
there is the difference that --keyring has an implicit default value
("yes") and never has an additional parameter, in contrast to --sync,
which always is followed by one.
The new CmdlineTest::testKeyring covers different ways of using
--keyring. It relies on actually invoking keyring backends, something
not done by the default SyncContext UI. Therefore
CmdlineSyncClient+KeyringSyncCmdline were moved into libsyncevolution,
to be used by CmdlineTest.
A normal KDE user should already have the akonadi server running.
Others probably don't want it to be started, for example as part
of --print-databases.
Therefore auto-starting Akonadi is now disabled. Attempts to use
the Akonadi sources results in an error message with a hint
about akonadictl when Akonadi is not already running.
The same problem is less relevant for EDS (but still exists to
some extend): here listing databases "merely" starts gconfd.
Merged branch 'eds-api'. Several changes that were made in the
meantime on the master branch caused conflicts and/or required
updating the new code to avoid regressions. The code was also
updated to address some issues:
- g_propagate_error() steals ownership from GErrorCXX,
thus causing a double free. Replaced with consistent use
of GErrorCXX.
- Putting opening brackets into the middle of ifdefs
breaks automatic code formatting. Now the code has "if("
and ") {" outside of the ifdef/else/endif block.
The old code was updated to also use GErrorCXX. This revealed a memory
leak in the error path for opening databases, where the GError was not
cleared before returning.
SyncEvolution EDS backend currently uses the ECal API which is
deprecated as of EDS v3.2 and replaced by the new ECalClient API.
This patch brings support for this new ECalClient API (when EDS
3.2 is detected at compile time).
The new code based on the ECalClient API will enable a few
optimizations, especially the partial fetching of components to
avoid useless DBus trafic. This results in better component change
tracking by fetching only the UID/RID/REV attributes for each
component.
Note: The new code uses the new GLibSupport header instead of the
deprecated SmartPointer (eptr).
SyncEvolution EDS backend currently uses the EBook API which is
deprecated as of EDS v3.2 and replaced by the new EBookClient API.
This patch brings support for this new EBookClient API (when EDS
3.2 is detected at compile time).
The new code based on the EBookClient API enables a few
optimizations, especially the partial fetching of contacts to
avoid useless DBus trafic. This results in better contact change
tracking by fetching only the UID/REV attributes for each contact.
Note: The new code uses the new GLibSupport header instead of the
deprecated SmartPointer (eptr).
Changed the default sync format of the Evolution Data Server contact
source from vCard 2.1 to vCard 3.0. This is better for SyncEvolution<->SyncEvolution
sync, because it avoids some vCard 2.1 encoding issues in the Synthesis engine (found
during "testConversion").
Also more properties are officially part of the standard.
This uses the new libsynthesis support for adding and checking entries
in the SyncCap to detect per datastore whether UID/RECURRENCE-ID are
truly globally unique and thus can be used to finding pairs. The
presence of the property alone is no guarantee for that.
Previously this kind of pairing was enabled only for local sync, which
was a hack which didn't work for local backends which didn't support
UID (for example, Maemo 5 calendar). It also didn't work for mixtures
of datastores with and without that kind of support.
"1122583000" was randomly chosen as pseudo sync mode. It is a number
because strings confuse Funambol. Note that SYNCMODESUPPORTED() only
works inside the compare script.
Both begin() methods grab the item node and use it for tracking.
That must only be done once, otherwise the second cycle uses an empty
Boost pointer (=> exception).
First, FN is not mandatory in PBAP for vCard 2.1. Better use N as key,
which is mandatory. Second, finding the property name must stop at
colon or semicolon, whatever comes first. The previous code included
the colon and the first name component because it stopped at the first
semicolon.
The main goal is to test CalDAV/CardDAV sources as part
of a SyncML client and/or server. A test involving syncevo-http-server
is now named "<client storage><server storage>":
- edsfile = EDS in client, file in server (used to be syncevohttp)
- davfile = CalDAV/CardDAV in client, file in server (new)
- edsdav = EDS in client, CalDAV/CardDAV in server (new)
For this, WebDAVSourceRegister.cpp must be able to create test sources
which match the client 1/2 sync configs. The client "1" or "2" strings
are passed through the abstract ClientTest into the source A/B create
callbacks. WebDAVSourceRegister.cpp cannot get this directly from
ClientTest because it lives in a plugin which is not necessarily
linked against ClientTest.
A conceptual change is that CLIENT_TEST_EVOLUTION_PREFIX/USER/PASSWORD
no longer override existing config properties. That is necessary
because the shared prefix is too simplistic for WebDAV (needs full URL
in "database"); also helps KDE (needs resource URI). The env variables
and the default "SyncEvolution_Test_" value for the database prefix are
still used if the config does not exist. That is useful to prevent
accidentally running client-test against the default databases.
The nightly setup script might (should!?) be made public to simplify
configuring a server.
Another change is the user-configurable part of client-test now lives
entirely in the _1/_2 client sync configs and contexts. From there the
source properties are copied into the Client::Source context each time
client-test runs.
A CalDAV/CardDAV source could not be used as data storage in a SyncML
server or for direct syncing with a SyncML peer (phone, server)
because the source dependent on additional information that was taken
from the "target-config" (syncURL, credentials, log level).
Now the source settings are checked first, with the "target-config" as
fallback:
- "database" - must be set to the full URL of the resource,
use --print-databases or the servers documentation to find it
- "databaseUser" - username on server
- "databasePassword" - corresponding password
The log level for WebDAV operations is taken from the global logging
instance as fallback and thus from the log settings of the running
sync.
The downside of this approach is further duplication of the
credentials. This will be fixed eventually by introducing the concept
of "credential references", but that is harder to implement and will
depend on the "databaseUser/Password" change anyway.
XMLParser::initReportParser() allows an empty callback,
but doResponseEnd() didn't check for that. Caused exceptions
and thus failures in the Google 404 workaround (now also fixed
differently, by providing a callback).
The workaround for a 404 from Google Calendar for a GET (sending a
REPORT request matching the item's UID) was broken: first, processing
the result ended up calling the unset responseEnd boost function
pointer, which caused the request to fail. Second, getting multiple
items wasn't handled (data from all items concatenated together was
used).
That can happen in the somewhat unlike case that some items have a UID
which is a complete superset of the requested UID - not realistic in
real life, but happens during testing.
Fixed by using a responseEnd callback which only stores the right
item data and resets the buffer in all cases.
All user-facing functions (password handling, reading from
stdin) are now in a dedicated "UserInterface" class, instead of
spreading that between ConfigUserInterface and SyncContext.
SyncContext no longer has the misleading "is a" [Config]UserInterface]
relationship. Instead it "has a" UserInterface instance, which is set
at runtime. Long term the plan is remove the need to subclass
SyncContext. In the local sync that was already possible
(LocalTransportContext->LocalTransportUI).
The guarantee that there always is a usable instance that can be used
without checking for NULL is provided by the getUserInterfaceNonNull()
method, which falls back to a dummy instance if necessary.
Reading data and password from stdin is moved out of the core
libsyncevolution into the syncevolution binary. That way the D-Bus
server and client-test do not accidentally attempt to read from stdin
(has happened when setting up testing incorrectly).
The special semantic of the former RegisterSyncSource::InactiveSource
(invalid pointer of value 1) caused bugs, like using it in
--print-databases (=> segfault) or not being able to store the result
of a createSource() directly in a smart pointer (=> potential leak in
SyncSource::createSource()).
Obviously a bad idea to start with. Replaced with a
RegisterSyncSource::InactiveSource() method which creates a real,
inactive SyncSource instance which can and must be deleted by the
caller.
This is a SyncSource API change for backend developers.
Instead of RegisterSyncSource::InactiveSource, return
RegisterSyncSource::InactiveSource().
Comparisons against RegisterSyncSource::InactiveSource needs
to be replaced with a call to the new SyncSource::isInactive().
User visible fixes:
* --print-databases: no longer crashes when EDS or KDE backends
are not usable. Instead it prints "not enabled during compilation or
not usable in the current environment".
* --print-databases: continues with other backends even if
one backend throws an exception, as the KDE backend does
when it cannot find Akonadi. Error messages are printed.
The platform specific code which is of no value unless you run a
specific desktop now gets compiled as part of shared libraries, just
like the storage backends. The advantage is that the rest of
SyncEvolution keeps running even if one of these shared libraries
cannot be loaded due to missing depdendencies. syncevolution.org
packages will not declared these dependencies, to allow installing
each package without forcing the installation of unwanted libraries.
Distros can package the platform code separately.
Another advantage is reduced code duplication (password load/store
was duplicated in command line and D-Bus server).
Technically this uses almost the same mechnism as loadable sync
sources. The code resides in src/backends/[kde|gnome], where the
autotool magic finds the *Register.cpp files automatically and
includes them into executables. These files contain global singletons
which, when initialized, connect platform specific code to new signals
in the core (init, password load/save).
The actual code is in the backend libraries. Because
SE_ARG_ENABLE_BACKEND() is not used (in favor of the traditional
enable macros), linking against these libs must be set up by adding
them to the (now slightly misnamed) SYNCSOURCES variable in the
configure fragments.
EDS 3.3 has the "store PHOTO data in local files" patch, i.e. it
automatically copies the data from base64 encoded property value
into a file and then only stores an URI of that file in the vCard.
SyncEvolution already correctly handles such URIs when syncing.
In that case it is desirable to work with the URI as long as
possible and only inline once the data actually gets sent to a
peer (see d6d6e8ca, "vcard: inline local photo data (BMC #19661)").
But the backend should also inline the PHOTO file data when
exporting as vCard, because that vCard is meant to be a complete
representation of the entire contact, including the PHOTO (might
be send to other peer via email, used to restore later, etc.).
This patch adds that inlining for "raw reads" (aka "give me
your data in your native format") with the help of the EDS 3.3
e_contact_inline_local_photos() utility methods. Compile or
runtime checks are necessary to determine whether that methods
is available, depending on whether compatibility mode is active.
If EDS doesn't have the function, data will be exported with the URI,
as before. That should be fairly unlikely before EDS 3.3.
The camel backend failed to compile with latest EDS from master
(a 3.3.99 pre-release). Disabling compilation of camel
backend and eplugin (which might still compile, but wasn't
tested anyway in the nightly builds), to keep testing of activesyncd
for contacts and calendar working.
ClientSourceRevisions needs to be reset() before adding new items
while doing change detection. CalDAVSource also didn't properly
populate its m_cache in following cycles because the cache was assumed
to be set already.
When a server responds to a PROPFIND for a path with results for some
other path, then SyncEvolution crashed during the search for the
default calendar or address book because of a bug in the code which
was meant to handle that kind of response. Apparently Yahoo Calendar
did that. Now seen again in combination with Radicale 0.6.4.
The debug request failed with Radicale (details below). Now the result
of executing the request is ignored instead of failing the entire
operation.
PROPFIND /public_user/calendar/ HTTP/1.1
[<?xml version="1.0" encoding="utf-8"?>
<propfind xmlns="DAV:"><allprop/></propfind>
]
HTTP/1.0 500 Internal Server Error
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/python2.7/dist-packages/radicale/__init__.py", line 183, in __call__
status, headers, answer = function(environ, items, content, None)
File "/usr/lib/python2.7/dist-packages/radicale/__init__.py", line 365, in propfind
environ["PATH_INFO"], content, calendars, user)
File "/usr/lib/python2.7/dist-packages/radicale/xmlutils.py", line 184, in propfind
props = [prop.tag for prop in prop_element]
TypeError: 'NoneType' object is not iterable
Radicale 0.6.4 returns no results (neither error nor data) when asked
to deliver specific items with a multiget report. This broke certain
change tracking cases in SyncEvolution (one sync succeeded, items
added or updated later on).
Now SyncEvolution detects the missing responses and falls back to
individual GET requests (slower).
SyncEvolution strips the collection path from hrefs to produce
relative IDs. This assumed that the path ends in a slash, which was
not always the case when users manually set the URL (syncURL,
database). As a result, IDs had a leading slash, which later were
treated as absolute file paths on the server, leading either to
authentication or 404 errors.
Now the path is normalized as a collection when parsing the URL, which
adds the expected slash.
Radicale sends <href> values with more than one slash as separator:
<href>/public_user/calendar/calendar_1//20060406T211449Z-4562-727-1-63@gollum.ics</href>
SyncEvolution then only stripped the part up and including
".../calendar_1/" and used the rest as ID, which later failed because
the leading slash in "/20060406T211449Z-4562-727-1-63@gollum.ics" led
to requests for an item in the root of the server.
The previous patch was incomplete, or something changed again on the
Google side. We also need to deal with Google Calendar reporting items
in the etag query and then not being able to deliver their data in
a multiget (404 error). Affects change tracking.
A valid reason for the 404 would be a concurrent DELETE. So let's
do the right thing and remove all information that we still had about
the item, i.e., report it as deleted.
The mechanism for making D-Bus method calls changed on the master
branch. The normal call operator now does a blocking method call,
which breaks code depending on the former non-blocking semantic.
All D-Bus methods calls in the PBAB backend are now blocking. This
allows removing all callback methods and the start/stop main loop
tricks. A lot easier to read and problems while executing the method
calls are now properly reported back to the user of the backend via
exceptions.
Recently Google Calendar started to report empty VCALENDAR items
with no VEVENTs inside. Sending a DELETE for them didn't seem to
have an effect, or rather, might have created them in the first place.
This has all kinds of weird effects during syncing.
Now such empty items are silently ignored.
Always try to create address book or calendar database, because even
if there is a source there's no guarantee that the actual database was
created already; the original logic for only setting this when
explicitly requesting a new database therefore failed in some cases.
When using the "backend = addressbook/calendar/todo/memo" aliases, then
the EDS backend must be the only backend that accepts that, to avoid
ambiguities and regressions for traditional usage of SyncEvolution.
These aliases are used when following the current setup instructions
or the GTK sync UI.
Even if SyncEvolution somehow figured out that "addressbook" is meant to
be the KDE addressbook (for example by looking at KDE_FULL_SESSION once
more), fully automatic configuration for KDE still wouldn't work because
the "database" property has to be set explicitly to a URI of an Akonadi
resource. There is no code in the backend or Akonadi to identify the
default resource.
When building modules, the registation code must be part of
the module. Integration testing should always be offered by
the backend. The ifdef is only for the test driver.
readItem() might be asked to retrieve a non-existent item.
The fetch job succeeds in that case, but without returning
any item. Throw the 404 status error in this case.
Due to bitrot the Akonadi backend and KWallet support code no longer
worked. Moved the common code for KApplication initialization into
libsyncevolution's SyncContext::initMain() and fixed autotools rules.
The old code always tried to contact an X server (default constructor
of KApplication). That doesn't seem to be necessary and is avoided now.
Even better might be to skip KApplication entirely and instead use
QCoreApplication and KComponentData, as suggested by
http://api.kde.org/4.x-api/kdelibs-apidocs/kdeui/html/classKApplication.html
KAboutData was incorrectly passed the address of a string pointer, not
the pointer itself.
Testing the Akonadi backend in client-test failed because client-test
always overwrites the "backend" value with
"Test_kde_[contact/event/..]._[1/2]". Now this special case is
detected. The backend then uses the first resp. second resource that
it finds.
Populates the local cache with the addressbook given by the remote
device as result of PullAll().
The LUID is built from the full-name of the vcard entry, combined with a
suffix that avoid collisions in case of multiple entries with the same
full-name.
This instantiates the D-Bus wrapper and thus requests a obex-client
session.
A local cache of the addressbook has been added which will be loaded
when the database is opened.
Fixed the autotools rules and include statement so that the top-level
configure+Makefile can choose between the two different C++
bindings. The library must be linked explicitly to avoid problems when
linking statically.
When NEON_LIBS=-lneon-gnutls, the sed invocation didn't properly turn
that into -ldl. Instead it used -ldl-gnutls, which caused a link error.
Fixed with an extended regex.
Partial cherry-pick from 856576df99 (without the install check).
Re-insert the recurrence ID if not provided as part of the
data when updating an item. Necessary to support dumb local
storage which doesn't support UID/RECURRENCE-ID.
SyncEvolution now requires that "item not found" errors have
a 404 status code; the Synthesis engine depends on recognizing
such errors in some cases.
For calendars, the backend does all the checking itself and
doesn't event talk to the server.
For contacts, the same is done for deleting (because the
server accepts a delete request for a non-existent item
without complaining) and also detects the "not found error"
when reading (which would be nicer if it could be done
without string comparisons, but that doesn't seem to work yet).
The plain "char *" sync key used to be for a static buffer, now
is allocated dynamically. Transfer ownership of the buffer
to our eptr smart pointer, which will also throw an error if the
pointer is NULL.
activesyncd gets compiled completely (which should always succeed) and
then only the required pieces are installed (without the things which
are hard-coded for /usr, because installing those will fail).
Testing is done by running a similar set of client-test tests as for remove
CalDAV/CardDAV servers.
activesyncd is started anew each time client-test is run, done in
the new wrappercheck.sh script. Can be combined with valgrindcheck.sh:
wrappercheck.sh valgrindcheck.sh activesyncd -- valgrindcheck.sh client-test ...
The return code of wrappercheck.sh is the return code of the real command
or the daemon, if the real command succeeded. This way the special 100 return
code from valgrindcheck.sh is returned if the only problems were memory
issues.
When NEON_LIBS=-lneon-gnutls, the sed invocation didn't properly turn
that into -ldl. Instead it used -ldl-gnutls, which caused a link error.
Fixed with an extended regex. Also added a installcheck for this particular
aspect.
This is primarily for ActiveSync where the test failed until support
for removing properties was added to activesyncd. Is also applied to
all other sources, just in case.
The EDS contact backend needs to keep the X-EVOLUTION-FILE-AS property
because EDS keeps adding it, which makes it impossible to test its removal.
At the property level, the isDefault retval exposed whether the
property value was set explicitly in the config or taken from the
property default. That information got lost at the
SyncConfig/SyncSourceConfig level although there are cases where that
is relevant (like providing better error messages, BMC #23783).
Now that level uses the new InitState classes instead of plain
int/bool/std::string return values. Code which assigns these return
values to local variables doesn't need to be adapted. Directly using
the return value in an expression might need some work (typically
adding a get() if the compiler cannot infer the desired
type). Overriding the virtual methods always needs to be adapted.
If the engine got a parent event with X-SYNCEVOLUTION-EXDATE-DETACHED,
merged it internally and then wrote it back, the
X-SYNCEVOLUTION-EXDATE-DETACHED would have been stored in the CalDAV
server. Now this is avoided by removing all such properties before
storing the new or updated event.
This was previously done (and still is) as an extra precaution in the
code which adds the properties.
(cherry picked from commit ede6e65ccb)
The previous approach (updating the internal cache) had the drawback
that X-SYNCEVOLUTION-EXDATE-DETACHED was also sent to the CalDAV
server. The work of generating it was done in all cases, even if not
needed. Found when running the full test suite.
Now the X-SYNCEVOLUTION-EXDATE-DETACHED properties are only added to
the icalcomponent that is generated for the engine in
readSubItem(). There's still the risk that such an extended VEVENT
will be stored again (read/modify/write conflict resolution), so
further changes will be needed to filter them out.
To ensure that this change doesn't break the intended semantic of
X-SYNCEVOLUTION-EXDATE-DETACHED, the presence of these properties is
now checked in the LinkedItems::testLinkedItemsParentChild test.
(cherry picked from commit 1cd49e9ecd)
Required for Maemo 5 recurrences workaround.
icalproperty_get_value_as_string() is one of those
functions for which a _r variant exists; use that
if possible.
(cherry picked from commit 88b0cc2b62)
When deleting an item on phone and locally, the next sync fails with
ERROR messages about "object not found". This has several reasons:
- libsynthesis super data store attempts to read items
which may or may not exist (triggers ERROR message)
- it checks for 404 but Evolution backends only return a generic
database error (causes sync to fail)
It turned out that ReadItem and DeleteItem are expected to return a
404 status when the requested item does not exist. This patch documents
that (only in the TrackingSyncSource, though), adds tests and fixes
EDS, WebDAV, file and sqlite backends accordingly.
This patch also suppresses the 404 error logging inside DeleteItem(),
while still returning that error code to the Synthesis engine. Not
logging that particular situation is consistent with the previous
SyncEvolution behavior of silently returning successfully when there
wasn't anything to delete.
In addition, more recent libsynthesis versions also no longer do
a ReadItem() call to test for existence. That would still trigger
a spurious (albeit now harmless) ERROR message.
(cherry picked from commit ba289c899f)
Conflicts:
src/backends/webdav/CalDAVSource.cpp
test/ClientTest.cpp
test/ClientTest.h
The default implementation removes one VEVENT after the other, which
is slow for large merged events. Directly removing the entire event
series is faster.
When deleting an item on phone and locally, the next sync fails with
ERROR messages about "object not found". This has several reasons:
- libsynthesis super data store attempts to read items
which may or may not exist (triggers ERROR message)
- it checks for 404 but Evolution backends only return a generic
database error (causes sync to fail)
It turned out that ReadItem and DeleteItem are expected to return a
404 status when the requested item does not exist. This patch documents
that (only in the TrackingSyncSource, though), adds tests and fixes
EDS, WebDAV, file and sqlite backends accordingly.
This patch also suppresses the 404 error logging inside DeleteItem(),
while still returning that error code to the Synthesis engine. Not
logging that particular situation is consistent with the previous
SyncEvolution behavior of silently returning successfully when there
wasn't anything to delete.
In addition, more recent libsynthesis versions also no longer do
a ReadItem() call to test for existence. That would still trigger
a spurious (albeit now harmless) ERROR message.
If the engine got a parent event with X-SYNCEVOLUTION-EXDATE-DETACHED,
merged it internally and then wrote it back, the
X-SYNCEVOLUTION-EXDATE-DETACHED would have been stored in the CalDAV
server. Now this is avoided by removing all such properties before
storing the new or updated event.
This was previously done (and still is) as an extra precaution in the
code which adds the properties.
The previous approach (updating the internal cache) had the drawback
that X-SYNCEVOLUTION-EXDATE-DETACHED was also sent to the CalDAV
server. The work of generating it was done in all cases, even if not
needed. Found when running the full test suite.
Now the X-SYNCEVOLUTION-EXDATE-DETACHED properties are only added to
the icalcomponent that is generated for the engine in
readSubItem(). There's still the risk that such an extended VEVENT
will be stored again (read/modify/write conflict resolution), so
further changes will be needed to filter them out.
To ensure that this change doesn't break the intended semantic of
X-SYNCEVOLUTION-EXDATE-DETACHED, the presence of these properties is
now checked in the LinkedItems::testLinkedItemsParentChild test.
Required for Maemo 5 recurrences workaround.
icalproperty_get_value_as_string() is one of those
functions for which a _r variant exists; use that
if possible.
If the event has a DTSTART with TZID, then the EXDATE also should
have that same TZID. It is uncertain whether the backend provides
the TZID, but even if it does, because of the SIMPLE-EXDATE rule
the value wouldn't be parsed.
This must be done for regular EXDATE values in the EXDATE array field
(new SIMPLE-EXDATE rule) and for the addition EXDATE values created
for RECURRENCE-IDs in the EXDATES_DETACHED array field (new
HAVE-EXDATE-DETACHED-NO-TZID rule).
Both these rules are activated as subrules by the new MAEMO-CALENDAR
rule, which is set by the Maemo Calendar backend now.
There is one caveat the SIMPLE-EXDATE rule is also active when parsing
an EXDATE created by the storage and therefore TZID will be ignored,
if any is set at all (uncertain).
A vCalendar outgoing script could fix this by adding the DTSTART time
zone to the floating time value in the parsed EXDATEs.
Tell the engine to pass us EXDATEs created for each RECURRENCE-ID in a
detached recurrence. Necessary because the storage and app do not
support UID/RECURRENCE-ID and thus show duplicates without this
workaround.
Add X-SYNCEVOLUTION-EXDATE-DETACHED properties to main event for each
detached recurrence. Needed by some other SyncEvolution
backends (for example, Maemo 5).
?SyncEvolution=NoCTag as part of the syncURL disables change tracking
based on the CTag. Useful for simplifying the test logs because each
source instantiation starts with a full dump of the items.
