The issue with malloc being called in niam() in syncevo-dbus-server
and deadlocking was only found after adding gdb's output of the
syncevo-dbus-server stack backtrace.
"primary 80" ends up being recognized as phone number, adding the E.164
parameter if (and only if) EDS was compiled with libphonenumber support
(Debian Jessie!). Adding that breaks the comparison, so avoid the situation
by updating to a string which is not a phone number.
Akonadi uses different quoting for special characters. Probably would have to
be looked into and reported, but for now ignore it by not testing these
special cases.
When syncing Akonadi with file source, neither side recognizes duplicates
based on UID/RECURRENCE-ID. Should be added to Akonadi source. For now ignore
it.
When client-test starts, it determines all tests that would get
run, then runs all of them one-by-one in new instances. A single
test gets run directly.
The output changes slightly: the CppUnit summary with number of
failures or errors is no longer available and there are additional
blank lines between tests.
The advantage is that each single test is properly isolated from the
other, which is closer to how real syncs run. It also helps when
running under valgrind, because a leak can be attributed exactly to
one test and because it avoids permanently growing memory consumption
in long-running client-test runs (seen in the nightly testing even
when there were no leaks, perhaps because of memory fragmentation).
A potential downside of this change is that unexpected and undesirable
side effects of modules might no longer show up in testing, only when
combining them in real syncs. This should still be covered by
Client::Sync tests involving multiple modules.
When sent
SUMMARY:Simple
DESCRIPTION:Simple
Memotoo returns
SUMMARY:Simple
DESCRIPTION:Simple\nSimple
Remove this particular test case until the problem in the server
is fixed.
During TestCmdline.testSyncOutput also verify that syncs which don't
change the database also don't update the ~/.config meta data.
The listall() method from testpim.py is used and extended for that
and therefore moved to test-dbus.py.
This allows signal handlers to affect the operation execution and
result without having to throw an exception, which would get logged as
error and is not desirable when the operation gets skipped
intentionally. Needed for skipping SaveAdminData in the next patch.
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
The log prefix seems to be unused (except for some debug message) at
the moment but will be soon, so make sure it gets embedded in the
string sent to the syncevo-dbus-server.
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google-calendar" will pick the "Google_Calendar"
template automatically because it knows that the intention is to
configure the target side of a local sync. It does not know that when
using some other name for the config, in which case the template (if
needed) must be specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
The first progress signal gets emitted after sleeping for 10 seconds
at the start of the sync and then killing syncevo-dbus-server races
with completing the sync. What we want is to kill during the 10 second
wait, so we better wait for the debug output directly before it and
then kill directly.
syncevo-webdav-lookup is needed during testing when doing
WebDAV database scans, so build it in "src" like the rest
of the binaries and link to it in the installed test suite.
Google recently enhanced support for RECURRENCE-ID, so SyncEvolution
no longer needs to replace the property when uploading a single
detached event with RECURRENCE-ID. However, several things are still
broken in the server, with no workaround in SyncEvolution:
- Removing individual events gets ignored by the server;
a full "wipe out server data" might work (untested).
- When updating the parent event, all child events also get
updated even though they were included unchanged in the data
sent by SyncEvolution.
- The RECURRENCE-ID of a child event of an all-day recurring event
does not get stored properly.
- The update hack seems to fail for complex meetings: uploading them
once and then deleting them seems to make uploading them again
impossible.
All of these issues were reported to Google and are worked on there,
so perhaps the situation will improve. In the meantime, syncing with
Google CalDAV should better be limited to:
- Downloading a Google calendar in one-way mode.
- Two-way syncing of simple calendars without complex meeting
serieses.
While updating the Google workarounds, the alarm hack (sending a
new event without alarms twice to avoid the automatic server side
alarm) was simplified. Now the new event gets sent only once with a
pseudo-alarm.
Google seems to have changed its PHOTO rewriting. If that keeps
happening, we need to stop comparing the actual data.
However, comparing the actual data is useful to detect when we
do not properly handle it. For example, in testUpdateRemoteWins/local-synced
it was a bit surprising that only one contact had to be updated at first.
It turned out that libsynthesis did not compare the entire photo data,
only the part before embedded null bytes.
Funambol turned of the URL redirect from my.funambol.com to
onemedia.com. The Funambol template now uses the current URL. Users
with existing Funambol configs must updated the syncURL property
manually to https://onemediahub.com/sync
Kudos to Daniel Clement for reporting the change.
For performance reasons we only run selected Client::Sync
tests with Google CardDAV (each PUT/POST and DELETE take 10
seconds because of some intentional delay on the server).
Some Google-specific tests should also be run.
Traditionally, SyncEvolution only modified remote data via syncing.
This is not enough because it does not cover data on the remote side
that SyncEvolution cannot sync.
The new tests solve this by using the command line's import/update
operations which modify data more or less directly.
Now we can test:
- downloading data created remotely
- uploading data via sync, export directly, compare
- simulate conflicts with changes made remotely
The download and upload test include full round-trips, i.e.
the initial transfer plus a change that needs to be synced back.
Because this is highly dependent on the exact data stored by the
peer, all these tests depend on per-peer test data and scripts
for modifying that data. The tests only get enabled if the test
data is found.
The initial test data covers Apple Calendar server, EDS<->EDS
and Google Contacts.
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.
In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.
Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.
During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:
X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE
Any IMPP property declared as X-SERVICE-TYPE=AIM will get
mapped to X-AIM. Same for others. Some IMPP service types
have no known X- property extension; they are stored in
EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
(for example, GaduGadu and Groupwise) are stored with
X-SERVICE-TYPE values chosen by SyncEvolution so that
Google CardDAV preserves them (GroupWise with mixed case
got translated by Google into Groupwise, so the latter is used).
Google always sends an X-ABLabel:Other for IMPP. This is ignored
because the service type overrides it.
The value itself also gets transformed during the mapping. IMPP uses
an URI as value, with a chat protocol (like "aim" or "xmpp") and
some protocol specific identifier. For each X- extension the
protocol is determined by the property name and the value is the
protocol specific identifier without URL encoding.
X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel
The mapping is based on the X-ABLabel property attached to
the X-ABRELATEDNAMES property. This depends on the English
words "Spouse", "Manager", "Assistant" that Google CardDAV
and Apple devices seem to use regardless of the configured
language.
As with IMPP, only the subset of related names which have
a corresponding X- property extension get mapped. The rest
is stored in EDS using the X-ABRELATEDNAMES property.
X-ANNIVERSARY <-> X-ABDATE
Same here, with X-ABLabel:Anniversary as the special case
which gets mapped.
X-ABLabel parameter <-> property
CardDAV vCards have labels attached to arbitrary other properties
(TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
item1.X-ABDATE:2010-01-01
item1.X-ABLabel:Anniversary
The advantage is that property values can contain arbitrary
characters, including line breaks and double quotation marks,
which is not possible in property parameters.
Neither EDS nor KDE (judging from the lack of responses on the
KDE-PIM mailing list) support custom labels. SyncEvolution could
have used grouping as it is done in CardDAV, but grouping is not
used much (not at all?) by the UIs working with the vCards in EDS
and KDE. It seemed easier to use a new X-ABLabel parameter.
Characters which cannot be stored in a parameter get converted
(double space to single space, line break to space, etc.) during
syncing. In practice, these characters don't appear in X-ABLabel
properties anyway because neither Apple nor Google UIs allow entering
them for custom labels.
The "Other" label is used by Google even in case where it adds no
information. For example, all XMPP properties have an associated
X-ABLabel=Other although the Web UI does not provide a means to edit
or show such a label. Editing the text before the value in the UI
changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
other fields.
Therefore the "Other" label is ignored by removing it during syncing.
X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.
CardDAV needs to use test data with the new CardDAV vCard flavor.
Most CardDAV servers can store EDS vCards and thus
Client::Source::*::testImport passed (with minor tweaks in
synccompare) when using the default eds_contact.vcf, but
Client::Sync::*::testItems fails when comparing synced data with test
cases when the synced data uses the native format and the test cases
are still the ones from EDS.
A libsynthesis with URLENCODE/DECODE() and sharedfield parameter for
<property> is needed.
The previous, regular expression based reordering of parameters broke
parameters containing semicolons in quoted strings. The splitting must
be done using a proper parser which knows whether a semicolon
delimits a parameter (outside of quotation marks) or is part of the
parameter value (inside quotation marks).
synccompare now also normalizes the quoting of parameter values:
quotation marks are used for anything more complex than alphanumeric
plus underscore and hyphen.
Google CardDAV always adds the "Other" label to IMPP properties.
Ignore this by replacing the group of these two properties
with just the IMPP property. The normalization is intentionally
only done for two grouped properties. If we end up with more,
we need to check what that means instead of removing the label.
It's also more efficient and easier to implement this way.
Grouped properties are sorted first according to the actual property
name, then related properties are moved to the place where their group
tag appears first. The first grouped property gets a "- " prefix, all
following ones are just indended with " ". The actual group tag is not
part of the normalized output, because its value is irrelevant:
BDAY:19701230
- EMAIL:john@custom.com
X-ABLabel:custom-label2
...
FN:Mr. John 1 Doe Sr.
- IMPP;X-SERVICE-TYPE=AIM:aim:aim
X-ABLabel:Other
...
- X-ABDATE:19710101
X-ABLabel:Anniversary
Redundant tags (those set for only a single property) get removed as
part of normalizing an item.
testcase files used to be installed in src/testcases.am, using a duplicated
list of files. Better do it in test/test.am with the canonical list of files
and only install generated files in src/testcases.am. This makes it easier
to extend the list in the future.
As confirmed by Cyrus Daboo, Apple Calendar Server 5.2 should
return VTIMEZONE embedded in the item data if matching against
well-known timezones fails. This broken when Apple implemented
support for timezones via reference.
Long term we need to support that feature, but for now it and
this bug are not important because for most timezones, we should
be fine with our TZID based mapping. Ignore the issue during
testing...
This crept in during the conversion to the non-recursive build.
It was not a problem for "make dist", but when using the list in rules
it causes make warnings.
The test assumed that it can rename the main syncevo-dbus-server
executable to trigger the file watch mechanism. That's not correct:
- It might be the system's /usr/libexec/syncevo-dbus-server,
which a normal user cannot rename.
- The binary might be also active in some other, parallel tests.
Renaming it interferes with those other tests.
The latter happened in the nightly testing: HTTP server tests with
a long-running syncevo-dbus-server failed because the daemon terminated
during the tests.
Raising exceptions via throwError() looses the original source code
location information. Fixing that by explicitly passing that
information as additional parameter, created with the preprocessor
macro SE_HERE.
Several files included the complex SyncContext.h only because needed
throwError(). A better place for the revised implementation is the
Exception class which used to be in util.h, now Exception.h.
Simplifying the include statements exposed indirect include
dependencies and removed "using namespace std" from the compilation of
some source files which happened to rely on it. We want to get rid of
that name space polution, so fix the code instead of adding it back.
The WebDAV backends contained a hack where the UID inside the data was forced
to be identical to the resource name. This is wrong for items created by us
via POST (because the server may choose a resource name != UID) or by some
other entity (where we have no idea how the resource name got chosen).
This commit removes the hack. Testing must be updated to pass correct data
with the same UID as on the server when updating an item, because the backend
will no longer ensure that and changing the UID of a resource gets rejected by
some servers.
The hack was introduced for peers which do not store the UID (for example, a
vCard or iCalendar 1.0 based SyncML client). A better solution must be found,
probably involving the Synthesis engine and its read/update/write cycle.
PUT has the disadvantage that a client needs to choose a name and then
figure out what the real name on the server is. With Google CardDAV that
requires sending another request and only works because the server happens
to remember the original name (which is not guaranteed!).
POST works for new items without a name and happens to be implemented
by Google such that the response already includes all required
information (new name and revision string).
POST is checked for as described in RFC 5995 once before creating a new
item. Servers which don't support it continue to get a PUT.
The 403 status code must be checked to detect UID conflicts when adding
an item that already exists on the server.
When running Akonadi with the reference home dir, it creates
a socket symlink that we must not copy. Otherwise all Akonadi
instances end up sharing the same Unix domain socket name and
thus Akonadi instance.
