libphonenumber 1.6.x has the header files reorganized such that the
caller has to define how he is using the library. We want to use
boost.
Setting this doesn't hurt with older libphonenumber releases.
folks_note_field_details_new() takes an additional uid parameter.
Passing NULL is okay, but SyncEvolution wasn't doing that due to
an incorrect function type cast. Found by valgrind only after
a valgrind and tool chain update. Probably we passed a valid
value accidentally before.
Fixed by using a wrapper function.
Ideally the function typecast should get replaced entirely with just casting
the returned pointer.
The parsed number always has a country code, whereas SyncEvolution expected it
to be zero for strings without an explicit country code. This caused a caller
ID lookup of numbers like "089788899" in DE to find only telephone numbers in
the current default country, instead of being more permissive and also finding
"+189788899". The corresponding unit test was broken and checked for the wrong
result. Found while investigating an unrelated test failure when updating
libphonenumber.
EDS handles this differently, by calling ParseAndKeepRawInput() if necessary
(checked by configure) and looking at the source of the country code. Instead
of replicating that logic, let's use EPhoneNumber. This means that EDS has to
be compiled with libphonenumber support, because SyncEvolution can no longer
fall back to using libphonenumber directly.
For testing purposes it is still useful to not depend on X-EVOLUTION-E164.
testLocaledPhone uses this at the moment, because re-generating
X-EVOLUTION-E164 during a locale change seems to be broken at the moment
in the intel-work-3-12 branch.
The test itself has to be updated for the newer libphonenumber (6.1.1 instead
of 5.3.2). The "12345" string it relied upon now gets parsed consistently in
US and DE. Instead we use the "01164 3 331 6005" string (as in libphonenumber
tests) which is treated differently.
The "Fixed compilation error when using libphonenumber from revision
>= 568" patch caused a double free error because SetLogger() owns the
logger instance and, with libphonenumber >= r571 actually frees the
instance.
Old libphonenumber release are compatible with the revised call,
however, they never free the instance.
Empty field filter is supposed to mean "return all supported
fields". This used to work and stopped working with Android phones
after an update to 4.3 (seen on Galaxy S3); now the phone only
returns the mandatory TEL, FN, N fields.
The workaround is to replace the empty filter list with the list of
known and supported properties. This means we only pull data we really
need, but it also means we won't get to see any additional properties
that the phone might support.
The new "signon" provider only depends on lib[g]signon-glib. It uses
gSSO if found, else UOA. Instead of pulling parameters and the
identity via libaccounts-glib, the user of SyncEvolution now has to
ensure that the identity exists and pass all relevant parameters
in the "signon:" username.
This does not have to be user-friendly, so the machine-readable
GVariant text dump format is used to pass all parameters.
gSSO >= 2.0 requires a list of realms to which the identity
applies. We take this list from a new "Realms" setting for the
provider in Accounts.
This is how SyncEvolution does it; it hasn't been checked what
upstream will do around this. In Tizen, libaccounts is not used
and thus a different solution is needed there.
The API of libgsignond >= 2.0 was also changed to be more compatible
with the original libsignon. This allows (and requires) removing
some gSSO ifdefs.
The calendar home set URL on iCloud (the one ending in /calendars/) is
declared as containing calendar data. That was enough for
SyncEvolution to accept it incorrectly as calendar. However, the home
set only contains calendar data indirectly.
We must use the stricter check for leaf collections containing the
right data.
When finding a new URL, we must be prepared to reinitialize the Neon
session with the new host settings. To implement this, candidates are
now full URIs, not just paths on the initial host.
A home set on iCloud contains full URLs, not just paths. We need to
parse the individual entries, which happens to work for paths and URLs
because paths are just special URLS without an explicit host.
iCloud does not have .well-known support on its www.icloud.com
server. To support lookup with a non-icloudd.com email address, we
must do DNS SRV lookup when access to .well-known URLs fails. We do
this without a www prefix on the host first, because that is what happens
to work for icloud.com.
With these changes it becomes possible to do database scans on Apple
iCloud, using syncURL=https://www.icloud.com or
syncURL=https://icloud.com. The former is a bit faster because the
icloud.com redirects to www.icloud.com before we end up doing the DNS
SRV lookup to find the CalDAV resp. CardDAV hosts.
Treat URI with explicit port as equal to an URI where the port number
is implied by the scheme. Add compare() operation similar to
std::string::compare and add full set of compare operators based on it.
501 means "not implemented", in which case resending the same request
is unlikely to succeed. This is relevant for scanning iCloud, because
PROPFIND on http://www.icloud.com returns that.
Apple iCloud servers reject requests unless they contain a User-Agent
header. The exact value doesn't seem to matter. Making the string
configurable might be better, but can still be done later when it
is more certain whether and for what it is needed.
Moved into a separate function and fixed return code handling: due to
missing decoding of the shell scripts return code, all errors were
treated as potentially temporary errors and thus lookup was retried
instead of giving up immediately when there is no DNS SRV entry.
Funambol turned of the URL redirect from my.funambol.com to
onemedia.com. The Funambol template now uses the current URL. Users
with existing Funambol configs must updated the syncURL property
manually to https://onemediahub.com/sync
Kudos to Daniel Clement for reporting the change.
For performance reasons we only run selected Client::Sync
tests with Google CardDAV (each PUT/POST and DELETE take 10
seconds because of some intentional delay on the server).
Some Google-specific tests should also be run.
Traditionally, SyncEvolution only modified remote data via syncing.
This is not enough because it does not cover data on the remote side
that SyncEvolution cannot sync.
The new tests solve this by using the command line's import/update
operations which modify data more or less directly.
Now we can test:
- downloading data created remotely
- uploading data via sync, export directly, compare
- simulate conflicts with changes made remotely
The download and upload test include full round-trips, i.e.
the initial transfer plus a change that needs to be synced back.
Because this is highly dependent on the exact data stored by the
peer, all these tests depend on per-peer test data and scripts
for modifying that data. The tests only get enabled if the test
data is found.
The initial test data covers Apple Calendar server, EDS<->EDS
and Google Contacts.
When resolving a merge conflict, repeating properties were taken
wholesale from the winning side (for example, all email addresses). If
a new email address had been added on the loosing side, it got lost.
Arguably it is better to preserve as much data as possible during a
conflict. SyncEvolution now does that in a merge script by checking
which properties in the loosing side do not exist in the winning side
and copying those entries.
Typically only the main value (email address, phone number) is checked
and not the additional meta data (like the type). Otherwise minor
differences (for example, both sides have same email address, but with
different types) would lead to duplicates.
Only addresses are treated differently: for them all attributes
(street, country, city, etc.) are compared, because there is no single
main value.
When copying properties which may have labels, the new entries must be
added such that a unique label can be copied or set. Other properties
can be added at the end of their array.
The modern profile typically doesn't generate empty properties, and therefore
it is useless to create an IMPP, X-ABRELATEDNAMES or X-ABDATE entry with has
an empty value. It just has the effect of creating X-ABLabels which are not
attached to any property.
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.
In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.
Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.
During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:
X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE
Any IMPP property declared as X-SERVICE-TYPE=AIM will get
mapped to X-AIM. Same for others. Some IMPP service types
have no known X- property extension; they are stored in
EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
(for example, GaduGadu and Groupwise) are stored with
X-SERVICE-TYPE values chosen by SyncEvolution so that
Google CardDAV preserves them (GroupWise with mixed case
got translated by Google into Groupwise, so the latter is used).
Google always sends an X-ABLabel:Other for IMPP. This is ignored
because the service type overrides it.
The value itself also gets transformed during the mapping. IMPP uses
an URI as value, with a chat protocol (like "aim" or "xmpp") and
some protocol specific identifier. For each X- extension the
protocol is determined by the property name and the value is the
protocol specific identifier without URL encoding.
X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel
The mapping is based on the X-ABLabel property attached to
the X-ABRELATEDNAMES property. This depends on the English
words "Spouse", "Manager", "Assistant" that Google CardDAV
and Apple devices seem to use regardless of the configured
language.
As with IMPP, only the subset of related names which have
a corresponding X- property extension get mapped. The rest
is stored in EDS using the X-ABRELATEDNAMES property.
X-ANNIVERSARY <-> X-ABDATE
Same here, with X-ABLabel:Anniversary as the special case
which gets mapped.
X-ABLabel parameter <-> property
CardDAV vCards have labels attached to arbitrary other properties
(TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
item1.X-ABDATE:2010-01-01
item1.X-ABLabel:Anniversary
The advantage is that property values can contain arbitrary
characters, including line breaks and double quotation marks,
which is not possible in property parameters.
Neither EDS nor KDE (judging from the lack of responses on the
KDE-PIM mailing list) support custom labels. SyncEvolution could
have used grouping as it is done in CardDAV, but grouping is not
used much (not at all?) by the UIs working with the vCards in EDS
and KDE. It seemed easier to use a new X-ABLabel parameter.
Characters which cannot be stored in a parameter get converted
(double space to single space, line break to space, etc.) during
syncing. In practice, these characters don't appear in X-ABLabel
properties anyway because neither Apple nor Google UIs allow entering
them for custom labels.
The "Other" label is used by Google even in case where it adds no
information. For example, all XMPP properties have an associated
X-ABLabel=Other although the Web UI does not provide a means to edit
or show such a label. Editing the text before the value in the UI
changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
other fields.
Therefore the "Other" label is ignored by removing it during syncing.
X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.
CardDAV needs to use test data with the new CardDAV vCard flavor.
Most CardDAV servers can store EDS vCards and thus
Client::Source::*::testImport passed (with minor tweaks in
synccompare) when using the default eds_contact.vcf, but
Client::Sync::*::testItems fails when comparing synced data with test
cases when the synced data uses the native format and the test cases
are still the ones from EDS.
A libsynthesis with URLENCODE/DECODE() and sharedfield parameter for
<property> is needed.
The previous, regular expression based reordering of parameters broke
parameters containing semicolons in quoted strings. The splitting must
be done using a proper parser which knows whether a semicolon
delimits a parameter (outside of quotation marks) or is part of the
parameter value (inside quotation marks).
synccompare now also normalizes the quoting of parameter values:
quotation marks are used for anything more complex than alphanumeric
plus underscore and hyphen.
Google CardDAV always adds the "Other" label to IMPP properties.
Ignore this by replacing the group of these two properties
with just the IMPP property. The normalization is intentionally
only done for two grouped properties. If we end up with more,
we need to check what that means instead of removing the label.
It's also more efficient and easier to implement this way.
Grouped properties are sorted first according to the actual property
name, then related properties are moved to the place where their group
tag appears first. The first grouped property gets a "- " prefix, all
following ones are just indended with " ". The actual group tag is not
part of the normalized output, because its value is irrelevant:
BDAY:19701230
- EMAIL:john@custom.com
X-ABLabel:custom-label2
...
FN:Mr. John 1 Doe Sr.
- IMPP;X-SERVICE-TYPE=AIM:aim:aim
X-ABLabel:Other
...
- X-ABDATE:19710101
X-ABLabel:Anniversary
Redundant tags (those set for only a single property) get removed as
part of normalizing an item.
The code checking the session URL was never used because the
session did not exist yet during syncing. Adding a contactServer()
call fixes that.
The default WebDAV code in the dead code was redundant. The
default is set before checking for a specific server.
Different backend functions had to ensure that a connection
to the server existed. That work was meant to be done only
once during the existence of the backend instance, but due to
a missing "return" it was potentially (?) repeated multiple
times. Didn't cause any noticable problems.
The fields and their offsets were originally used for Synthesis vCard
extensions (X-CustomLabel-* and X-Synthesis-Ref). SyncEvolution never
used those. The extra fields cause unecessary overhead (logging,
merging, memory even when empty), so better remove them.
testcase files used to be installed in src/testcases.am, using a duplicated
list of files. Better do it in test/test.am with the canonical list of files
and only install generated files in src/testcases.am. This makes it easier
to extend the list in the future.
As confirmed by Cyrus Daboo, Apple Calendar Server 5.2 should
return VTIMEZONE embedded in the item data if matching against
well-known timezones fails. This broken when Apple implemented
support for timezones via reference.
Long term we need to support that feature, but for now it and
this bug are not important because for most timezones, we should
be fine with our TZID based mapping. Ignore the issue during
testing...
When doing a recursive scan of the home set, preserve the order of
entries as reported by the server and check the first one first. The
server knows better which entries are more relevant for the user (and
thus should be the default) or may have some other relevant
order. Previously, SyncEvolution replaced that order with sorting by
URL, which led to a predictable, but rather meaningless order.
For example, Google lists the users own calendar first, followed by
the shared calendars sorted alphabetical by their name. Now
SyncEvolution picks the main calendar as default correctly when
scanning from https://www.google.com/calendar/dav/.
Zimbra has a principal URL that also serves as home set. When using it
as start URL, SyncEvolution only looked the URL once, without listing
its content, and thus did not find the databases.
When following the Zimbra principal URL indirectly, SyncEvolution did
check all of the collections there recursively. Unfortunately that
also includes many mail folders, causing the scan to abort after
checking 1000 collections (an internal safe guard).
The solution for both includes tracking what to do with a URL. For the
initial URL, only meta data about the URL itself gets
checked. Recursive scanning is only done for the home set. If that
home set contains many collections, scanning is still slow and may run
into the internal safe guard limit. This cannot be avoided because the
CalDAV spec explicitly states that the home set may contain normal
collections which contain other collections, so a client has to do the
recursive scan.
When looking at a specific calendar, Google CalDAV does not report
what the current principal or the home set is and therefore
SyncEvolution stopped after finding just the initial calendar. Now it
detects the lack of meta information and adds all parents also as
candidates that need to be looked at. The downside of this is that it
doesn't know anything about which parents are relevant, so it ends up
checking https://www.google.com/calendar/ and
https://www.google.com/.
In both cases Basic Auth gets rejected with a temporary redirect to
the Google login page, which is something that SyncEvolution must
ignore immediately during scanning without applying the resend
workaround for "temporary rejection of valid credentials" that can
happen for valid Google CalDAV URLs.
The sorting of sub collections is redundant and can be removed,
because sorting already happens when storing the information for each
path. When scanning Google CalDAV starting at
https://www.google.com/calendar/dav/, the main calendar gets listed by
Google first, but because we reorder by sorting by path, it ends up
last in the SyncEvolution database list and thus is not picked as
default. This needs to be fixed separately by preserving the server's
order.
The previous change to the syncevo-webdav-lookup broke out-of-tree
compilation because it created a symlink to syncevo-webdav-lookup.sh
inside the current directory, which is not where the file resides when
compiling out-of-tree.
Instead of trying to be smart with symlinks (was only done for to avoid
a small file copy), simply copy the file.
syncevo-webdav-lookup.sh uses bash features (pipefail) and thus
must choose /bin/bash as shell explicitly. On systems with dash
as /bin/sh the script failed.
Server::run() still does some initialization (backend loading and file
watching). We need to wait with processing calls until that is done,
otherwise we have a race condition in TestFileNotify.testSession3:
- server starts
- test calls GetVersion()
- server answers as part of sending out log messages or loading backends (?),
sending incomplete information (no backends loaded yet)
- test touches syncevo-dbus-server
- server sets up watch
=> test fails because server missed the file modification and incorrect information returned
by GetVersions()
When we have attached a name to GDBusConnectionPtr in
dbus_get_bus_connection(), we must copy that name when
passing the connection on.
Otherwise requesting a name via dbus_get_bus_connection()
and later calling dbus_bus_connection_undelay() on a copy
fails to request the name.
This crept in during the conversion to the non-recursive build.
It was not a problem for "make dist", but when using the list in rules
it causes make warnings.
This takes advantage of the engine setting (new items) and preserving
(existing items) the UID for backends and then generating it as part of
serialization.
The older hacks in create/setResourceName() relied on resource name matching
the UID, which is not guaranteed for items created by us via POST (depending
on the server) and/or other CardDAV clients.
The hacks could be removed; they are simply not used anymore at the moment.
Before, the UID property in a vCard was ignored by the engine.
Backends were responsible for ensuring that the property is
set if required by the underlying storage.
This change moves this into the engine:
- UID is now field. It does not get used for matching
because the engine cannot rely on it being stored
by both sides.
- It gets parsed if present, but only generated if
explicitly enabled (because that is the traditional
behavior).
- It is never shown in the DevInf's CtCap
because the Synthesis engine would always show it
regardless whether a rule enabled the property.
That's because rules normally only get triggered
after exchanging DevInf and thus DevInf has to
be rule-independent. We don't want it shown because
then merging the incoming item during a local sync
would use the incoming UID, even if it is empty.
- Before writing, ensure that UID is set.
When updating an existing item, the Synthesis engine reads
the existing item, preserves the existing UID unless the peer
claims to support UID, and then updates with the existing UID.
This works for local sync (where SyncEvolution never claims
to support UID when talking to the other side). It will break
with peers which have UID in their CtCap although they
rewrite the UID and backends whose underlying storage cannot
handle UID changes during an update (for example, CardDAV).
A backend must still activate the UID property by setting m_backendRule
to LOCALSTORAGE-WITH-UID instead of the default LOCALSTORAGE.
Custom backend rules must include the new sub-rule HAVE-VCARD-UID.
See "[os-libsynthesis] UID + CardDAV" for a discussion of this change.
Long term it would be better to configure profiles without depending
on the rule hack.
The "show" attribute of the property values is not supported
and was silently ignored by the Synthesis engine and thus had
no effect. Remove the useless attributes.
Check whether a collection is read-only by reading ACL properties.
The check is intentionally a bit fuzzy to avoid accidentally marking
a collection as read-only.
All read-only collections are moved to the end of the found databases, to
avoid picking them as default when there are read/write collections. This
needs to be done in both "list all databases" mode (aka --print-databases)
and in "pick default database" mode (aka syncing with a source which
had no explicit 'database' set).
To use a read-only collection, configure it in the 'database'
property.
This is relevant for OwnDrive, aka ownCloud, which has a read-only "birthday"
calendar which was picked as default instead of the real calendar.
The WebDAV backend determines what database to use based on 'database',
'syncURL', and (for DNS SRV discovery) 'databaseUser' respectively
'username'. The new messages help figuring out which settings were
really used.
Read-only database are shown with a new "<read-only>" tag.
There is no explicit read/write flag, so a lack of that tag
is no guarantee that a database is really writable. It depends
on the backend whether it checks for write acccess.
Mateusz Polrola
Patrick Ohly