Sometimes GNOME keyring and libsecret fail to set up the right temporary keys
(https://bugzilla.gnome.org/show_bug.cgi?id=778357). This has been fixed
upstream, but still breaks with the distros used by the automated testing
occassionally.
Retrying the operations after disconnecting from the server is an attempt
to recover from this sporadic error.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The GNOME keyring library has been obsoleted for a long time now,
long enough that the replacement libsecret is available on all
supported distros. Therefore we can switch unconditionally.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
libical v3 removes some deprecated functions (like icaltime_from_timet)
and removes the "is_utc" member from icaltimetype. The replacement
code works with old and new libical and thus needs no ifdefs.
However, that struct is part of the ABI, which impacts the tricks that
syncevolution.org binaries use to get built against libical v2 and then
run with more recent libs like libical v3.
Depending on the platform ABI, it may still be okay, because the calling code
in SyncEvolution reserves and copies enough bytes for the icaltimetype
instances and because that code never directly accesses any member (is_date,
is_daylight, zone) whose offset changes.
Original author: Milan Crha <mcrha@redhat.com>
Slightly modified it so that icaltime_t.zone is not set.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The account data was unreferenced once too often, or rather, a suitable ref
count increase was missing. A debug build of glib detects that ("GLib:
g_variant_unref: assertion 'value->ref_count > 0' failed"), but without that
check the code might also crash.
Prevents the wallet backend from crashing SyncEvolution when
enabled. Functionality not really tested, though.
PIM backend had compile problems when enabled.
When a backend is inactive, it is meant to ignore generic types like
"calendar". The idea behind that is that typically users install or
compile just the backends they want, and then ask for the "calendar"
backend using the generic sync templates or instructions.
When adding the TDEPIM calendar backend, that broke because it also
instantiated itself for those terms when active. The other TDEPIM
backends already did this as intended.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The latest activesyncd contains GSettings schema files
which get installed by the top-level makefile, therefore
we have to install everything.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The *Register.cpp files and everything they include must compile
without hard dependencies on header files which are TDE specific,
so add some ifdefs. Compiling with TDE backends disabled broke
because of this.
When enabled (untested!), it is unclear how some of these *Register.cpp
could have worked without including the header file that defines the
class they instantiate. Added the necessary includes.
A closing } was missing (found by cppcheck, which tests all variations
of the code, not just those currently enabled).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
syncevolution.org binaries are now getting compiled on Ubuntu Trusty and thus
no longer support distros with older EDS. The code should still compile
against older EDS (for example, for Maemo), but that is not getting tested
anymore.
This allows removing the dynamic linker hacks related to older libraries,
which was only used in those binaries. Instead, backends using libical or EDS
get compiled on Ubuntu Trusty and then the soname of those libs get patched to
make the backend module usable in combination with a different set of
libs. That patching is part of a script maintained in the syncevolution.org
build infrastructure.
This approach was already used before to generate different EDS backends
for EDS versions with the newer EClient API, because that turned out to be
easier than the dynamic loading approach. It works because none of the methods
used by SyncEvolution changed their ABI, only some other parts of the
libraries did. Should there ever be a situation again that cannot be handled
like this, then backends might also get compiled on different distros than
Ubuntu Trusty (however, that may lead to problems due to the libstdc++ ABI
changes - to be decided...).
libical still requires one special hack: system time zone loading in
libical v1 (and only in that version, v2 has builtin support again) must
be overridden such that time zones are generated with rules instead
of transitions because that is more compatible with the peers that
SyncEvolution exchanges data with.
That hack now relies on overriding the two relevant functions inside the main
binaries (has to be there, otherwise libical still ends up calling its own
internal implementation). The overriding code is in
libsyncevo-icaltz-util.so.0 and depends on libical.so.1. If
libsyncevo-icaltz-util.so.0 can be loaded, the wrappers in the main binary use
it, otherwise they fall through to the code from the current libical.so, which
then should be libical.so.2 or more recent.
This hack is active by default when libical v1 is detected during configuration.
Recent shells filter out environment variables that are not valid shell
variables, so for example,
SYNCEVOLUTION_FILE_SOURCE_DELAY_OPEN_addressbook-slow-server did not get
passed through to syncevo-dbus-server because of the hyphen. As a result, the
tests became unreliable (non-deterministic timing) or outright failed.
Now we ensure that these variables are valid also in a shell and in addition,
make the test stricter such that it detects when the file backend did not
wait.
Extracting a meaningful description of each item from the Synthesis
engine when updating and adding items is easy to do for items of
certain known types (contacts and calendar items) and arguably an
improvement; in particular it makes tests like
TestCmdline.testSyncOutput more realistic.
When sending an access token with insufficient scope (for example,
because the Ubuntu Online Accounts service definition was incomplete,
as documented in FDO #86824), Google responds with a 403 "service
denied" error.
Neon (arguably correctly) treats this as a permanent error and not
as a transient authentication error. Google should better send
a 401 error.
To activate the 401 error handling in SyncEvolution, detect this
special case and turn the general SE_ERROR error into SE_AUTH.
The Ubuntu Online Accounts signon backend complained about
invalid "username" content with an error message containing "sso",
the string used for gSSO.
Define these magic strings once and then use only the defines.
Use the information contained in the AgAuthData object to decide which
authentication method is being used.
Move the common authentication code into a private authenticate()
method.
[PO: fixed g_variant_builder_end() mem leak]
[PO: avoid issue found via cppcheck: returning std::string::c_str() in
a method returning a std::string forces a string copy, which is
inefficient.]
[PO: UiPolicy was not set because of an invalid g_variant_builder_add()
"sv" parameter.]
Instead of passing the session data and the mechanism, pass the full
AgAuthData object to the SignonAuthProvider constructor.
This object can be used to build the session data without converting all
dictionaries to and from GHashTable.
[PO: fixed g_variant_builder_end() mem leak]
[PO: ForceTokenRefresh was not set because of an invalid g_variant_builder_add()
"sv" parameter.]
It's not sync-evolution task to complete the account creation; in those
platforms where this backend is deployed (Ubuntu, Elementary, KDE) the
account should already be created with a proper SignonIdentity attached
to it.
Remove the failure count from the getOAuth2Bearer() method, and add an
invalidateCachedSecrets() method instead.
This moves the logic of how to deal with failures back to the
AuthProvider backend and simplifies the session code, which only needs
to call invalidateCachedSecrets() when the token is wrong.
This will help implementing a similar logic for the getCredentials()
method, where authentication errors could lead to requesting a new
password.
[PO: avoid issue found via cppcheck: returning std::string::c_str() in
a method returning a std::string forces a string copy, which is
inefficient.]
The libaccounts-based backends ("gsso" and "uoa") are going to use more
of libaccounts-glib, making sharing of code with the plain "signon"
plugin much harder to maintain.
Since 1.4.99.4, syncing WebDAV collections always checks first
whether there are items in the collections. This was partly done for
slow sync prevention (which is not necessary for empty collections),
partly for the "is the datastore usable" check.
However, this did not take into account that for CalDAV collections,
the entire content gets downloaded for this check. That is because
filtering by item type (VEVENT vs. VJOURNAL) is not implemented
correctly by all servers. So now all CalDAV syncs, whether incremental
or slow, always transfered all items, which is not the
intention (incremental syncs should be fast and efficient).
This commit adds a more efficient isEmpty() check: for simple CardDAV
collections, only luid and etag get transferred, as in
listAllItems(). This is the behavior from 1.5.
For CalDAV, a report with a filter for the content type is used and
the transfer gets aborted after the first item, without actually
double-checking the content of the item. This is different from
listAllItems(), which really transfers the content. This extra content
check would only be needed for some old servers (Radical 0.7) and is
not essential, because reporting "not empty" even when empty is safe.
Capture the item status and pass it to the response handler.
Response handlers are allowed to return a non-zero integer when using
the initAbortingReportParser(), which then aborts processing of the
response.
This leads to errors being returned by
ne_xml_dispatch_request(). Session::run() needs to be told if the
request was aborted, in which case all errors are ignored.
It turned out that finding databases on an Apple Calendar server accessed via
http depends on sending Basic Auth even when the server does not ask for it:
without authentication, there is no information about the current principal,
which is necessary for finding the user's databases.
To make this work again, sending the authentication header is now forced for
plain http if (and only if) the request which should have returned the
principal URL fails to include it. This implies sending the same request
twice, but as this scenario should be rare in practise (was only done for
testing), this is acceptable.
cppcheck warned about the non-portable initialization of a float
via memset. It is okay on all current architectures, but better
assign a real float 0 value anyway, while keeping the memset for
the rest of the elements.
The second memset() had already become redundant when adding the
PullParams constructor.
This fixes a crash/memory corruption issue in the plain signon backend
as used in Tizen.
The real bug was that HashTable2Variant() creates a GHashTableCXX by
stealing a floating GVariant without actually acquiring the reference,
leading to various glib errors caused by double free and/or invalid
memory access.
The code in signon.cpp receiving the GHashTableCXX was correct, albeit
overly complicated (creating a new reference via g_variant_ref_sink()
explicitly instead of relying on GHashTableCXX semantic).
The shared providersignon.so ended up being compiled with "gsso" as
prefix for the username. We need to check that this really works by also
testing for USE_ACCOUNTS.
Setting SYNCEVOLUTION_PBAP_CHUNK_TRANSFER_TIME to value <= 0 is not
working like described in README file.
Currently in such case desired chunk size is very quickly decreasing
to 0 and causing synchronization finish without downloading all
contacts.
This patch is disabling tuning desired chunk size in such case,
keeping transfer size constant and equal to initially set value.
That the backend is based on a refresh token is an implementation
detail; it might even change at some point if we figure out how
to do OAuth2 internally.
Better use the shorter "oauth2" name in the "username" property.
Also better align .so name and global variable names/defines in
configure and Makefile with the name of the backend.
There is no guarantee that the password can be updated. Allow the attempt
to fail without treating it as an error (that would be confusing in the most
common case, where "password" was given on the command line) and instead
just print an INFO message with the new token and some instructions.
All the values in the username hash are strings. This is unlikely to
change and if it does, we can always do some string-to-value
conversion in our own code. Therefore use the simpler syntax without
the angle brackets marking a variant value.
Simplifies our C++ code, too, and now also compiles on Ubuntu Trusty
(which doesn't have a recent enough glib for G_VARIANT_TYPE_VARDICT).
Avoid the term HMI because it is only used in certain communities.
Make the example username work for CalDAV and CardDAV and base it on
GOA, because that's what most users will have installed.
The only difference is the name of the module and the path to json.h,
which can be found as just json.h when using the -I statements from
the .pc files.
New backend implements identity provider for obtaining OAuth2 access
token for systems without HMI support.
Access token is obtained by making direct HTTP request to OAuth2 server
and using refresh token obtained by user in some other way.
New provider automatically updates stored refresh token when OAuth2
server is issuing new one.
So far, only the "username" property was used once identity providers
were involved. The upcoming oauth2 provider uses the "password"
property for the refresh token and needs the ability to store a new
token if the OAuth2 server updates it.
Setting the new value will not always be possible (for example,
when running a command line operation where all properties were
provided on the command line without a permanent config). This still
needs to be handled.
The same code will also be needed by the upcoming oauth2 backend. The
header file depends on glib, so don't install it as a public header
file of libsyncevolution (i.e. only allow using it internally).
The original signon code used a custom GVariant auto_ptr because it
needed the simple assignment semantic of "transfers ownership". Now
that the GVariantCXX class is in a shared header file, better make it
identical to the other CXX classes (i.e., a shared pointer). Avoid
returning plain pointers because ownership of those cannot be checked
by the compiler.
The signon case can be handled with GVariantStealCXX, which by default
takes ownership in the constructor and the default assignment
operator.
When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution
"databaseFormat", all parsing and conversion is skipped. The backend's
data is identical to the item data in the engine. Finding duplicates
in a slow sync is very limited when using these types because the entire
item data must match exactly.
This is useful for the file backend when the goal is to store an exact copy
of what a peer has or for limited, read-only backends (PBAP).
The implementation uses the "raw" base type which does nothing but storing
the item content verbatim. That type requires a field named "ITEMDATA". After
renaming the variable used by our MAKETEXTWITHPROFILE/PARSETEXTWITHPROFILE
with other, more complex types, all we have to do is skip this conversion
by not providing a profile name.
In a refresh-from-remote local sync of 1000 contacts with photo data
using the file backend, this reduces the number of CPU cycles from
2861476184 to 1840952543 on the server side.
A SuspendPeer() only succeeded while the underlying Bluetooth transfer
was active. Outside of that, Bluez errors caused SyncEvolution to
attempt a cancelation of the transfer and stopped the sync.
When the transfer was still queueing, obexd returns
org.bluez.obex.Error.NotInProgress. This is difficult to handle for
SyncEvolution: it cannot prevent the transfer from starting and has to
let it become active before it can suspend the transfer. Canceling
would lead to difficult to handle error cases (like partially parsed
data) and therefore is not done.
The Bluez team was asked to implement suspending of queued transfers
(see "org.bluez.obex.Transfer1 Suspend/Resume in queued state" on
linux-bluetooth@vger.kernel.org), so this case might not happen
anymore with future Bluez.
When the transfer completes before obexd processes the Suspend(),
org.freedesktop.DBus.Error.UnknownObject gets returned by
obexd. SyncEvolution can ignore errors which occur after the active
transfer completed. In addition, it should prevent starting the next
one. This may be relevant for transfer in chunks, although the sync
engine will also stop asking for data and thus typically no new
transfer gets triggered anyway.
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
When syncing memos with a peer which also supports iCalendar 2.0 as
data format, the engine will now pick iCalendar 2.0 instead of
converting to/from plain text. The advantage is that some additional
properties like start date and categories can also be synchronized.
The code is a lot simpler, too, because the EDS specific iCalendar 2.0
<-> text conversion code can be removed.
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
The syncURL property may contain multiple different space or tab
separated URLs. Previously, the WebDAV backend only used the first one
when scanning for databases. Now it tries all of them.
This will be useful for configuring all Google endpoints in one
template.
Certains domains (like googleapis.com) are aliases. The lookup tools
do not fail with NXDOMAIN in this case, causing the
syncevo-webdav-lookup script to return only an unknown lookup error,
in which case the WebDAV backend will retry for a while.
We need to detect aliases resp. missing SRV information and return a
"no information error".
This reverts commit c435e937cd.
Commit 7b636720a in libsynthesis fixes an unitialized memory read in
the asynchronous item update code path.
Testing confirms that we can now used batched writes reliably with EDS
(the only backend currently supporting asynchronous writes +
batching), so this change enables it again also for local and
SyncEvolution<->SyncEvolution sync (with asynchronous execution of
contact add/update overlapped with SyncML message exchanges) and other
SyncML syncs (with changes combined into batches and executed at the
end of each message).
