syncevo-webdav-lookup.sh uses bash features (pipefail) and thus
must choose /bin/bash as shell explicitly. On systems with dash
as /bin/sh the script failed.
Server::run() still does some initialization (backend loading and file
watching). We need to wait with processing calls until that is done,
otherwise we have a race condition in TestFileNotify.testSession3:
- server starts
- test calls GetVersion()
- server answers as part of sending out log messages or loading backends (?),
sending incomplete information (no backends loaded yet)
- test touches syncevo-dbus-server
- server sets up watch
=> test fails because server missed the file modification and incorrect information returned
by GetVersions()
When we have attached a name to GDBusConnectionPtr in
dbus_get_bus_connection(), we must copy that name when
passing the connection on.
Otherwise requesting a name via dbus_get_bus_connection()
and later calling dbus_bus_connection_undelay() on a copy
fails to request the name.
This takes advantage of the engine setting (new items) and preserving
(existing items) the UID for backends and then generating it as part of
serialization.
The older hacks in create/setResourceName() relied on resource name matching
the UID, which is not guaranteed for items created by us via POST (depending
on the server) and/or other CardDAV clients.
The hacks could be removed; they are simply not used anymore at the moment.
Before, the UID property in a vCard was ignored by the engine.
Backends were responsible for ensuring that the property is
set if required by the underlying storage.
This change moves this into the engine:
- UID is now field. It does not get used for matching
because the engine cannot rely on it being stored
by both sides.
- It gets parsed if present, but only generated if
explicitly enabled (because that is the traditional
behavior).
- It is never shown in the DevInf's CtCap
because the Synthesis engine would always show it
regardless whether a rule enabled the property.
That's because rules normally only get triggered
after exchanging DevInf and thus DevInf has to
be rule-independent. We don't want it shown because
then merging the incoming item during a local sync
would use the incoming UID, even if it is empty.
- Before writing, ensure that UID is set.
When updating an existing item, the Synthesis engine reads
the existing item, preserves the existing UID unless the peer
claims to support UID, and then updates with the existing UID.
This works for local sync (where SyncEvolution never claims
to support UID when talking to the other side). It will break
with peers which have UID in their CtCap although they
rewrite the UID and backends whose underlying storage cannot
handle UID changes during an update (for example, CardDAV).
A backend must still activate the UID property by setting m_backendRule
to LOCALSTORAGE-WITH-UID instead of the default LOCALSTORAGE.
Custom backend rules must include the new sub-rule HAVE-VCARD-UID.
See "[os-libsynthesis] UID + CardDAV" for a discussion of this change.
Long term it would be better to configure profiles without depending
on the rule hack.
The "show" attribute of the property values is not supported
and was silently ignored by the Synthesis engine and thus had
no effect. Remove the useless attributes.
Check whether a collection is read-only by reading ACL properties.
The check is intentionally a bit fuzzy to avoid accidentally marking
a collection as read-only.
All read-only collections are moved to the end of the found databases, to
avoid picking them as default when there are read/write collections. This
needs to be done in both "list all databases" mode (aka --print-databases)
and in "pick default database" mode (aka syncing with a source which
had no explicit 'database' set).
To use a read-only collection, configure it in the 'database'
property.
This is relevant for OwnDrive, aka ownCloud, which has a read-only "birthday"
calendar which was picked as default instead of the real calendar.
The WebDAV backend determines what database to use based on 'database',
'syncURL', and (for DNS SRV discovery) 'databaseUser' respectively
'username'. The new messages help figuring out which settings were
really used.
Read-only database are shown with a new "<read-only>" tag.
There is no explicit read/write flag, so a lack of that tag
is no guarantee that a database is really writable. It depends
on the backend whether it checks for write acccess.
This is relevant for WebDAV database scanning (read-only
databases should not be the default) and could also be used
to enhance automatic setups (for example, do not use two-way
syncing for read-only databases).
Include the name of the file which got modified. This helped track down why
the server sometimes shut down unexpectedly during parallel testing (main
executable was renamed by D-Bus testing).
Not all exceptions thrown while executing a source operation contain
the source name. SyncSource::throwError() does that, but SE_THROW() as
used in helper code does not. It is better to add the source name as
logging prefix. The other advantage is that all lines will have the
prefix, not just the first one.
The SyncSource operations need access to the SyncSource class which
contains them to access the display name of the SyncSource instance.
A positive a side effect of telling the wrappers about the instance at
construction time is that the caller no longer has to pass the source
reference.
This change allows removing the SyncSource::throwError() special
cases, which completes the transition towards having correct source
code location information for all exceptions.
This could happen when using item operations on the command line.
In that case a source without name was created and that empty
name was inserted as prefix before errors encountered while
using the source.
Now empty string is the same as no string.
Raising exceptions via throwError() looses the original source code
location information. Fixing that by explicitly passing that
information as additional parameter, created with the preprocessor
macro SE_HERE.
Several files included the complex SyncContext.h only because needed
throwError(). A better place for the revised implementation is the
Exception class which used to be in util.h, now Exception.h.
Simplifying the include statements exposed indirect include
dependencies and removed "using namespace std" from the compilation of
some source files which happened to rely on it. We want to get rid of
that name space polution, so fix the code instead of adding it back.
Additional databases where not found for several
reasons. SyncEvolution ignored all shared calendars
(http://calendarserver.org/ns/shared) and Google marks the additional
calendars that way. The other problem was that the check for leaf
collections (= collections which cannot contain other desired
collections) incorrectly excluded those collections instead of only
preventing listing of their content.
With this change,
https://www.google.com/calendar/dav/?SyncEvolution=Google can be used
as starting point for Google Calendar.
The WebDAV backends contained a hack where the UID inside the data was forced
to be identical to the resource name. This is wrong for items created by us
via POST (because the server may choose a resource name != UID) or by some
other entity (where we have no idea how the resource name got chosen).
This commit removes the hack. Testing must be updated to pass correct data
with the same UID as on the server when updating an item, because the backend
will no longer ensure that and changing the UID of a resource gets rejected by
some servers.
The hack was introduced for peers which do not store the UID (for example, a
vCard or iCalendar 1.0 based SyncML client). A better solution must be found,
probably involving the Synthesis engine and its read/update/write cycle.
PUT has the disadvantage that a client needs to choose a name and then
figure out what the real name on the server is. With Google CardDAV that
requires sending another request and only works because the server happens
to remember the original name (which is not guaranteed!).
POST works for new items without a name and happens to be implemented
by Google such that the response already includes all required
information (new name and revision string).
POST is checked for as described in RFC 5995 once before creating a new
item. Servers which don't support it continue to get a PUT.
The 403 status code must be checked to detect UID conflicts when adding
an item that already exists on the server.
The life time and content of the member was not defined. It got used
in multiple places. It's cleaner to bind the openPropCallback() to
a local Props_t instance.
The "addressbook" source was written for the original iPhone and
Mac OS X. It hasn't been in use for a long time, keeping the code
around without actually compiling it makes no sense anymore.
The "--update <dir name>" operation was supposed to take the
item luids from the file names inside the directory. That part
had not been implemented, turning the operation accidentally
into an "--import".
Also missing was the escaping/unescaping of luids. Now the
same escaping is done as in command line output and command
line parsing to make the luids safe for use as file name.
When aborting, our AbortHandler gets called to close down logging.
This may involve memory allocation, which is unsafe. In FDO #76375, a
deadlock on a libc mutex was seen.
To ensure that the process shuts down anyway, install an alarm and give
the process five seconds to shut down before the SIGALRM signal will kill
it.
The information that the sync is freezing is now also handed down to
the transport and all sources. In the case of PBAP caching, the local
transport notifies the child where the PBAP source then uses Bluez
5.15 Transfer1.Suspend/Resume to freeze/thaw the actual OBEX transfer.
If that fails (for example, not implemented because Bluez is too old
or the transfer is still queueing), then the transfer gets cancelled
and the entire sync fails. This is desirable for PBAP caching and
Bluetooth because a failed sync can easily be recovered from (just
start it again) and the overall goal is to free up Bluetooth bandwidth
quickly.
When using GIO, it is possible to avoid the DBusServer listening on a
publicly accessible address. Connection setup becomes more reliable,
too, because the D-Bus server side can detect that a child died
because the connection will be closed.
When using libdbus, the traditional server/listen and client/connect
model is still used. GDBus GIO mimicks the same API to minimize
changes in ForkExec. The API gets simplified to support both
approaches:
- Choosing an address is no longer possible (was only used in the
test example anyway)
- The new connection callback must be specified already when starting
the server.
The GDBus GIO implementation uses SyncEvolution utility code. Strictly
speaking, this creates a circular dependency
libsyncevolution<->libgdbus. This is solved by not linking libgdbus
against libsyncevolution and expecting executables to do that instead.
The GDBus GIO example no longer linked because of that; it gets removed
because it was of little value.
Now that we started using CLOEXEC, we might as well use it for the
permanent file descriptors created for each ForkExec instance.
g_clear_object() unnecessarily creates a dependency on a more recent glib
version. It probably isn't used correctly here anyway (g_clear_object() checks
for NULL itself, so the code around it doesn't need and should do that), so we
can use the older g_object_unref() instead.
The code is a verbatim copy of the corresponding files from gSSO. The
only difference is that we compile the implementation as C++ code,
because that is what we set compile flags for in libsyncevolution.
This helper class will be used for D-Bus over plain files in
combination with g_dbus_connection_new().
Currently implements the new API by stopping to consume data on the
local side of the sync. The Bluetooth transfer on the remote side
continues to run and isn't even notified about the suspend; need a new
obexd API to also suspend that and then notify the remote side when we
want it to do that.
This adds GetPeerStatus() and "progress" events.
To detect DB changes as they happen, the SyncSource operations are
monitored. Upon entry, a counter gets increased and transmitted
through to the PIM manager in syncevo-dbus-server using extended
SourceProgress structs (only visible internally - public API must not
be extended!). This will count operations which fail and count those
twice which get resumed, so the counts will be too high
occasionally. That is in line with the API definition; because it is
not exact, the API only exposes a "modified" flag.
Progress is reported based on the "item received" Synthesis event and
the total item count. A modified libsynthesis is needed where the
SyncML binfile client on the target side of the local sync actually
sends the total item count (via NumberOfChanges). This cannot be done
yet right at the start of the sync, only the second SyncML message
will have it. That is acceptable, because completion is reached very
quickly anyway for syncs involving only one message.
At the moment, SyncContext::displaySourceProgress() holds back "item
received" events until a different event needs to be emitted. Progress
reporting might get more fine-grained when adding allowing held back
events to be emitted at a fixed rate, every 0.1s. This is not done yet
because it seems to work well enough already.
For testing and demonstration purposes, sync.py gets command line
arguments for setting progress frequency and showing progress either
via listening to signals or polling.
The code was meant to check whether a contact really has contact
data; because the if check was missing, the sync result checking
in testSync failed for test data where one or more contacts had
not photos.
The function now keeps the event loop running while waiting for
completion. That ensures that events get received immediately.
testpim.py now matches the po scanning process because of the _(' text
pattern. We need to exclude it explicitly.
The is new API and flag grant control over the PBAP sync mode. It is
implemented by setting the SYNCEVOLUTION_PBAP_SYNC env variable in the
forked helper process.
The main advantage is that processed data can be discarded
immediately. When using a plain file, the entire address book must be
stored in it.
It also enables suspending a transfer by stopping to read from the
pipe, either via some internal API or simply freezing the
syncevo-local-sync process with SIGSTOP.
The drawback is that obexd does not react well to a full pipe. It
simply gets stuck in a blocking write(); in other words, all obexd
operations get frozen and obexd stops responding on D-Bus.
SyncEvolution was meant to load the syncecal or syncebook shared object
which uses the most recent libraries (libical, libecal/libebook) on
the system and then stop loading further backends. Due to a string
handling bug the check for already backends always found nothing,
leading to multiple conflicting backends loaded on some systems (for
example, those with libical0 and libical1 installed).
If that happened, the backend became unusable.
Nightly testing uses a "Client_Sync_Current" symlink to point
the server to a location where it shall write its per-test logs
without having to reconfigure the server.
When the server-side session runs longer than a specific test, it ends up
using the "Client_Sync_Current" link of the next test. "permission denied"
errors were also seen, although it is less certain whether that had the same
root cause.
To avoid such issues, the sync session now tries to resolve the log dir
path to an absolute path at the start of the session.
The KDE Notes resources store items under a different MIME type than the one
used in AKonadi (see "[Kde-pim] note format"). SyncEvolution use the same type
as Akonadi and thus did not find existing KDE Notes resources.
To support both while KDE and Akonadi transition to the same type,
SyncEvolution now looks for notes resources using both MIME types and accepts
both kinds of items when reading. When writing, SyncEvolution picks the MIME
type that is supported by the resource, which hopefully avoids confusing the
KDE app using the resource (untested).
As a positive side effect, the "database" value used for opening a resource is
now checked more thoroughly. Non-existent resources and the type mismatches
like pointing a "kde-contacts" backend to a calendar resource are now detected
early.
Akonadi resources do not enforce iCalendar 2.0 semantic like
"each VEVENT must have a UID" (see "[Kde-pim] iCalendar semantic").
When receiving an event from a peer which itself does not enforce
that semantic (Funambol, vCalendar 1.0 based phones), then we
need to generate a UID, otherwise KOrganizer will ignore the
imported event.
Akonadi and Akonadi resources do not enforce iCalendar 2.0 semantic
like "UID+RECURRENCE-ID must be unique in a collection". Therefore we
must disable all tests expecting that behavior.
Regular syncing should work okay as long as the peer behaves. A misbehaving
peer will be able to send us invalid sets of items and syncing will not be
able to detect that because the sync engine itself is also agnostic to the
special iCalendar 2.0 semantic.
See "[Kde-pim] iCalendar semantic".
When used as storage in a server, Akonadi got called in a background thread
that gets created to handle slow initialization of sources and preventing
ensuing timeouts in HTTP clients (probably not needed for Akonadi itself,
but may still be useful when combining it with other sources).
Akonadi cannot be used like that, leading to false "Akonadi not running"
errors or (if one got past that check) failing item operations.
This commit avoids the situation by shifting the work back into the main
thread.
