The same code will also be needed by the upcoming oauth2 backend. The
header file depends on glib, so don't install it as a public header
file of libsyncevolution (i.e. only allow using it internally).
The original signon code used a custom GVariant auto_ptr because it
needed the simple assignment semantic of "transfers ownership". Now
that the GVariantCXX class is in a shared header file, better make it
identical to the other CXX classes (i.e., a shared pointer). Avoid
returning plain pointers because ownership of those cannot be checked
by the compiler.
The signon case can be handled with GVariantStealCXX, which by default
takes ownership in the constructor and the default assignment
operator.
When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution
"databaseFormat", all parsing and conversion is skipped. The backend's
data is identical to the item data in the engine. Finding duplicates
in a slow sync is very limited when using these types because the entire
item data must match exactly.
This is useful for the file backend when the goal is to store an exact copy
of what a peer has or for limited, read-only backends (PBAP).
The implementation uses the "raw" base type which does nothing but storing
the item content verbatim. That type requires a field named "ITEMDATA". After
renaming the variable used by our MAKETEXTWITHPROFILE/PARSETEXTWITHPROFILE
with other, more complex types, all we have to do is skip this conversion
by not providing a profile name.
In a refresh-from-remote local sync of 1000 contacts with photo data
using the file backend, this reduces the number of CPU cycles from
2861476184 to 1840952543 on the server side.
The Synthesis API does not say explicitly, but in practice all map
items get updated in a tight loop. Rewriting the m_mappingNode (case
insensitive string comparisons) and serialization to disk
(std::ostrstream) consume a significant amount of CPU cycles and cause
extra disk writes.
It is better to allow changes to be cached in memory instead and only
flush to disk at the end. There is no explicit flush API, so we use
SaveAdminData as trigger for flushing instead.
In a refresh-from-remote local sync of 1000 contacts with photo data
using the file backend, this reduces the number of CPU cycles from
4138538276 to 2861476184 (measured with callgrind on x86_64).
Encoding/decoding of the uint8_t array in D-Bus took a surprisingly
large amount of CPU cycles relative to the rest of the SyncML message
processing. Now the actual data resides in memory-mapped temporary
files and the D-Bus messages only contain offset and size inside these
files. Both sides use memory mapping to read and write directly.
For caching 1000 contacts with photos on a fast laptop, total sync
time roughly drops from 6s to 3s.
To eliminate memory copies, memory handling in libsynthesis or rather,
libsmltk is tweaked such that it allocates the buffer used for SyncML
message data in the shared memory buffer directly. This relies on
knowledge of libsmltk internals, but those shouldn't change and if they
do, SyncEvolution will notice ("unexpected send buffer").
Unusually small values were ignored during syncing. Doing this in the config
access methods instead of the sync session setup ensures that all components
in SyncEvolution use the same value.
For this to work, the memory must be mapped MAP_SHARED (otherwise
writes are not visible outside of the process) and the recipient
of a file descriptor must be able to create a TmpFile for it, which
then can be used to create a memory mapping.
During TestCmdline.testSyncOutput also verify that syncs which don't
change the database also don't update the ~/.config meta data.
The listall() method from testpim.py is used and extended for that
and therefore moved to test-dbus.py.
The sync meta data (sync anchors, client change log) get updated after
a sync even if nothing changed and the existing meta data could be
used again. This can be skipped for local sync, because then
SyncEvolution can ensure that both sides skip updating the meta
data. With a remote SyncML server that is not possible and thus
SyncEvolution has to update its data.
This optimization is only used for local syncs with one source. It is
based on the observation that when the server side calls
SaveAdminData, the client has sent its last message and the sync is
complete. At that point, SyncEvolution can check whether anything has
changed and if not, skip saving the server's admin data and stop the
sync without sending the real reply to the client.
Instead the client gets an empty message with "quitsync" as content
type. Then it takes shortcuts to close down without finalizing the
sync engine, because that would trigger writing of meta data
changes. The server continues its shutdown normally.
This optimization is limited to syncs with a single source, because
the assumption about when aborting is possible is harder to verify
when multiple sources are involved.
During syncs where nothing changes, reseting the database revision and
setting it again at the end of the sync causes undesirable disk
writes. To avoid that, reset the database revision the first time an
item change is done.
Sometimes the new content of the file is exactly the same as the old
one, for example when the tracking node gets reset and
recreated. Detect that for all cases at the lower level by reading the
existing file first and comparing it against the new content.
Synthesis cleanup was supposed to happen between "closing session" and
"session closed" log messages. Because we kept additional references
to the session, this only happened later. Need to drop also references
to shared buffers, because those hold a reference to their session.
This allows signal handlers to affect the operation execution and
result without having to throw an exception, which would get logged as
error and is not desirable when the operation gets skipped
intentionally. Needed for skipping SaveAdminData in the next patch.
The engine gets instantiated twice: once for logging, once for
syncing. The config generated in each case is different because
sources only become active for syncing. Therefore the config change
check unnecessarily triggered during each sync (first getConfigXML
call created one hash, then the second another), causing DevInf to be
sent unnecessarily and always rewriting the .ini file.
The check is only needed when instantiating the engine that is getting
used for syncing. However, the config date must always be valid.
This adds "protocol: CardDAV" as a valid value, with corresponding
changes to the interpretation of some existing properties and some new
ones. The API itself is not changed.
Suspending a CardDAV sync is possible. This freezes the internal
SyncML message exchange, so data exchange with the CardDAV server may
continue for a while after SuspendPeer().
Photo data is always downloaded immediately. The "pbap-sync" flag
in SyncPeerWithFlags() has no effect.
Syncing can be configured to be one-way (local side is read-only
cache) or two-way (local side is read/write). Meta data must be
written either way, to speed up caching or allow two-way syncing. The
most common case (no changes on either side) will have to be optimized
such that existing meta data is not touched and thus no disk writes
occur.
A SuspendPeer() only succeeded while the underlying Bluetooth transfer
was active. Outside of that, Bluez errors caused SyncEvolution to
attempt a cancelation of the transfer and stopped the sync.
When the transfer was still queueing, obexd returns
org.bluez.obex.Error.NotInProgress. This is difficult to handle for
SyncEvolution: it cannot prevent the transfer from starting and has to
let it become active before it can suspend the transfer. Canceling
would lead to difficult to handle error cases (like partially parsed
data) and therefore is not done.
The Bluez team was asked to implement suspending of queued transfers
(see "org.bluez.obex.Transfer1 Suspend/Resume in queued state" on
linux-bluetooth@vger.kernel.org), so this case might not happen
anymore with future Bluez.
When the transfer completes before obexd processes the Suspend(),
org.freedesktop.DBus.Error.UnknownObject gets returned by
obexd. SyncEvolution can ignore errors which occur after the active
transfer completed. In addition, it should prevent starting the next
one. This may be relevant for transfer in chunks, although the sync
engine will also stop asking for data and thus typically no new
transfer gets triggered anyway.
When syncevo-dbus-server was shutting down due to a signal and there
was a sync session running that was triggered via the PIM Manager API,
then a segfault occurred because the manager instance got destroyed
before invoking a Session callback holding a pointer to it.
The fix is to use slot tracking of the weak pointer.
CTRL-C while waiting for the end of a sync causes an interactive
prompt to appear where one can choose been suspend/resume/abort and
continuing to wait. CTRL-C again in the prompt aborts the script.
The help text used single quotes for the JSON example instead of
the required double quotes. Running without --sync-flags was broken
because of trying to parse the empty string as JSON.
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
The log prefix seems to be unused (except for some debug message) at
the moment but will be soon, so make sure it gets embedded in the
string sent to the syncevo-dbus-server.
Will become relevant with the next commit, which moves the
source name from being part of the text into the prefix. So far,
the prefix is unused during Cmdline tests.
When deriving from DummySyncSource we need to implement less
pure virtual functions. Only the ones where the behavior
needs to differ from DummySyncSource are needed.
When syncing memos with a peer which also supports iCalendar 2.0 as
data format, the engine will now pick iCalendar 2.0 instead of
converting to/from plain text. The advantage is that some additional
properties like start date and categories can also be synchronized.
The code is a lot simpler, too, because the EDS specific iCalendar 2.0
<-> text conversion code can be removed.
When sending a VEVENT, DESCRIPTION was set to the SUMMARY if empty. This may
have been necessary for some peers, but for notes (= VJOURNAL) we don't know
that (hasn't been used in the past) and don't want to alter the item
unnecessarily, so skip that part and allow empty DESCRIPTION.
When receiving a plain text note, the "text/calendar+plain" type
used to store the first line as summary and the rest as description.
This may be correct in some cases and wrong in others.
The EDS backend implemented a different heuristic: there the first
line is copied into the summary and stays in the description. This
makes a bit more sense (the description alone is always enough to
understand the note). Therefore and to avoid behavioral changes
for EDS users when switching the EDS backend to use text/calendar+plain,
the engine now uses the same approach.
For this to work in all cases, the datatype must always call
MEMO_INCOMING_SCRIPT.
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
The syncURL property may contain multiple different space or tab
separated URLs. Previously, the WebDAV backend only used the first one
when scanning for databases. Now it tries all of them.
This will be useful for configuring all Google endpoints in one
template.
Certains domains (like googleapis.com) are aliases. The lookup tools
do not fail with NXDOMAIN in this case, causing the
syncevo-webdav-lookup script to return only an unknown lookup error,
in which case the WebDAV backend will retry for a while.
We need to detect aliases resp. missing SRV information and return a
"no information error".
The asyncError relied on calling
folks_individual_aggregator_remove_individual() incorrectly
with a NULL pointer. folks in Debian Testing just crashes
now, so we can no longer do this.
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google-calendar" will pick the "Google_Calendar"
template automatically because it knows that the intention is to
configure the target side of a local sync. It does not know that when
using some other name for the config, in which case the template (if
needed) must be specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
When configuring a WebDAV server with username = email address and no
URL (which is possible if the server supports service discovery via
the domain in the email address), then storing the credentials in the
GNOME keyring used to fail with "cannot store password in GNOME
keyring, not enough attributes".
That is because GNOME keyring seemed to get confused when a network
login has no server name and some extra safeguards were added to
SyncEvolution to avoid this.
To store the credentials in the case above, the email address now gets
split into user and domain part and together get used to look up the
password.
When doing PBAP caching, we don't want any meta data written because
the next sync would not use it anyway. With the latest libsynthesis
we can configure "/dev/null" as datadir for the client's binfiles and
libsynthesis will avoid writing them.
The PIM manager uses this for PBAP syncing automatically. For testing
it can be enabled by setting the SYNCEVOLUTION_EPHEMERAL env variable.
The recent change which introduced checking of the target session only
worked if the entries were listed in the "right" (= target-config
entry last) order. Don't reply on that, instead check for the one new entry
that we should find and use that.
As pointed out by compiler warnings on recent distros, the write()
result in the SuspendFlags signal handler was ignored. Not sure
whether it really can fail in practice. Handle some
possible errors with retrying.
The Cmdline.cpp unit tests did not check the symlink() result,
causing a warning resp. error depending on the configure options.
Reported by Emiliano Heyns.
This reverts commit c435e937cd.
Commit 7b636720a in libsynthesis fixes an unitialized memory read in
the asynchronous item update code path.
Testing confirms that we can now used batched writes reliably with EDS
(the only backend currently supporting asynchronous writes +
batching), so this change enables it again also for local and
SyncEvolution<->SyncEvolution sync (with asynchronous execution of
contact add/update overlapped with SyncML message exchanges) and other
SyncML syncs (with changes combined into batches and executed at the
end of each message).
Instead of downloading contacts one-by-one with GET, SyncEvolution now
looks at contacts that are most likely going to be needed soon and
gets all of them at once with addressbook-multiget REPORT.
The number of contacts per REPORT is 50 by default, configurable by
setting the SYNCEVOLUTION_CARDDAV_BATCH_SIZE env variable.
This has two advantages:
- It avoids round-trips to the server and thus speeds up a large
download (100 small contacts with individual GETs took 28s on
a fast connection, 3s with two REPORTs).
- It reduces the overall number of requests. Google CardDAV is known
to start issuing intermittent 401 authentication errors when the
number of contacts retrieved via GET is too large. Perhaps this
can be avoided with addressbook-multiget.
This is similar to read-ahead in EDS contacts. However, because we
cannot run Neon requests asynchronously (at least not easily), a
batched read must complete before any contact from it can be
returned to the caller.
Arbitrary sync flags can be passed to SyncPeerWithFlags() with the
new --sync-flags command line parameter. The value of that parameter
must be a JSON formatted hash.
The advantage of this approach over explicit command line parameters
(like --progress-frequency) is that we do not need to add further code
when adding new flags. We can also pass intentionally broken flags
to check the PIM Managers error handling.
The downside is that it is a bit less user-friendly.
With this change, progress frequency and PBAP sync can be set in two
ways, either as part of --sync-flags or with the individual command
line parameters. The latter override --sync-flags.
When the value of a flag has the wrong type, then dumping the value
may provide some hint about what is wrong. Note that values of types
not expected at all by the SyncFlags type already get filtered out
already by our D-Bus binding, in which case the error message shows
no value.
syncevo-webdav-lookup is needed during testing when doing
WebDAV database scans, so build it in "src" like the rest
of the binaries and link to it in the installed test suite.
