The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
A user reported via email that the Nokia 515 sends
'application/vnd.syncml+wbxml; charset=UTF-8' as type of its messages
this tripped up the syncevo-http-server, leading to:
[ERROR] syncevo-dbus-server: /org/syncevolution/Server: message type 'application/vnd.syncml+wbxml; charset=UTF-8' not supported for starting a sync
We need to strip the '; charset=UTF-8' suffix also when checking for
WBXML.
Sessions created via the Server.Connect() API were announced via
signals, but not activated. Therefore querying their status or
aborting them via D-Bus was not possible. Found while writing the
TestHTTP.testAbortThread D-Bus test.
Running the fork/exec implementation under valgrind caused
some tests to fail because a) some tests ran longer (fixed
by increasing timeouts) and b) some tests resulted in
different D-Bus communication depending on the timing.
Added more debug logging in the syncevo-dbus-server Connection
class and the syncevo-dbus-helper DBusTransport to track this
down.
There were multiple reasons, usually related to handling aborted
connections.
The D-Bus API explicitly says about the "Abort" signal sent by the
server: "This signal is sent at most once for each connection. No
reply will be sent on an aborted connection." The old code did send an
empty, final reply after aborting and the test-dbus.py actually
checked for it. Now that final message is really only send when the
connection is still waiting for a reply (state == PROCESSING) and
hasn't been aborted. The test was fixed accordingly.
The "Abort" documentation also says that "all further operations on it
[= Connection] will fail". Despite that comment one D-Bus test did a
Connection.Close() after receiving the Abort signal. The server now
destroys the Connection instance once it has failed and thus the
Close() call failed. It was removed.
The Connection class now consistently uses delayed deletion, instead
of destructing itself while some of its methods are still active. A
bit safer.
While thinking about the server<->helper communication I noticed that
a Connection.Close() succeeds even if the helper hasn't shut down
yet. Not sure whether there are relevant error scenarios where we
need to tell the client that shutdown of the helper failed.
This commit moves the blocking syncing, database restore and command
line execution into a separate, short-lived process executing the
syncevo-dbus-helper. The advantage is that the main
syncevo-dbus-server remains responsive under all circumstances (fully
asynchronous now) and suffers less from memory leaks and/or crashes
during a sync.
The core idea behind the new architecture is that Session remains the
D-Bus facing side of a session. It continues to run inside
syncevo-dbus-server and uses the syncevo-dbus-helper transparently via
a custom D-Bus interface between the two processes. State changes of
the helper are mirrored in the server.
Later the helper might also be used multiple times in a Session. For
example, anything related to loading backends should be moved into the
helper (currently the "is config usable" check still runs in the
syncevo-dbus-server and needs to load/initialize backends). The
startup code of the helper already handles that (see boolean result of
operation callback), but it is not used yet in practice.
At the moment, only the helper provides a D-Bus API. It sends out
signals when it needs information from the server. The server watches
those and replies when ready. The helper monitors the connection to
the parent and detects that it won't get an answer if that connection
goes down.
The problem of "helper died unexpectedly" is also handled, by not
returning a D-Bus method reply until the requested operation is
completed (different from the way how the public D-Bus API is
defined!).
The Connection class continues to use such a Session, as before. It's
now fully asynchronous and exchanges messages with the helper via the
Session class.
Inside syncevo-dbus-server, boost::signals2 and the dbus-callbacks
infrastructure for asynchronous methods execution are used heavily
now. The glib event loop is entered exactly once and only left to shut
down.
Inside syncevo-dbus-helper, the event loop is entered only as
needed. Password requests sent from syncevo-local-sync to
syncevo-dbus-helper are handled asynchronously inside the event loop
driven by the local transport.
syncevo-dbus-helper and syncevo-local-sync are conceptually very
similar. Should investigate whether a single executable can serve both
functions.
The AutoSyncManager was completely rewritten. The data structure is a
lot simpler now (basically just a cache of transient information about
a sync config and the relevant config properties that define auto
syncing). The main work happens inside the schedule() call, which
verifies whether a session can run and, if not possible for some
reasons, ensures that it gets invoked again when that blocker is
gone (timeout over, server idle, etc.). The new code also uses
signals/slots instead of explicit coupling between the different
classes.
All code still lives inside the src/dbus/server directory. This
simplifies checking differences in partly modified files like
dbus-sync.cpp. A future commit will move the helper files.
The syslog logger code is referenced by the server, but never used.
This functionality needs further thought:
- Make usage depend on command line option? Beware that test-dbus.py
looks for the "ready to run" output and thus startup breaks when
all output goes to syslog instead of stdout.
- Redirect glib messages into syslog (done by LogRedirect, disabled when
using LoggerSyslog)?
The syncevo-dbus-server now sends the final "Session.StatusChanged
done" signal immediately. The old implementation accidentally delayed
sending that for 100 seconds. The revised test-dbus.py checks for
more "session done" quit events to cover this fix.
Only user-visible messages should have the INFO level in any of the
helpers. Messages about starting and stopping processes are related to
implementation details and thus should only have DEBUG level.
The user also doesn't care about where the operation eventually
runs. All messages related to it should be in INFO/DEBUG/ERROR
messages without a process name. Therefore now syncevo-dbus-server
logs with a process name (also makes some explicit argv[0] logging
redundant; requires changes in test-dbus.py) and syncevo-dbus-helper
doesn't.
syncevo-local-sync is different from syncevo-dbus-helper: it produces
user-relevant output (the other half of the local sync). It's output
is carefully chosen so that the process name is something the user
understands (target context) and output can be clearly related to one
side or the other (for example, context names are included in the sync
table).
Output handling is based on the same idea as output handling in the
syncevo-dbus-server:
- Session registers itself as the top-most logger and sends
SyncEvolution logging via D-Bus to the parent, which re-sends
it with the right D-Bus object path as output of the session.
- Output redirection catches all other output and feeds it back
to the Session log handler, from where it goes via D-Bus
to the parent.
The advantage of this approach is that level information is made
available directly to the parent and that message boundaries are
preserved properly.
stderr and stdout are redirected into the parent and logged there as
error. Normally the child should not print anything. While it runs,
LogRedirect inside it will capture output and log it
internally. Anything reaching the parent thus must be from early
process startup or shutdown.
Almost all communication from syncevo-dbus-helper to
syncevo-dbus-server is purely information for the syncevo-dbus-server;
syncevo-dbus-helper doesn't care whether the signal can be
delivered. The only exception is the information request, which must
succeed.
Instead of catching exceptions everywhere, the optional signals are
declared as such in the EmitSignal template parameterization and
no longer throw exceptions when something goes wrong. They also don't
log anything, because that could lead to quite a lof of output.
Instead of overloading std::pair with additional meaning, use
GDBusCXX::DBusArray when we want to pass byte arrays. Allows defining
"normal" traits for std::pair.
API change, must be done in both D-Bus CXX layers and in the code using it.
DBusCallObject and friends were not used anywhere. Removed.
DBusObject and DBusRemoteObject used pure virtual methods to let
derived classes provide information about interface, path,
destination, method and the connection. The idea behind that was that
most of these strings are static and thus do not need to be copied.
The downside is that one had to derive from these classes in order to
provide the required information. The same class could not own two
instances of the generic DBusObject to access two different
destinations. It also sprinkled the code for setting up D-Bus
operations into several different places (constructor, class definition).
Now the information is passed to the DBus[Remote]Object constructor
and stored in the classes, which are thus merely containers for the
information and thus easier to use. Users of the classes still derive
from them, to keep the change smaller, although that is no longer
necessary.
Removed plain DBusConnection pointers from the C++ interface, return
DBusConnectionPtr instead. The exception is DBusObject, which still
returns plain pointers owned by the instance (as before) to ease
integration with the underlying D-Bus library. DBUS_CONNECTION_TYPE
is not needed.
This revealed that quite some code incorrectly took another reference
to the connection when assigning to DBusConnectionPtr (assignment
always increases the refcount, only in the constructor is that
optional). As a result the private connections were never
destructed. Apparently there were some global pointers to active
connections, so that this (minor) resource leak didn't show up in
valgrind.
It also showed that the closing of the D-Bus connection never happened
properly, although libdbus requires it. The ref counting mechanism
cannot be used for this because it cannot be checked whether the last
reference is about to be dropped. The slightly hackish solution is to
designate one DBusObject as the "main owner" of the connection. When
that object destructs, it closes the connection. There might still be
some other references; they simply cannot (and shouldn't) send or
receive messages anymore.
Patrick Ohly
Chris Kühl