When running test-dbus.py or testpim.py with SYNCEVOLUTION_DEBUG
set, they now skip the redirection of syncevo-dbus-server output
into a file. Useful when doing interactive debugging without running
under a debugger.
When running syncevo-dbus-server under valgrindcheck.sh,
the following happened occasionally:
- syncevo-dbus-server main thread quits, some threads keep running
=> ps shows the process as <defunct> with ppid = 1 = init
- valgrindcheck.sh notices that the process is done,
reports status and quits
- test-dbus.py fails to wait for the syncevo-dbus-server
process (because it is not the parent) and assumes that
the process is gone
At this point there is a lingering process which occupies the
well-known D-Bus name (= all further tests fail) and which
prevents unmounting the chroot.
It's unknown how the syncevo-dbus-server gets into that state.
Could be valgrind 3.7.0 or the kernel 3.4.28-2.20-xen.
As a workaround, let test-dbus.py collect the pids of all processed that it
couldn't wait for and send them SIGKILLs until that returns with "not
found".
When starting the server fails, an exception gets thrown when
trying to determine its pid. This used to abort the whole
test script without recording much information. In particular
the server's output was not included.
Now the exception is caught, recorded as error and testing continues
with the next test.
This did not fix the root cause (a stuck process occupied the
D-Bus name) but at least it helped to identify the problem.
Adds the possibility to check the servers standard output
similar to its D-Bus log output. Both can now also be set
before invoking runTest() because that method no longer
sets the members.
Log message now contain time stamps. A NullLogging class mimics the
Logging interface and can be used instead of that to suppress logging.
As a side effect of turning the log() method into a wrapper, the D-Bus
signal is now called "log2", which makes it possible to search for it
case-insensitively in emacs without finding the LogOutput signal.
The dbus-monitor output can be very large. Handle that a bit better by
compressing the file with a gzip pipe. Experimental and a bit broken:
output is not flushed properly when killing dbus-monitor + gzip.
An different approach is taken when runUntil() is called with
may_block=True: it lets the main loop run for 0.5 seconds and then
returns to do the status checking. No logging is done for each check.
This is meant for long-running operations where the 0.5 second latency
doesn't matter and too frequent checking and logging cause overhead.
The default code path checks every 0.1 second and recognizes
progress (defined as "there was an event to be processed"), which then
causes debug logging about waiting.
Sending DEBUG messages via the LogOutput signal is expensive and often
not necessary. Ideally LogOutput should only send data that someone is
interested in. The problem is that the SyncEvolution D-Bus API has no
way of specifying that and some clients, like dbus-monitor, would not be
able to use it.
Therefore this commit makes the default level of detail configurable
when syncevo-dbus-server is started, via a separate --dbus-verbosity
option.
This gets applied to output generated from syncevo-dbus-server itself
as well as output from sync sessions.
test-dbus.py exposes that via a new TEST_DBUS_QUIET option. The
default is to include DEBUG output. When TEST_DBUS_QUIET is set to a
non-empty string, only ERROR and INFO messages are sent.
The new default is to log error messages to syslog.
Command line options can increase (or reduce) verbosity
and choose whether stdout and/or syslog shall be active:
-d, --duration=seconds/'unlimited'
Shut down automatically when idle for this duration
-v, --verbosity=level
Choose amount of output, 0 = no output, 1 = errors,
2 = info, 3 = debug; default is 1.
-o, --stdout
Enable printing to stdout (result of operations) and
stderr (errors/info/debug).
-s, --no-syslog
Disable printing to syslog.
To get the same behavior as before when debugging via the Python
scripts, "--no-syslog --stdout --verbosity=3" is passed to
syncevo-dbus-server when started via Python.
Found while (accidentally) setting an ENV variable with a = inside the
value part: splitting into name and value may only split once, at the first
equal sign.
The notification texts were already localized, but using localization was not
turned on in the D-Bus server. The code which was meant to turn it on was
incomplete.
The setlocale() is required and was only done indirectly in syncevolution.org
binaries if the KDE platform module was installed. Now it is done in main() as
part of the process setup. Removed the incomplete setup code from the notify
backend completely (instead of fixing it) because a plugin should not modify
the process state like that.
As part of adding a test for this feature, a new environment was added: the
SYNCEVOLUTION_LOCALE_DIR environment variable can be set to the locale
directory which contains the different translations.
The English default text was already tested. The German translation gets
activated via env variables in the same test and checks that localization
works.
Now that translation is working, better ensure that English is selected
in those tests which expect English notifications.
Renaming "listing" to "acessing" broke testPrintDatabases (because
it needs to suppress that error for KDE, which is not running
during D-Bus testing) and introduced a typo.
Avoid printing the message in runUntil() every 0.1 second.
When running syncevo-dbus-server in a debugger, print to stdout so
that the developer can follow the progress of the Python test.
Allow checks to change their result without changes inside the event
loop. Previously, the checks were only invoked if there were
events inside the event loop.
Necessary for checking events for a certain period of time.
unittest.assertEqual() works for standard Python to dbus type
comparisons, but the output for mismatches is very unreadable. The
custom version which is now used everywhere reduces the dbus types to
their Python counterparts, which makes the error output much more
useful (diff with Python 2.7).
Also added an optional sortLists parameter which, if true, normalizes
the lists in the values. This is necessary for D-Bus APIs which return
lists without guaranteeing a specific order of the elements.
Avoid race condition after starting gdb by simply checking for the
presence of syncevo-dbus-server on the session bus. Previously the
output was checked, which failed due to race conditions and/or
unexpected output when setting breakpoints (not investigated further).
Once the test failed, print the failure or error immediately instead
of in the summary after gdb quit. This is information which might be
relevant for debugging the running syncevo-dbus-server.
After gdb quit, don't attempt to read the syncevo-dbus-server output
file (doesn't exit) and don't dump the D-Bus output to the screen
(usually too long to read without less, which is not possible when
doing interactive debugging). Instead point to the log file.
When TEST_DBUS_GDB=1 is set, syncevo-dbus-server gets run under gdb.
That gdb is passed the modified environment with a (potentially)
replaced HOME. In that case gdb needs an explicit -x parameter,
otherwise it won't find the user's .gdbinit.
It was observed that raising an exception in the signal handler did
not propagate to the calling Python code when it was currently in
loop.get_context().iteration().
This commit adds debug logging for timeouts and some utility methods
which avoid the problem by adding logging to the iteration()
call. Python then re-raises the exception in the D-Bus logging code.
This looks like a deficiency (bug?!) of the Python bindings for
iteration() which is merely worked around.
However, the utility methods may also be useful by their own right,
for example to avoid the awkward loop.run() + loop.quit() pairs.
Looking up the method via eval() no longer worked when test-dbus.py
was used as submodule. Better look up the function in the current
instance of the class via the method name, which itself can be
extracted from the current test id.
Creating a database is only possible with a chosen name. The UID is
chosen automatically by the storage. A new property would be needed
to also specify the UID.
There are no command line tests for the new functionality because
support for it in backends is limited.
While waiting for the processes to quit, at least sleep for short
periods of time. Generally a good idea, and seems to be necessary
on the new nightly test setup to give the processes enough CPU time
to really shut down.
Running EDS and possibly Akonadi caused dbus-session.sh to
return a failure despite the actual command working fine,
because the daemons returned 133 as status. Avoid starting
them in the first place, because they are not needed.
This patch introduces support for true one-way syncing ("caching"):
the local datastore is meant to be an exact copy of the data on the
remote side. The assumption is that no modifications are ever made
locally outside of syncing. This is different from one-way sync modes,
which allows local changes and only temporarily disables sending them
to the remote side.
Another goal of the new mode is to avoid data writes as much as
possible.
This new mode only works on the server side of a sync, where the
engine has enough control over the data flow.
Most of the changes are in libsynthesis. SyncEvolution only needs to
enable the new mode, which is done via an extension of the "sync"
property:
- "local-cache-incremental" will do an incremental sync (if possible)
or a slow sync (otherwise). This is usually the right mode to use,
and thus has "local-cache" as alias.
- "local-cache-slow" will always do a slow sync. Useful for
debugging or after (accidentally) making changes on the server side.
An incremental sync will ignore such changes because they are not
meant to happen and thus leave client and sync out-of-sync!
Both modes are recorded in the sync report of the local side. The
target side is the client and records the normal "two-way" or "slow"
sync modes.
With the current SyncEvolution contact field list, first, middle and
last name are used to find matches during any kind of slow sync. The
organization field is ignored for matching during the initial slow
sync and used in all following ones. That's okay, the difference won't
matter in practice because the initial slow sync in PBAP caching will
be done with no local data. The test achieve the same result in both
cases by keeping the organization set in the reduced data set.
It's also okay to include the property in the comparison, because it
might help to distinguish between "John Doe" in different companies.
It might be worthwhile to add more fields as match criteria, for
example the birthday. Currently they are excluded, probably because
they are not trusted to be supported by SyncML peers. In caching mode
the situation is different, because all our data came from the peer.
The downside is that in cases where matching has to be done all the
time because change detection is not supported (PBAP), including the
birthday as criteria will cause unnecessary contact removed/added
events (and thus disk IO) when a contact was originally created
without birthday locally and then a birthday gets added on the phone.
Testing is done as part of the D-Bus testing framework, because usually
this functionality will be used as part of the D-Bus server and writing
tests in Python is easier.
A new test class "TestLocalCache" contains the new tests. They include
tests for removing extra items during a slow sync (testItemRemoval),
adding new client items under various conditions (testItemAdd*) and
updating/removing an item during incremental syncing
(testItemUpdate/Delete*). Doing these changes during a slow sync could
also be tested (not currently covered).
The tests for removing properties (testPropertyRemoval*) cover
removing almost all contact properties during an initial slow sync, a
second slow sync (which is treated differently in libsynthesis, see
merge=always and merge=slowsync), and an incremental sync.
When delaying processes as part of the sync, disable the normal
test timeouts. Useful for attaching to the delayed process
with gdb and then debugging without running into the test
timeouts.
When syncevo-dbus-server was started on demand by the D-Bus daemon,
then it registered itself with the daemon before it was ready to
serve requests. Only happened in combination with GIO D-Bus and
thus was not a problem before 1.2.99.x.
One user-visible effect was that the GTK UI did select the default
service when it was started for the first time, because it could not
retrieve that information from syncevo-dbus-server.
The fix consists of delaying the name acquisition. That gives the
caller a chance to register D-Bus objects first, before completing the
connection setup. The semantic change is that
dbus_bus_connection_undelay() must be called on new connections which
have a name (a NOP with libdbus).
This patch tries to minimize code changes. The downside of not
changing the GDBusCXX API more radically is that the bus name must be
attached to DBusConnectionPtr, where it will be copied into each
reference to the connection. Hopefully std::string is smart enough to
share the (small) data in this case. Should be solved cleaner once
libdbus support can be deprecated.
A test for auto-activation and double start of syncevo-dbus-server
is also added.
Auto syncing was not getting triggered when using an autoSyncDelay > 0;
by default it is 5 minutes. Thanks to Vladimir Elisseev for reporting
this problem.
The root cause was a "greater than" instead of a "less than"
comparison. This was not found by the tests because they set
autoSyncDelay to zero to speed up testing.
Changing one test (TestSessionAPIsDummy.testAutoSyncNetworkFailure) so
that it uses non-zero autoSyncDelay triggered the problem. Also added
a variation of that test that simulates the "no Connman and no
NetworkManager" setup used by Vladimir.
valgrind apparently got a bit slower. Previous timeouts were too
small, had to be increased to get tests to pass when using valgrind.
Found some more tests which result in unpredictable error messages
when processes failed. Allow multiple outcomes.
If the command line tool returned before test-dbus.py recorded
all D-Bus events of that session, then the check of the events
failed. Need to wait until "status: done" is also seen via D-Bus.
Not getting it will result in test timeouts.
Instead of using delays to kill processes at the right time, watch old
and new debug output via D-Bus and then kill the processes. To avoid
race conditions, these processes get delayed at the right point.
Added tests for local sync and command line, covering killing of
all involved processes.
Added a "target side of local sync ready" INFO message to introduce
the output which has the target context in the [INFO] tag. The sync report
from the target side now has the target context embedded in brackets
after the "Changes applied during synchronization" header, to avoid
ambiguities.
Output from the target side of a local sync was passed through stderr
redirection as chunks of text to the frontends. This had several
drawbacks:
- forwarding only happened when the local sync parent was processing
the output redirection, which (due to limitations of the implementation)
only happens when it needs to print something itself
- debug messages were not forwarded
- message boundaries might have been lost
In particular INFO messages about delays on the target side are
relevant while the sync runs and need to be shown immediately.
Now the output is passed through D-Bus, which happens immediately,
preserves message boundaries and is done for all output. The frontend
can decide separately whether it shows debug messages (not currently
supported by the command line tool).
Implementing this required extending the D-Bus API. The
Server.LogOutput signal now has an additional "process name"
parameter. Normally it is empty. For messages originating from the
target side, it carries that extra target context string.
This D-Bus API change is backward compatible. Older clients can still
subscribe to and decode the LogOutput messages, they'll simply ignore
the extra parameter. Newer clients expecting that extra parameter
won't work with an older D-Bus daemon: they'll fail to decode the
D-Bus message.
This revealed that the last error messages in a session was
incorrectly attributed to the syncevo-dbus-server. Might also have
happened with several other error messages. Now everything happening
in the server while working on code related to a session is logged as
coming from that sessions. It's not perfect either (some of the output
could be from unrelated events encountered indirectly while running
that code), but it should be better than before.
The patch changes the handling or errors in the local sync parent
slightly: because it logs real ERROR messages now instead of plain
text, it'll record the child's errors in its own sync report. That's
okay, user's typically shouldn't have to care about where the error
occurred. The D-Bus tests need to be adapted for this, because it
removes the "failure in local sync child" from the sync report.
Another, more internal change is that the syncevo-local-sync helper
does its own output redirection instead of relying on the stderr
handling of the parent. That way libneon debug output ends up in the
log file of the child side (where it belongs) and not in the parent's
log.
Added INFO output about checking sources. This helps with WebDAV when
the server cannot be contacted (dead, misconfigured) because otherwise
there would be no indication at all why the --configure operation
seems to hang.
Here is some example output, including aborting:
$ syncevolution --configure --template webdav \
syncURL=http://192.168.1.100:9000/ \
username=foo password=bar retryDuration=2s \
target-config@webdav-temp
[INFO] creating configuration target-config@webdav-temp
[INFO] addressbook: looking for databases...
[INFO] addressbook: no database to synchronize
[INFO] calendar: looking for databases...
[INFO] calendar: no database to synchronize
[INFO] memo: looking for databases...
[INFO] memo: no database to synchronize
[INFO] todo: looking for databases...
[INFO] todo: no database to synchronize
It timed out fairly quickly here because of the retryDuration=2s. That
also gets placed in the resulting config, which is probably not desired.
$ syncevolution --configure \
--template webdav \
syncURL=http://192.168.1.100:9000/ \
username=foo password=bar \
target-config@webdav-temp
[INFO] creating configuration target-config@webdav-temp
[INFO] addressbook: looking for databases...
^C[INFO] Asking to suspend...
[INFO] Press CTRL-C again quickly (within 2s) to stop immediately (can cause problems in the future!)
^C[INFO] Aborting immediately ...
[ERROR] error code from SyncEvolution aborted on behalf of user (local, status 20017): aborting as requested by user
It would be good to make the CTRL-C handling code aware that it can
abort immediately instead of doing the intermediate "asking to suspend"
step, which only makes sense for sync sessions.
Also added task and memo sources to the WebDAV template.
Both requires changes to the testing. Instead of updating the C++
CmdlineTest, the affected tests were removed and only the
corresponding Python tests were updated.
Google does not implement refresh-from-server, therefore using it has
to be made configurable. It is enabled by default for Funambol and
disabled for everything else.
Existing user configs must be updated to use refresh-from-server with
Funambol:
syncevolution --configure enableRefreshSync=1 funambol
Running a sync in "dry run" mode is not supported. The error
message about that was reported like an internal error.
Also added tests covering such output.
Due to the way how properties were handled internally, it wasn't
possible to explicitly set a property to its default value. Instead
the property was unset. For example, explicitly setting database= was
not possible.
This is necessary for client-test and ActiveSync, because client-test
needs to know that the testing is expected to run with the default
databases (something which normally is avoided by overwriting empty
database properties).
Now the "is set" state is tracked explicitly in the config storage and
command line property APIs. Unsetting a property via the command line
could be implemented with an explicit command line option, but is not
supported at the moment.
Tests were extended to cover the new functionality and adapted to the
change behavior for "type" migration: syncFormat was empty
already (because the empty string matched the default), but
forceSyncFormat was unnecessarily set explicitly. Now it is not.
Deleted C++ version, fixed D-Bus version: must ignore errors
about Akonadi not running. Those happen if (and only if) Akonadi
support is enabled and Akonadi wasn't started manually before
running test-dbus.py.
KWallet support was broken: syncevo-dbus-server checked
KDE_FULL_SESSION to determine whether it should use KWallet instead of
GNOME Keyring. That did not work, because the env variable was not set
for D-Bus daemons.
Automatically detecting KDE users is not possible at the
moment. Instead KDE users have to manually set the new "keyring"
global config property to "KDE" (case insensitive) if the
SyncEvolution installation supports both, because GNOME Keyring is the
default to avoid surprises for traditional users. If only KWallet
support is enabled, then this is not necessary.
"GNOME" and "true/false/1/0/yes/no" can also be set. This has the
advantage that keyring usage can be enabled permanently for the
command line in --daemon=no mode; normally keyrings are not used in
that mode because accessing them can bring up UI dialogs.
It also becomes possible to disable keyring usage in syncevo-dbus-server,
something which couldn't be done before.
The --keyring command line option is still supported, as an alias for
"[--sync-property] keyring=<value>". The default value for --keyring
is true, to match the traditional behavior. In contrast to other sync
properties, setting "keyring" does not require an explicit --run
parameter. Again this is done to mirror traditional usage.
Reading a password also (unintentionally) checked all supported
storages while searching for the password. Now it uses exactly
one storage and falls back to asking for the password directly.
The commit itself also cleans up the code a bit (reformatted, fixed
comments). Choosing the right slot in the password signals is done via
a new InitStateTri parameter which contains the "keyring" setting.
Error checking (unsupported keyring string, --keyring=yes and no
keyring enabled) is done in additional slots which run after all the
regular ones.
Parameter parsing for --sync and --keyring were unified. However,
there is the difference that --keyring has an implicit default value
("yes") and never has an additional parameter, in contrast to --sync,
which always is followed by one.
The new CmdlineTest::testKeyring covers different ways of using
--keyring. It relies on actually invoking keyring backends, something
not done by the default SyncContext UI. Therefore
CmdlineSyncClient+KeyringSyncCmdline were moved into libsyncevolution,
to be used by CmdlineTest.
The new "notifyLevel" per-peer configuration option allows users to
control how many desktop notifications the D-Bus server produces while
executing an automatic sync:
0 - suppress all notifications
1 - show only errors
2 - show information about changes and errors (in practice currently the same as level 3)
3 - show all notifications, including starting a sync (default)
As in the CmdlineTest, all tests now look for methods in DBusUtil
first, which allows overriding unittest.TestCase methods in our
own code. Done for the sake of consistency (no immediate need).
One of the goals of the fork/exec rewrite is to let a sync complete
even when syncevo-dbus-server dies. The new TestLocalSync.testNoParent
covers that; to pass it, the server had to be changed so that its
signal delivery to the D-Bus server became optional (= no exception
thrown when it fails).
Use the new getChildren() to check for children, instead of relying on
ps. Wait for syncevo-dbus-server, to avoid race conditions between
signal() and waitpid(). Check status and error message in the sync
report written by syncevo-dbus-helper.
Instead of killall (which may have the side effect of killing
unrelated process instances and the drawback of only covering known
processes), find and kill all child processes. The information about
child processes is taken from parsing /proc.
Some child process are special:
- The forked test-dbus.py must be kept running.
- dbus-monitor is meant to be running longer and still gets killed
via the subprocess handle.
Python unittest has a size limit for the diffs that it prints. Our
output is already large and must be directed to logs, so allow diffs
of unlimited size.
Normal print statements are not very useful because they are
out-of-band (not tied to either the D-Bus log, the SyncEvolution
output or the failure message of a failed test).
The new logging.log() (for simple string) and logging.printf() (for
string formatting with arbitrary arguments) get the output included in
the .dbus.log of a test by sending out a signal on a dummy
object. Search for =log in the .dbus.log:
signal sender=:1.1194 -> dest=(null destination) serial=35 path=/test/dbus/py; interface=t.d.p; member=log
string "killing syncevo-dbus-server with pid 7851"
Introduced a specific assertNoErrors() for stderr output. A bit nicer
to read in the tests and allows filtering of the error output (was
necessarily temporarily to ignore an error that occured when valgrind
didn't handle shutdown via signal properly).
TestLocalSync.testParentFailure covers the situation that
syncevo-dbus-server is killed while syncevo-dbus-helper and
syncevo-local-sync are waiting for a password. With
syncevo-dbus-server gone, they'll never get that information and need
to abort syncing.
When running syncevo-dbus-server under valgrindcheck.sh, both the
wrapper script and the actual syncevo-dbus-server must be killed,
starting with the wrapper script because otherwise it'll initiate a
shutdown when it sees the death of the syncevo-dbus-server.
The test-dbus.py infrastructure needs to be improved for this:
- allow removing DBusUtil.pserver during a test
- collect zombies after running a test
- kill syncevo-dbus-helper and syncevo-local-sync
Instead of killing a known list of processes, it might be better to
kill all children of test-dbus.py.
Binary data cannot be sent via D-Bus as string, because D-Bus will
balk on non-UTF8 sequences of bytes. GIO D-Bus failed particularly
spectacularly, with internal asserts followed by a segfault.
This affected sending the initial message as part of SyncParams and
thus all testing with syncevo-http-server. D-Bus testing itself
didn't have a Connection test involving WBXML.
Fixed by using SharedBuffer inside SyncParams with D-Bus traits based
on DBusArray. DBusArray itself is less suitable because it cannot be
copied easily inside the app. Also added a test, with a b64 encoded
binary message inside the Python script.
Running the fork/exec implementation under valgrind caused
some tests to fail because a) some tests ran longer (fixed
by increasing timeouts) and b) some tests resulted in
different D-Bus communication depending on the timing.
Added more debug logging in the syncevo-dbus-server Connection
class and the syncevo-dbus-helper DBusTransport to track this
down.
There were multiple reasons, usually related to handling aborted
connections.
The D-Bus API explicitly says about the "Abort" signal sent by the
server: "This signal is sent at most once for each connection. No
reply will be sent on an aborted connection." The old code did send an
empty, final reply after aborting and the test-dbus.py actually
checked for it. Now that final message is really only send when the
connection is still waiting for a reply (state == PROCESSING) and
hasn't been aborted. The test was fixed accordingly.
The "Abort" documentation also says that "all further operations on it
[= Connection] will fail". Despite that comment one D-Bus test did a
Connection.Close() after receiving the Abort signal. The server now
destroys the Connection instance once it has failed and thus the
Close() call failed. It was removed.
The Connection class now consistently uses delayed deletion, instead
of destructing itself while some of its methods are still active. A
bit safer.
While thinking about the server<->helper communication I noticed that
a Connection.Close() succeeds even if the helper hasn't shut down
yet. Not sure whether there are relevant error scenarios where we
need to tell the client that shutdown of the helper failed.
This commit moves the blocking syncing, database restore and command
line execution into a separate, short-lived process executing the
syncevo-dbus-helper. The advantage is that the main
syncevo-dbus-server remains responsive under all circumstances (fully
asynchronous now) and suffers less from memory leaks and/or crashes
during a sync.
The core idea behind the new architecture is that Session remains the
D-Bus facing side of a session. It continues to run inside
syncevo-dbus-server and uses the syncevo-dbus-helper transparently via
a custom D-Bus interface between the two processes. State changes of
the helper are mirrored in the server.
Later the helper might also be used multiple times in a Session. For
example, anything related to loading backends should be moved into the
helper (currently the "is config usable" check still runs in the
syncevo-dbus-server and needs to load/initialize backends). The
startup code of the helper already handles that (see boolean result of
operation callback), but it is not used yet in practice.
At the moment, only the helper provides a D-Bus API. It sends out
signals when it needs information from the server. The server watches
those and replies when ready. The helper monitors the connection to
the parent and detects that it won't get an answer if that connection
goes down.
The problem of "helper died unexpectedly" is also handled, by not
returning a D-Bus method reply until the requested operation is
completed (different from the way how the public D-Bus API is
defined!).
The Connection class continues to use such a Session, as before. It's
now fully asynchronous and exchanges messages with the helper via the
Session class.
Inside syncevo-dbus-server, boost::signals2 and the dbus-callbacks
infrastructure for asynchronous methods execution are used heavily
now. The glib event loop is entered exactly once and only left to shut
down.
Inside syncevo-dbus-helper, the event loop is entered only as
needed. Password requests sent from syncevo-local-sync to
syncevo-dbus-helper are handled asynchronously inside the event loop
driven by the local transport.
syncevo-dbus-helper and syncevo-local-sync are conceptually very
similar. Should investigate whether a single executable can serve both
functions.
The AutoSyncManager was completely rewritten. The data structure is a
lot simpler now (basically just a cache of transient information about
a sync config and the relevant config properties that define auto
syncing). The main work happens inside the schedule() call, which
verifies whether a session can run and, if not possible for some
reasons, ensures that it gets invoked again when that blocker is
gone (timeout over, server idle, etc.). The new code also uses
signals/slots instead of explicit coupling between the different
classes.
All code still lives inside the src/dbus/server directory. This
simplifies checking differences in partly modified files like
dbus-sync.cpp. A future commit will move the helper files.
The syslog logger code is referenced by the server, but never used.
This functionality needs further thought:
- Make usage depend on command line option? Beware that test-dbus.py
looks for the "ready to run" output and thus startup breaks when
all output goes to syslog instead of stdout.
- Redirect glib messages into syslog (done by LogRedirect, disabled when
using LoggerSyslog)?
The syncevo-dbus-server now sends the final "Session.StatusChanged
done" signal immediately. The old implementation accidentally delayed
sending that for 100 seconds. The revised test-dbus.py checks for
more "session done" quit events to cover this fix.
Only user-visible messages should have the INFO level in any of the
helpers. Messages about starting and stopping processes are related to
implementation details and thus should only have DEBUG level.
The user also doesn't care about where the operation eventually
runs. All messages related to it should be in INFO/DEBUG/ERROR
messages without a process name. Therefore now syncevo-dbus-server
logs with a process name (also makes some explicit argv[0] logging
redundant; requires changes in test-dbus.py) and syncevo-dbus-helper
doesn't.
syncevo-local-sync is different from syncevo-dbus-helper: it produces
user-relevant output (the other half of the local sync). It's output
is carefully chosen so that the process name is something the user
understands (target context) and output can be clearly related to one
side or the other (for example, context names are included in the sync
table).
Output handling is based on the same idea as output handling in the
syncevo-dbus-server:
- Session registers itself as the top-most logger and sends
SyncEvolution logging via D-Bus to the parent, which re-sends
it with the right D-Bus object path as output of the session.
- Output redirection catches all other output and feeds it back
to the Session log handler, from where it goes via D-Bus
to the parent.
The advantage of this approach is that level information is made
available directly to the parent and that message boundaries are
preserved properly.
stderr and stdout are redirected into the parent and logged there as
error. Normally the child should not print anything. While it runs,
LogRedirect inside it will capture output and log it
internally. Anything reaching the parent thus must be from early
process startup or shutdown.
Almost all communication from syncevo-dbus-helper to
syncevo-dbus-server is purely information for the syncevo-dbus-server;
syncevo-dbus-helper doesn't care whether the signal can be
delivered. The only exception is the information request, which must
succeed.
Instead of catching exceptions everywhere, the optional signals are
declared as such in the EmitSignal template parameterization and
no longer throw exceptions when something goes wrong. They also don't
log anything, because that could lead to quite a lof of output.
KDE complains with "realPath called for relative path ..., please fix"
if the XDG env variables point to a relative path. Using absolute paths
makes sense because it allows processes to change their current directory.
Always unsetting SYNCEVOLUTION_DEBUG removed an important debugging
method: SYNCEVOLUTION_DEBUG is useful in cases where output
redirection and/or handling is broken.
Therefore this commit removes the forced unsetting of
SYNCEVOLUTION_DEBUG again and instead let's at least
TestCmdline.testConfigureTemplates pass with and without the
additional debug output.
Other tests fail when it is set, so the right way to invoke
test-dbus.py is without it.
The doSync() method does several steps which can take a long time
(for example, send SAN message) before finally entering the while
loop which checks for abort/suspend. In the case that a suspend or
abort request arrives early, doSync() should not complete the
sync setup. Now it checks more often and returns early.
This was found because the fork/exec sync changed the timing so that
the syncevo-dbus-helper ended up being killed by SIGTERM before it
even started syncing. That had the downside that the process didn't
clean up, resulting in many "potentially lost" memory chunks.
Better choose the timing so that doSync() really is ready to handle
the signal. Aborting at that time is also more realistic (normally
sync startup isn't that slow, only valgrind makes it slow).
When a test is stuck in a loop.run() and times out before the expected
event arrives, then the logged message wasn't very informative (just a
generic "timed out"). Now the error message includes a stack backtrace
(useful to find out where the test was waiting, and thus for what) and
the list of all events received so far (useful to determine which one
is missing).
Both old and new implementation only look at the start times
of syncs to determine whether the autoSyncInterval is over.
D-Bus test should do the same. It still passed because of
the inaccurate time comparison. Now the comparison is more
likely to succeed.
Specifying - for password in a local sync doesn't make sense: the
password is not needed. Apparently the
TestSessionAPIsDummy.testAutoSyncLocalSuccess test worked because the
password was never checked. There's no guarantee that this is always
the case and in fact, the fork/exec version of syncevo-dbus-server did
ask for the password.
Better fix the test so that it does not depend on passwords being
ignored for local sync.
Extended the test for "auto sync failed permanently" to also check
that the auto sync manager doesn't retry syncing. Probably
SyncEvolution 1.2.x fails this test (untested, unsure whether it had
code for it).
The TestLocalSync.testConcurrency test does not wait for the session
to really start. If the abort request is processed quickly, there
never is a sync report for the session, which is okay. Happened after
the server rewrite, because then "running" state is issued some time
before the helper really runs the sync and creates the report.
Making the report checking optional solves the issues.
The script incorrectly checked for "queuing" status instead
of "queueing" and thus did not wait correctly when the
session really took a while to become active.
Not sure how this worked with SyncEvolution 1.2. Acccording to the code,
this test should have failed because auto-termination was turned off
forever when there was an auto sync config.
Instead of scheduling a shutdown session, set a boolean flag and check
that when scheduling/accepting new sessions and connections. That way
it is a bit more obvious in which state the server is.
Preventing shutdown when there is an auto sync config at startup is
wrong. The auto sync flag might get removed later, in which case
shutting down the server becomes okay again. The auto sync manager
must take the shutdown lock as needed. Somehow that worked before,
despite the "m_autoSync.preventTerm() ? -1 : duration".
A test for this will be committed separately.
Instead of coupling different modules directly to each other via
method calls or callbacks, this commit introduces signals which are
connected to methods as necessary when combining the different modules
in the server.
When TEST_DBUS_GDB is set to a non-empty value, syncevo-dbus-server is
run under gdb. Previously it was necessary to edit test-dbus.py to
achieve that.
The old implementation queued session requests, but then ran its own
high-priority shutdown session instead as soon as possible. A client
then loose the connection to the server instead of getting its pending
session activated.
The fork/exec server rewrite initially allowed the session to run,
instead of shutting down. The goal is to reject session requests while
shutting down.
Watch status and progress events and check them, both with the
existing checkSync() method (does semantic sanity checks) and with a
regex match against the pretty-printed event log (checks the content,
like sync mode and sources).
DBusUtil.checkSync() now includes a pretty-printed dump of the status
and progress events whenever any of the assertions about the events
fails. This is done in a wrapper method because a) including a message
in the assertions themselves overwrites the assertion information when
using Python 2.6 and b) assing these messages would require changing
lots of lines.
The calls to statusChanged and progressChanges are fixed. They passed
all parameters as a single tuple. This affected
TestSessionAPIsReal.progressChanged(), which compared an integer
against a tuple and always got 'True' as result.
To make checkSync() more flexible, the number of expected reports is
now a parameter with default 1.
The path emitting a signal is included as 'path' keyword for the
callbacks - will be useful for command line testing, which otherwise
wouldn't know which session ran the command line.
The test uses a local sync config with just one address book. It
checks the right ordering of stdout/stderr output by merging both into
one stream. Checking D-Bus signals during the sync could (should?!)
be added.
If, and only if, SyncEvolution is compiled with module support,
then the error message will include information about the
loaded modules. Covered by checking the expected error with a regex
that has that additional information as an optional clause.
The syncevo-dbus-server, in contrast to the Cmdline C++ class, always
uses a keyring and then only returns "-" instead of the real password.
ScheduleWorldConfig() is no longer a global function.
Looking into config.h is one way of making the test optional. That'll
work for the nightly testing. A cleaner approach might be to look at
the output of backend=?.
The Python createFiles() split at all occurences of a colon, instead
of just at the first one. That threw away all property values in the
vCard test data of testItemOperations.
The "if True" was used for debugging the fallback code. The check for
availability was broken, it needs to look at unittest.TestCase. Now
the code uses the fallback only if necessary, as originally intended.
The old ~/.sync4j config must be created in the simulated home
directory and $HOME of syncevo-dbus-server/syncevolution must be set
to that directory. The latter cannot be done unconditionally (some
tests may depend on a functional EDS), so a new "own_home" flag
was added.
If we pass False to runCmdline's expectSuccess then the test should
fail, because we expected failure. Previously only checking if
application succeeded when expected was done.
Patrick Ohly
patrick.ohly@intel.com
Chris Kühl
Krzesimir Nowak
Krzesimir Nowak