This patch makes the configuration layout with per-source and per-peer
properties the default for new configurations. Migrating old
configurations is not implemented. The command line has not
been updated at all (MB #8048). The D-Bus API is fairly complete,
only listing sessions independently of a peer is missing (MB #8049).
The key concept of this patch is that a pseudo-node implemented by
MultiplexConfigNode provides a view on all user-visible or hidden
properties. Based on the property name, it looks up the property
definition, picks one of the underlying nodes based on the property
visibility and sharing attributes, then reads and writes the property
via that node. Clearing properties is not needed and not implemented,
because of the uncertain semantic (really remove shared properties?!).
The "sync" property must be available both in the per-source config
(to pick a backend independently of a specific peer) and in the
per-peer configuration (to select a specific data format). This is
solved by making the property special (SHARED_AND_UNSHARED flag) and
then writing it into two nodes. Reading is done from the more specific
per-peer node, with the other node acting as fallback.
The MultiplexConfigNode has to implement the FilterConfigNode API
because it is used as one by the code which sets passwords in the
filter. For this to work, the base FilterConfigNode implementation must
use virtual method calls.
The TestDBusSessionConfig.testUpdateConfigError checks that the error
generated for an incorrect "sync" property contains the path of the
config.ini file. The meaning of the error message in this case is that
the wrong value is *for* that file, not that the property is already
wrong *in* the file, but that's okay.
The MultiplexConfigNode::getName() can only return a fixed name. To
satisfy the test and because it is the right choice at the moment for
all properties which might trigger such an error, it now is configured
so that it returns the most specific path of the non-shared
properties.
"syncevolution --print-config" shows errors that are in files. Wrong
command line parameters are rejected with a message that refers to the
command line parameter ("--source-property sync=foo").
A future enhancement would be to make the name depend on the
property (MB#8037).
Because an empty string is now a valid configuration name (referencing
the source properties without the per-peer properties) several checks
for such empty strings were removed. The corresponding tests were
updated resp. removed. Instead of talking about "server not found",
the more neutral name "configuration" is used. The new
TestMultipleConfigs.testSharing() covers the semantic of sharing
properties between multiple configs.
Access to non-existant nodes is routed into the new
DevNullConfigNode. It always returns an empty string when reading and
throws an error when trying to write into it. Unintentionally writing
into a config.ini file therefore became harder, compared with the
previous instantiation of SyncContext() with empty config name.
The parsing of incoming messages uses a SyncContext which is bound to
a VolatileConfigNode. This allows reading and writing of properties
without any risk of touching files on disk.
The patch which introduced the new config nodes was not complete yet
with regards to the new layout. Removing nodes and trees used the
wrong root path: getRootPath() refers to the most specific peer
config, m_root to the part without the peer path. SyncConfig must
distinguish between a view with peer-specific properties and one
without, which is done by setting the m_peerPath only if a peer was
selected. Copying properties must know whether writing per-specific
properties ("unshared") is wanted, because trying to do it for a view
without those properties would trigger the DevNullConfigNode
exception.
SyncConfig::removeSyncSource() removes source properties both in the
shared part of the config and in *all* peers. This is used by
Session.SetConfig() for the case that the caller is a) setting instead
of updating the config and b) not providing any properties for the
source. This is clearly a risky operation which should not be done
when there are other peers which still use the source. We might have a
problem in our D-Bus API definition for "removing a peer
configuration" (MB #8059) because it always has an effect on other
peers.
The property registries were initialized implicitly before. With the
recent changes it happened that SyncContext was initialized to analyze
a SyncML message without initializing the registry, which caused
getRemoteDevID() to use a property where the hidden flag had not been
set yet.
Moving all of these additional flags into the property constructors is
awkward (which is why they are in the getRegistry() methods), so this
was fixed by initializing the properties in the SyncConfig
constructors by asking for the registries. Because there is no way to
access them except via the registry and SyncConfig instances (*), this
should ensure that the properties are valid when used.
(*) Exception are some properties which are declared publicly to have access
to their name. Nobody's perfect...
Different tests may need different files. setupFiles() now takes
a parameter that selects a subdirectory of $srcdir/test/test-dbus
for copying.
setupFiles() is meant to be used in tests with own_xdg=True. Using
it differently is an error, resulting in a test failure. Creating
the "test-dbus" directory is not necessary.
Test Session.GetReports by reference files. The idea
is to GetReports from the reference sessions files
so that we could know the expected results. It could
do extractly comparsions.
To achieve the goal,
1) create 5 directories of sessions files
2) copy reference files from 'test/test-dbus/cache'
to current xdg cache directory 'test-dbus/cache'.
3) calling 'GetReports' and compare returned results
with expected value.
This unit test covers below cases of GetReports:
1) start=0, index=0
2) start=0, index=1
3) start=0, index=0xFFFFFFFF
4) start=2, index=0xFFFFFFFF
5) start=5, index=0xFFFFFFFF
The case 4) will cause an integer overflow issue.
Thus nothing is returned, which is not expected.
The fix for this will be in the next commit.
Remove all reduntant 'cleanAllConfig' in TestSessionAPIsDummy.
The reason is that each unit test of this class
will create a new xdg directory before running.
The function 'setupFiles' is used to copy reference directory
trees from 'test/test-dbus' to 'xdg_root(./test-dbus)'. It
won't do copy unless own_xdg is enabled. Otherwise, user's
directories might be polluted.
This function is for those unit tests which are
hard(or impossbile) to do comparisons. To solve this
problem, we can create reference files in 'test/test-dbus'
and copy them to 'xdg_root' and compare expected results
with values returned from DBus server.
Currently 4 sub-directories are copied, but
maybe with different names, mappings are:
test/test-dbus ./test-dbus
sync4j .sync4j
data data
config config
cache cache
Since now TestSessionAPIsDummy is not only for config,
but also for other session APIs, change the server name
in this class from 'dummy-test-for-config-purpose' to
'dummy-test'. The server name hasn't conflicts because
new 'own_xdg/config' directory is created for unit tests
of this class.
The function was defined, but not part of the server implementation at
all. The order of the returned sessions is now defined in the
specification.
Note that the spec allows for *multiple* running sessions although
currently we only can have at most one.
All sessions started in m_syncStatus = SYNC_IDLE state. The
transition from m_active = FALSE (= queueing) to m_active = TRUE
(= real "idle") was not announced via a "StatusChanged" signal.
Added a new SYNC_QUEUEING state. m_syncStatus starts in that state
and changes to idle together with sending the signal when the session
gets activated. I kept the m_active/setActive() part because it is
somewhat orthogonal to the state transitions and m_active is easier
to check than different states.
testSyncSecondSession is similar to testSecondSession: it creates two
sessions and checks that the second one becomes ready once the first
one is done. But instead of detaching from the first session after a
certain delay (as testSecondSession) does, the first session runs
a real sync, which deactivates it when the sync is done.
The DBusUtil.setUpSession() method was refactored so that it can be
used to create multiple sessions.
Remove signal handler in createSession() because callers do not expect
events getting record by the function, should the signal fire later on.
However, if the caller explicitly asks for it by indicating that it
does the waiting itself, then keep the handler in place.
The signal handlers were registered with the session path as object
path to match against, but the org.syncevolution.Server signals
are all sent with "org.syncevolution.Server" as path.
The signal handlers must match against the expected session path in
their own code.
The testSecondSession test only worked because the timeout fired
when the second session was indeed ready. Use the default timeout
which will correctly report the test as failed and fixed waiting
for second session becoming ready.
Registered Python D-Bus signal handlers cannot be removed.
This had the undesired effect that when running multiple tests,
old signal handlers where still active and might have undesired
effects, like calling loop.quit() or recording events multiple
times (because both the old and new handler are invoked).
To solve this problem, DBusUtil.runTest() sets self.running = True
while the test is active. All signal handlers must check this
flag and not do anything if it is False.
Debugging such problems was harder than necessary because recording
the "quit_events" inside DBusUtil instances had the effect that the
records of previous signal handlers where not visible in the ensuing,
failing test.
Therefore this patch also turns the DBusUtil instance variables into
class variables which are shared by the tests. As a side effect, it is
no longer necessary to call DBusUtil.__init__. Instead the variables
are reset in runTest().
Calling arbitrary functions sometime in the future is implemented via
SIGALRM and code which multiplexes that single signal among several
different timouts. The signal is always set so that it triggers in
time for the closest timeout, kept in a heap.
The timeouts are sometimes used to call loop.quit(). There is no guarantee
that the process really is in loop.run() when SIGALRM fires. Therefore
timeouts are installed with glib.timeout_add() by default and if available.
With this change, the testSecondSession test works without glib Python
binding.
In addition, because the new alarm functionality works outside and
inside loop.quit(), it can be used to implement timeouts for
tests. These timeouts are mandatory and implemented by
DBusUtil.runTest(). The default is 5 seconds. It can be overridden
via a @timeout(seconds) function decorator which sets the non-default
timeout in the "timeout" attribute of the test function.
Getting to that function instance in runTest() is a bit tricky. It is
done by looking up the TestCase.id() in the global name space, minus
the "__main__." part which would prevent finding the function.
When running all tests I noticed that the test runner showed
the two testSync methods with their method and class name. Not sure
whether this was because of the missing docstring or the duplication
of the method name; renamed the methods and added a description to
the test instead of the test suite.
The TestSessionAPIsReal.testSync() had failed with a description
that pointed towards the syncevo-dbus-server getting killed prematurely.
In an attempt to track this down I added a check of self.quit_events,
but then the test succeeded.
Add 4 unit tests for Session.GetReports:
1) Test the error is reported when the server name is empty
2) Test when the given server does not exist
3) Test when the given server has no reports. Also covers
boundaries
4) Test when the given server exists and reports are
returned correctly. Also covers boundaries
For unit test 4), we change TestDBusSync to
'TestSessionAPIsReal'.
The new class is used to test those unit tests of
session APIs, depending on doing sync. Thus we
need a real server configuration to confirm sync
could be run successfully.
Add 4 unit tests for Session.GetDatabases:
1) Test the error is reported when the server name is empty
2) Test the error is reported when the given server does
not exist
3) Test nothing is returned when the given server exists
but it has no given source
4) Test when the given server exists and a correct source.
All work correctly.
Change 'TestDBusSessionConfig' to 'TestSessionAPIsDummy'
to let this class also cover other Session APIs.
Add 5 unit tests for Session.CheckSource:
1) Test the error is reported when the server name is empty
2) Test the error is reported when the given server does
not exist
3) Test the error is reported when the given server exists
but it has no given source
4) Test the error is reported when the given server exists
but the given source with an invalid evolutionsource
5) Test when the given server exists and a correct source.
Then all work correctly.
For detail, please see test cases design.
Add this class is to test APIs that the correct
errors are reported when the server name is empty.
Also, add 2 unit tests to cover this case for
GetConfig/SetConfig when the server name is empty.
The extra "mustAuthenticate" parameter to the Session::sync() method
is replaced with temporarily setting username/password to empty
via the Session::m_syncFilter.
To catch undesired prototype changes of D-Bus API calls (as it
happened when adding that parameter), all these methods now come with
a comment that links them to the corresponding D-Bus API call.
test-dbus.py tests cover the different credential checks:
- TestConnection.testStartSync: invalid MD5 creds, no authorization
necessary => client accepted
- TestConnection.testCredentialsWrong: invalid MD5 creds, authorization
necessary => client rejected
- TestConnection.testCredentialsRight: correct basic creds, authorization
necessary => client accepted
To be clear and easily understood, a new
TestDBusSessionConfig class is to cover unit
tests of configuration.
8 situations are tested and all are passed:
1) parameters are invalid
2) create a new config with 'clear' mode
3) update temporarily, not flush to files
4) update specific properties
5) update properties with invalid value
6) update properties for a non-existing server
7) clear and empty a config
8) clear specific sources
The Server.GetConfig() is stricly read-only. When looking at the
error message for NoSuchConfig I noticed that it creates the config
from a template with the name if the config doesn't exist yet. This
was discussed before and should have been removed, but this instance
must have slipped in.
Removed and adapted test for error message accordingly.
Reading a non-existant configuration should result in
org.syncevolution.NoSuchConfig (MB#6548).
Running a sync without a valid configuration should
result in a bad status code.
Both tests passes.
The server now checks whether it has other sessions active or in the
queue with the same device ID and kills the older sessions when adding
a new one. Because it does this on each Process(), it is unlikely
(impossible?!) to to have more than one older session, but because the
session might be running or still inactive in the queue, both are
checked.
As part of testing with test-dbus.py it was observed that a connection
that was to be aborted was also closed normally first. That was because
the sync engine did not set an error status in this case. Better check
for abort requests explicitly before calling agent->shutdown(). However,
this should be fixed as part of the server progress handling (MB#7709).
This patch fixes signal handling and shutdown of syncevo-dbus-server
when signals were received. This problems were found in combination
with automated tests via test-dbus.py.
Signals handled by the signal handlers in the core had no effect on
syncevo-dbus-server, which called its own event loop once the
suspended or aborted sync returned. Signals received while outside
of a sync and outside of an event loop also had no effect.
Now signals are always caught by the niam() function in
syncevo-dbus-server. It uses the new SyncContext::handleSignal() API
to tell the core engine about it and in addition, remembers that
syncevo-dbus-server is meant to shut down (suspendRequested). This is
then checked by the syncevo-dbus-server event loop (DBusServer::run())
before blocking again.
The implementation of Session.Abort() and Session.Suspend() didn't
work. isSuspend() and isAbort() mixed up the state. They also did not
wake up DBusTransportAgent, so the syncevo-dbus-server might get stuck
although an abort was already requested. Partially fixed by adding
g_main_loop_quit(). It's not entirely sure whether g_main_loop_quit()
is reentrant (see below). There is other code which uses it in signal
handlers, but that doesn't mean that this is correct. The right long
term solution would be:
- avoid entering and leaving the event loop
- feed signals into the event loop as normal events
SoupTransportAgent solves this by polling for a signal once per second.
This also should be improved.
Because such code was used in several places for a connection, that common
code was moved into Connection::wakeupSession().
Things which went in unnoticed when the original signal handling was merged:
- SyncContext::getSuspendFlags() should not allow the caller to modify
the SyncContext state. Made the returned SuspendFlags const.
- A signal handler may only call re-entrant functions (Stevens, "Advanced
Programming in a Unix Environment", 10.6).
All printing therefore has to be done outside of the signal handlers.
Now the signal handlers just set a message and the regular code
checking for abort/suspend prints it. There's a slight race condition
(a message might get overwritten before it is printed), but printing
just the last message might actually make more sense. There might be
a slight delay between receiving the signal and printing now.
When the syncevo-dbus-server receives a SyncML message as initial
data from a transport stub, it peeks into the data with the help
of the Synthesis server engine and our SyncEvolution_Session_CheckDevice()
and extracts the LocURI = device ID of the client. This value is
guaranteed to be in the initial SyncML message.
Then it searches for a peer configuration (still called "server
configuration" in the current code) which has the same ID in its
remoteDeviceID property and uses that configuration.
Instantiating SyncContext and Synthesis engine multiple times
is a bit tricky. The context for the SynthesisDBPlugin currently
has to be passed via a global variable, SyncContext::m_activeContext.
SyncContext::analyzeSyncMLMessage() temporarily replaces that
with the help of a new sentinal class SwapContext, which ensures
that the previous context is restored when leaving the function.
Some common code was moved around for this (SyncContext::initEngine()).
The "default config" parameter in syncevo-http-server.py was
removed because it is no longer needed. The possibility to
select some kind of config context via the path below the
sync URL still exists. This is passed to syncevo-dbus-server via
the "config" peer attribute. It has no effect there at the
moment.
TestConnection.testStartSync() in test-dbus.py covers this kind of
config selection. To run it, a peer configuration with remoteDeviceID
"sc-api-nat" must exist.
The new TestConnection suite covers some Connection API related
aspects:
- creating a connection and closing it
- sending an invalid message
- sending a valid message
Right now these tests assume that a "syncevolution_server" config
exists. They depend on getting one Abort signal per connection
to shut down each test.
The signal recording code from TestDBusSync was moved into DBusUtil
so that the different tests can share this utility code.
Some sleeps are necessary after sync sessions because SIGTERM might
get caught without properly shutting down synevo-dbus-server (MB#7555).
First, kill all running syncevo-dbus-server instances. Having
one running was one way how the tests could have failed because
it wasn't running with the right environment.
Second, use files for output instead of in-memory pipes. The latter
might have caused deadlocks when a process writes large amount of
data.
syncevo.log and dbus.log in the current directory are used. The
additional benefit is that these files can be read while the test
runs, for example when it is deadlocked for some other reason.
Third, when syncevo-dbus-server returns with a non-zero error code,
give dbus-monitor time to settle down before stopping it. Otherwise
the final D-Bus traffic might not get recorded.
Running the script failed on Debian Lenny failed because Python
and python-gobject are too old. Python 2.6 was required because
of subprocess.terminate(), replaced with os.kill(). python-gobject
2.16 was needed for glib, which is used in some tests to run
the event loop for a short while. These tests now fail if glib
is not available, while the rest of the tests can be run normally.
The environment with XDG_ redirected into a local directory
also needs to be passed to gdb when debugging interactively.
Otherwise the tests would not run as without gdb.
The common code is now in a DBusUtil class. Different tests suites
use it differently via their setUp() and run() methods:
- TestDBusServer: read-only access to server with private config dirs
- TestDBusSession: access to active session with private config dirs
- TestDBusSync: run a real sync, using the normal config dirs
Currently TestDBusSync is hard-coded to use "scheduleworld_1" as
config. This assumes that the user of the script has set up
such a config for "client-test".
With a patched dbus-monitor which properly catches SIGTERM
the output can be added to a failed test. The time stamps
from --profile would be useful, but for debugging the normal
mode is more useful.
A special run() method ensures that the syncevo-dbus-server
is started for every single test. For this to work, the server
must print a single line right before it is ready to start.
The run() method checks the result of the D-Bus server and adds
its output to test failures. It also checks the return code, so
if syncevo-dbus-server ran under valgrind (not currently possible),
valgrind failures would show up.
The run() method also runs "dbus-monitor" in an attempt to log the
traffic on the D-Bus, but because dbus-monitor does not flush its
output upon receiving SIGINT, the output is currently lost. Pretty
useless...
It is possible to start syncevo-dbus-server under a debugger before
running the test. For this set "debugger = 'gdb'" at the top of
the file, then in the gdb prompt set breakpoints etc. before running
the server. It has to be stopped manually, too.
Such runtime options should be selected via the command line, which
is not possible yet. The existing tests are just a proof of concept.
The really interesting ones (actually checking the content of
configs or templates) still have to be written.
Patrick Ohly
Zhu, Yongsheng