This is the result of 2to3, with the interpreter path updated manually,
tabs replaced by spaces and some obsolete packages removed.
Required because the current nightly build host no longer has Python2 installed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
twisted.web.error.NoResource was replaced by
twisted.web.resource.NoResource. This has become a real problem for
example on Fedora 22 where the old name is no longer available.
Sometimes time stamps are necessary to understand timing-dependent
issues. A separate flag for timestamps on/off might be useful.
For now, enable them in debug mode.
Under heavy load while testing, syncing occasionally failed with a D-Bus
message timeout, for example in the Close() call. Let's disable these timeouts
by explicitly passing something very large (infinite is not supported by the
Python bindings).
If HTTP processing takes to long, the client may resend, but in contrast to a
failed D-Bus method call that case is handled.
The increased timeout has no negative effect on detecting when
syncevo-dbus-server dies, because then the D-Bus daemon will notify
us immediately.
If a client gave up waiting for the server's response and resent its message
while the server was still processing the message, syncing failed with
"protocol error: already processing a message" raised by the
syncevo-dbus-server because it wasn't prepared to handle that situation.
The right place to handle this is inside the syncevo-http-server, because it
depends on the protocol (HTTP in this case) whether resending is valid or
not. It handles that now by tracking the message that is currently in
processing and matching it against each new message. If it matches, the new
request replaces the obsolete one without sending the message again to
syncevo-dbus-server. When syncevo-dbus-server replies to the old message, the
reply is used to finish the newer request.
This situation is covered by
Client::Sync::eds_event_eds_contact::testManyDeletes with 100 items and
client and server running under valgrind. The test failed earlier and works
now.
Python failed to call logSyncEvoOutput() after adding the additional
'process' parameter to LogOutput because it extracts all four
parameters and then cannot pass them to logSyncEvoOutput().
Now logSyncEvoOutput() uses the new process information to instantiate
a logger with the right prefix, using 'sync' as fallback for messages
without that information (as before).
Added debug output and double-check D-Bus signal watch
handling. Explicitly remove watches. Not sure whether
they remained active earlier.
This was done as an attempt to track down a "connection: cancelled"
error in the HTTP client during stress testing. The real reason
turned out to be a deadlock inside syncevo-dbus-server caused by
libsynthesis, which prevented serving the new connection.
The shutdown of that connection remains mysterious. It seems
that Twisted itself cancelled the request.
So far, the file pointed to by --certificate-file had to
contain the server certificated (signed by a CA known to the client)
and (optionally) a client certificate.
With this patch, the file may also contain additional intermediate
certificates which will be sent to the client (chained certificates).
See http://twistedmatrix.com/pipermail/twisted-python/2010-July/022597.html
The code which was meant to raise the 404 error used the wrong
exception name and thus produced a Python exception instead, which
translated into an internal server error. Raising an exception
was not the right way to produce a 404 reply.
Replaced with a combination of setResponseCode() and writing a fixed
HTML response. Purely cosmetical, did not have an effect in practice.
The documentation of Twisted's Request.notifyFinish() doesn't say so
explicitly, but errors are passed to the Errback, not the normal
callback. Adding self.done() as both of these allows the server to
detect premature client disconnects.
That error handling code itself was not working properly: instead of
keeping the connection to syncevo-dbus-server open in case that the
client resends the POST, it close the D-Bus connection. With this
patch it is kept open.
The new SuspendProxy tests covered this failure.
Including the Content-Length header allows clients to detect
incomplete replies. Without this fix the new SuspendProxy test failed
because the proxy returned a header, then truncated the connection
without sending any data. libsoup treated that as an empty reply,
which resulted in a parser error in libsmltk and aborted the sync,
without resending the POST.
Instantiating gobject.MainLoop had two negative side effects:
- shutdown after CTRL-C got stuck
- SIGCHLD after syncevo-dbus-server shutdown also caused a deadlock
Not sure why the "loop = gobject.MainLoop()" was in the code;
it doesn't seem to serve any purpose anymore.
--start-syncevolution
sets up the right environment for syncevo-dbus-server
and (re)starts it explicitly, instead of depending on
D-Bus auto-activation; to be used when SyncEvolution
is not installed at the location it was compiled for
--syncevolution-path=PATH
sets the installation path (the directory which
contains 'bin', 'libexec', etc.) to be used in
--start-syncevolution, default is the location where
syncevo-http-server itself is installed
Starting syncevo-dbus-server in a synchronous manner similar to the
traditional D-Bus get_object() is tricky. It is currently implemented
as a subprocess spawn, followed by a loop which checks for the
well-known name to appear on the session bus.
Open issues:
- What if syncevo-dbus-server was already running?
- The code is meant to restart syncevo-dbus-server on demand, but that
doesn't work. Once the spawned process terminates, twisted seems to get
stuck handling the SIGCHLD signal for a process it doesn't know anything about.
--start-dbus-session creates a new D-Bus session for communication with
syncevo-dbus-server and (inside that server) with
other D-Bus services like Evolution Data Server,
removes it when shutting down; should only be used if
it is guaranteed that the current user will not have
another session running, because these services can
get confused when started multiple times; without this
option, syncevo-http-server a) uses the session
specified in the environment or b) looks for a session
of the current user (depends on ConsoleKit and might
not always work)
--server-certificate=CERT
certificate file used by the server to identify itself
(required for https)
--server-key=KEY key file used by the server to identify itself
(optional, certificate file is used as fallback, which
then must contain key and certificate)
Example keys for localhost and a README for creating/using them are
included in this commit.
Instead of logging a
org.freedesktop.DBus.Error.UnknownMethod: Method "Close" with signature "bs" on interface "org.syncevolution.Connection" doesn't exist
error, catch this error and only print it at debug level.
This is caused by syncevo-dbus-server removing the connection
right away after sending the Abort signal, so when Close() is
called on it, the instance no longer exists. The D-Bus error
is a bit misleading.
Not calling Close() would be the alternative, but determining
in advance when Close() is not necessary is hard, so better try in
all cases.
Exceptions from syncevo-dbus-server which will also appear as error
message from that server need to be logged at debug level, to avoid
showing internal details like stack backtrace to users.
Also, the observer must *replace* the default logging instead of
adding itself. Done with startLoggingWithObserver().
The observer replaces the default logger. As a special case
Output produced by syncevo-dbus-server is now shown as part of the
regular syncevo-http-server. It is thus no longer necessary to watch
the output of syncevo-dbus-server.
Added new command line options:
-d, --debug enables debug messages
-q, --quiet limits output to real error messages; ignored if
--debug is given
--log-config=LOGCONFIG
configure logging via Python logging config file;
--debug and --quiet override the log level in the root
logger
An example config file is included in the distribution.
The output is now similar to SyncEvolution command line output:
[DEBUG] twisted: twisted.web.server.Site starting on 9000
The implementation is based on the standard Python logging module.
Twisted output is redirected into it at log levels DEBUG and ERROR.
Because people asked for it, the Twisted-based HTTP server is
now installed when D-Bus support (which it depends on) is enabled.
It is installed as "syncevo-http-server" without the .py suffix
because that is an implementation detail that might change.
Because people need a way to figure out how to start it, usage
information is provided via optparse.
When a client posts exactly the same data again (identified via
session ID and message content) because it didn't get a reply and we
have a reply, we can send back that reply immediately. This is not
tied to actually having that session around, because the server might
have closed and thus removed that session already (the "last reply in
session lost" problem).
Only one reply is buffered, which would be a problem if we supported
multiple parallel sessions, which syncevo-dbus-server doesn't.
This patch also adds some more debug prints and removes an obsolete
TODO comment about identifying clients which start a new session while
there is an active one for them. That is already handled inside
syncevo-dbus-server.
When the syncevo-dbus-server receives a SyncML message as initial
data from a transport stub, it peeks into the data with the help
of the Synthesis server engine and our SyncEvolution_Session_CheckDevice()
and extracts the LocURI = device ID of the client. This value is
guaranteed to be in the initial SyncML message.
Then it searches for a peer configuration (still called "server
configuration" in the current code) which has the same ID in its
remoteDeviceID property and uses that configuration.
Instantiating SyncContext and Synthesis engine multiple times
is a bit tricky. The context for the SynthesisDBPlugin currently
has to be passed via a global variable, SyncContext::m_activeContext.
SyncContext::analyzeSyncMLMessage() temporarily replaces that
with the help of a new sentinal class SwapContext, which ensures
that the previous context is restored when leaving the function.
Some common code was moved around for this (SyncContext::initEngine()).
The "default config" parameter in syncevo-http-server.py was
removed because it is no longer needed. The possibility to
select some kind of config context via the path below the
sync URL still exists. This is passed to syncevo-dbus-server via
the "config" peer attribute. It has no effect there at the
moment.
TestConnection.testStartSync() in test-dbus.py covers this kind of
config selection. To run it, a peer configuration with remoteDeviceID
"sc-api-nat" must exist.
The "must authenticate" parameter for Server.Connect() had no
effect so far. Netherless, HTTP clients should have been required
to authenticate from the beginning.
This uses the new combined client/server Synthesis engine. When
building shared modules, the engine is opened dynamically only
while needed, thus reducing overall memory consumption.
The HTTP server is implemented in Python, using the the 'twisted'
server framework because it can use the same glib event loop as the
D-Bus binding. This allows us to keep the same event loop running
forever and react to both kinds of events.
The server takes a base url (including host name and port)
and a default configuration as name. The host name is currently
ignored. It could be used to bind to a specific interface.
The path is what a client has to use as part of his sync URL
to synchronize against the default configuration. In addition
the client can add another path to select that as his server
configuration.
For example, if the script is called with
http://localhost:9000/syncevolution default
then syncURL = http://localhost:9000/syncevolution will synchronize
against the configuration called "default". With syncURL =
http://localhost:9000/syncevolution/my_server, it will
synchronize against "my_server".
