One of the features of the Synthesis engine is that it can preserve
properties locally which are not supported by the
peer. "testExtensions" covers that, by updating all items on the
server (via client B) and reimporting them as updates into client A.
The data comparison then is done without the usual "allow server to
drop data" simplifications in synccompare.
This test is not enabled yet. "config.update" needs to be set for it,
but doing so then trips over other changes introduced by servers, like
for example re-encoding photos. Needs some more thoughts and testing...
When the glib event loop is left because the D-Bus client has
requested an abort, the LocalTransportAgent should simply return a
"failed" status and let the caller handle the abort. The return code
of write/readMessage() must be able to convey that - extended from
boolean to an enum.
SyncContext did not do that correctly in server mode: the check for
abort must be done before giving up by throwing an exception.
The D-Bus test now checks that the right status is recorded (wasn't
the case earlier).
Local sync inside syncevo-dbus-server must keep the main event
loop running, otherwise the D-Bus server would stop responding
to D-Bus calls while a sync runs.
This is tested as part of new test-dbus.py tests. They set up local
sync between two directories and then test syncing, timeout detection
when using the glib support code, and aborting via the D-Bus API
while a sync runs.
This testing depends on delaying the sync inside the child
process. Setting the SYNCEVOLUTION_LOCAL_CHILD_DELAY env variable
achieves that (might also be useful for interactive debugging). It is
set for those tests which need it via additional function properties,
similar to the older @timeout() mechanism, which now uses the
more general @property function decorator.
This test verifies that timeout detection works for SyncML/HTTP and
local sync. The test does not depend on the source which is active,
so to cut down redundant testing it registers itself only for
the first source in CLIENT_TEST_SOURCES.
The implementation of the test creates a dead listening socket which
the HTTP transports can connect to without ever getting a reply. Then
the syncURL for outgoing HTTP is redirected towards that dead port.
The expectation is that the overall sync times out roughly after 10
seconds.
Instead of allowing users of the API to register a callback
which can choose between aborting and time out, only accept
the timeout duration and always treat that as a timeout.
The advanced functionality wasn't used and the simplification
makes implementing the API easier.
The test failed only because the test's own introduction text on
stdout ended up being redirected. Flushing it before running the
test avoids that.
The flushing was added to both the client runner (because some other
tests also accidentally redirected the output, and it should be
made visible right away) and to the "glib" test (because it shouldn't
depend on flushing in the runner).
The FN property was missing from the vCard 3.0 template item. As a
result, some servers created it, which showed up as a comparison
failure. Added FN back and, for good measure, use a new method for
making them unique the same way: replace <<UNIQUE>> with the sequence
number.
The meaning of "type" was horribly complex and had effects on the
backend and the peer. It was impossible to specify the sync format to
be used for a specific peer independently of the local backend and its
format, so adding a peer to a context broke the context configuration
(BMC #1023).
This is now fixed by splitting "type" into four independent properties:
- backend = plugin which interfaces with the data
- databaseFormat = data format used inside backend, only relevant for file backend
- syncFormat = data format preferred when talking to peer
- forceSyncFormat = disable format auto-negotiation, use preferred format
With that split, it is now possible to specify the format in which the
file backend stores items independently of the format in which they
are exchanged with the peer.
Old configurations with "type" can still be read. The values specified
inside it are transparently mapped to the new properties. Command line
and D-Bus API users will only see the new properties.
The command line tool still accepts "type" as an alias for the four new
properties. Using that has the same disadvantage as before: it will modify
the context even if only modifying the peer was intended.
The D-Bus API accepts only the new properties. Clients using "type"
must be adapted to the new property names. Clients not using that
continue to run unchanged.
Writing into the configuration requires a migration of the peer config
*and* the context in which it is defined. That is necessary because
the new semantic (independent database format) cannot be stored in the
old format. The migration is handled by rewriting first the context,
then all peers defined inside it.
Other user-visible changes:
- updated help texts
- the canonical "backend" value for the file backend is just "file"
instead of the long "Files in one directory", which is now an alias
(used to be the other way around); done because "type = file"
was expanded to the long name, which was a bit unexpected and showed
how unintuitive the long name is
Internal changes:
- getMimeVersion() is still present, although it hasn't been used
for a long time; FileSyncSource::getMimeVersion() now derives
the version from the supported Mime types, in case that the
function will be needed again in the future
- setSourceType() with string as argument was replaced with one
taking a SourceType instance; to emulate the old behavior if
desired, construct SourceType from an old-style string
- ConfigProperty methods need to be virtual so that derived classes
like SourceBackendConfigProperty can generate content at runtime
(a recent commit broke that feature)
- File templates were stripped down to the essential properties,
with "type" replaced by the per-peer "syncFormat". "type" would
still have been accepted (so it is not necessary to adapt
syncevo-phone-config right away), but has the original
disadvantage of modifying "backend" and "databaseFormat".
The LogRedirect instance created in main() was not flushed often
enough because the LoggerStdout instance pushed into the stack after
it intercepted all output, so the LogRedirect instance was never
invoked during a test.
This patch solves that by replacing the LoggerStdout with a
LogRedirect instance. This instance is then called by the LogDir
logger.
First, dump information about relevant valgrind parameters
if valgrindcheck.sh is part of the command line. Now
executing the logged command acts like the original invocation.
Second, include the current directory as a "cd" command, to
simplify re-execution of the command.
Previously test data for insert/copy/update tests was the same
as the template for creating items. But it was used without replacing
<<REVISION>> or REVISION, which confused some servers (for example,
Memotoo and memos).
This patch cleans this up such that the normal test data is simpler
and only the template items have a revision marker. Also changed
the templates so that the revision marker is <<REVISION>> across
the board (including memos) and can occur more than once, which is
used to get unique contact names.
Renaming "evolutionsource/user/password" to "database", "databaseUser",
"databasePassword" affects D-Bus API clients which depend on the old
names. No known clients use these properties, so making the change
is acceptable, but some caution is needed nevertheless.
This patch adds a validation step to SetConfig() that detects unknown
properties and top-level entries. "" for sync properties and
"source/<name>" for source properties are currently accepted,
everything else is rejected.
Property values are also checked in advance, compared to checking them
later as part of copying them. The exception raised now for invalid
values is a more specific org.syncevolution.InvalidCall.
While adding the validation step it was noticed that the description
of temporary settings in SetConfig() was incorrect: temporary changes
*do* affect the result of GetConfig(). Documentation fixed.
Temporary settings can be reset now with SetConfig(update=False,
temporary=True), something which wasn't allowed earlier.
The previous commit wasn't complete, some events still used UTC time
where MemoToo converts them to local time.
Also RECURRENCE-ID and UID are not properly supported, so replace the
exception with one which has a separate UID and not RECURRENCE-ID.
What MemoToo does is:
- add EXDATE to recurring master event
- strip UID
- keep RECURRENCE-ID, although it is now redundant
With this patch Client::Sync::ical20::testItem passes for MemoToo.
MemoToo converts UTC time into the local time configured for a user
when the event recurrs. That transformation is flawed, but let's
ignore it in the nightly testing by using test data which was
transformed the same way.
Note that the test account must be configured for the German time
zone, otherwise test data and MemoToo still won't match.
SyncEvolution was changed recently to not duplicate the SUMMARY as
DESCRIPTION and also removes this redundant information when storing.
The test cases which had that redundant DESCRIPTION needed to be
updated, because synccompare is not smart enough to hide this
difference.
The code which was meant to raise the 404 error used the wrong
exception name and thus produced a Python exception instead, which
translated into an internal server error. Raising an exception
was not the right way to produce a 404 reply.
Replaced with a combination of setResponseCode() and writing a fixed
HTML response. Purely cosmetical, did not have an effect in practice.
The documentation of Twisted's Request.notifyFinish() doesn't say so
explicitly, but errors are passed to the Errback, not the normal
callback. Adding self.done() as both of these allows the server to
detect premature client disconnects.
That error handling code itself was not working properly: instead of
keeping the connection to syncevo-dbus-server open in case that the
client resends the POST, it close the D-Bus connection. With this
patch it is kept open.
The new SuspendProxy tests covered this failure.
Including the Content-Length header allows clients to detect
incomplete replies. Without this fix the new SuspendProxy test failed
because the proxy returned a header, then truncated the connection
without sending any data. libsoup treated that as an empty reply,
which resulted in a parser error in libsmltk and aborted the sync,
without resending the POST.
Compared to the existing Resend tests, the new ResendProxy tests have
the advantage that they cover one additional error case: loss of
connection before the server has a chance to send the full reply. This
happened to be one case that the syncevo-http-server wasn't handling
properly.
This failure cannot be simulated inside client-test because the
TransportAgent API doesn't provide enough control. The implementation
uses a twisted HTTP proxy which intercepts data transfer at three
different points (before sending, after sending and before receiving,
after receiving) and drops the client connection. The script listens
on four different ports, numbered consecutively, with the first one
not introducing errors, and all others representing one failure.
The test must set and unset proxy information in the underlying transport;
this patch makes that possible in SoupTransportAgent::setProxy().
To use the new tests, run "proxy.py <port>", then set
CLIENT_TEST_RESEND_PROXY=<port>.
Instantiating gobject.MainLoop had two negative side effects:
- shutdown after CTRL-C got stuck
- SIGCHLD after syncevo-dbus-server shutdown also caused a deadlock
Not sure why the "loop = gobject.MainLoop()" was in the code;
it doesn't seem to serve any purpose anymore.
--start-syncevolution
sets up the right environment for syncevo-dbus-server
and (re)starts it explicitly, instead of depending on
D-Bus auto-activation; to be used when SyncEvolution
is not installed at the location it was compiled for
--syncevolution-path=PATH
sets the installation path (the directory which
contains 'bin', 'libexec', etc.) to be used in
--start-syncevolution, default is the location where
syncevo-http-server itself is installed
Starting syncevo-dbus-server in a synchronous manner similar to the
traditional D-Bus get_object() is tricky. It is currently implemented
as a subprocess spawn, followed by a loop which checks for the
well-known name to appear on the session bus.
Open issues:
- What if syncevo-dbus-server was already running?
- The code is meant to restart syncevo-dbus-server on demand, but that
doesn't work. Once the spawned process terminates, twisted seems to get
stuck handling the SIGCHLD signal for a process it doesn't know anything about.
--start-dbus-session creates a new D-Bus session for communication with
syncevo-dbus-server and (inside that server) with
other D-Bus services like Evolution Data Server,
removes it when shutting down; should only be used if
it is guaranteed that the current user will not have
another session running, because these services can
get confused when started multiple times; without this
option, syncevo-http-server a) uses the session
specified in the environment or b) looks for a session
of the current user (depends on ConsoleKit and might
not always work)
--server-certificate=CERT
certificate file used by the server to identify itself
(required for https)
--server-key=KEY key file used by the server to identify itself
(optional, certificate file is used as fallback, which
then must contain key and certificate)
Example keys for localhost and a README for creating/using them are
included in this commit.
Instead of logging a
org.freedesktop.DBus.Error.UnknownMethod: Method "Close" with signature "bs" on interface "org.syncevolution.Connection" doesn't exist
error, catch this error and only print it at debug level.
This is caused by syncevo-dbus-server removing the connection
right away after sending the Abort signal, so when Close() is
called on it, the instance no longer exists. The D-Bus error
is a bit misleading.
Not calling Close() would be the alternative, but determining
in advance when Close() is not necessary is hard, so better try in
all cases.
Exceptions from syncevo-dbus-server which will also appear as error
message from that server need to be logged at debug level, to avoid
showing internal details like stack backtrace to users.
Also, the observer must *replace* the default logging instead of
adding itself. Done with startLoggingWithObserver().
The observer replaces the default logger. As a special case
Output produced by syncevo-dbus-server is now shown as part of the
regular syncevo-http-server. It is thus no longer necessary to watch
the output of syncevo-dbus-server.
Added new command line options:
-d, --debug enables debug messages
-q, --quiet limits output to real error messages; ignored if
--debug is given
--log-config=LOGCONFIG
configure logging via Python logging config file;
--debug and --quiet override the log level in the root
logger
An example config file is included in the distribution.
The output is now similar to SyncEvolution command line output:
[DEBUG] twisted: twisted.web.server.Site starting on 9000
The implementation is based on the standard Python logging module.
Twisted output is redirected into it at log levels DEBUG and ERROR.
Conflicts:
configure-pre.in
src/syncevo/SyncContext.cpp
src/syncevo/configs/datatypes/10calendar-fieldlist.xml
test/ClientTest.cpp
test/client-test-main.cpp
test/synccompare.pl
Conflicts all resolved by using the code from "master" branch. Caused
by similar changes on release branch.
When a sync completes, CLIENT_TEST_LOG=<path> copies everything it
finds at that path into "server-log" inside the sync's log
directory. This is useful when testing against a local server with its
own logs.
To use it with SyncEvolution, configure logdir=/tmp/server-log in the
server's configurations for the test clients and then set
CLIENT_TEST_LOG=/tmp/server-log.
Already a while back we added conversion of alarm times from relative
to UTC for Mobical.com. Later we used the same mechanism for Nokia
phones (BMC #1657).
Originally this was meant to work only for one-time events, for which
it doesn't matter how the alarm time is specified. For recurring
events, a fixed alarm time would only fire once, at least when
interpreted strictly. Experiments show that at least Nokia phones (and
thus perhaps also Mobical.com) interpret a fixed alarm as "repeat
alarm with the same relative offset as on first occurrence".
Therefore it makes sense to apply that same transformation when
parsing vCalendar 1.0 events into the internal representation (and
thus from there into iCalendar 2.0). With this patch, a full
SyncEvolution->phone->SyncEvolution sync is possible without changing
the relative alarm times. Tested with several test cases (one time
alarm, different offsets, before and after event), see
test/testcases/ical20-alarms-2010-12-31.ics.
One minor data loss remains: an alarm relative to the end time of an
event will become an alarm relative to the start of the event, because
there is no way to reconstruct that difference. Not relevant in practice,
IMHO.
Already a while back we added conversion of alarm times from relative
to UTC for Mobical.com. Later we used the same mechanism for Nokia
phones (BMC #1657).
Originally this was meant to work only for one-time events, for which
it doesn't matter how the alarm time is specified. For recurring
events, a fixed alarm time would only fire once, at least when
interpreted strictly. Experiments show that at least Nokia phones (and
thus perhaps also Mobical.com) interpret a fixed alarm as "repeat
alarm with the same relative offset as on first occurrence".
Therefore it makes sense to apply that same transformation when
parsing vCalendar 1.0 events into the internal representation (and
thus from there into iCalendar 2.0). With this patch, a full
SyncEvolution->phone->SyncEvolution sync is possible without changing
the relative alarm times. Tested with several test cases (one time
alarm, different offsets, before and after event), see
test/testcases/ical20-alarms-2010-12-31.ics.
One minor data loss remains: an alarm relative to the end time of an
event will become an alarm relative to the start of the event, because
there is no way to reconstruct that difference. Not relevant in practice,
IMHO.
Always remove RELATED=START, not only when talking to Memotoo.
It is a generally useful normalization.
Also affects the literal comparison of Google Calendar ALARM
removal.
Due to the known issue of not recording the final sync mode (BMC #2786),
the "delete item count" incremented by SyncEvolution for refresh-from-peer
is not happening when running as server. Keep the tests passing
by disabling the result checks when CLIENT_TEST_NOCHECK_SYNCMODE is set.
Due to a typo, testManyDeletes ran one sync with 0 buffer size. This
corner case has to be handled by transport agents, but testing it
shouldn't be part of testManyDeletes, so fixed the typo.
CLIENT_TEST_MODE=server enabled some tweaks for testing syncing with a
phone, but it did not go far enough. Tests like
Client::Sync::testRefreshFromServerSemantic were written under the
assumption that the peer is a SyncML server, and thus either used the
wrong refresh direction or expected the wrong result.
Instead of renaming all tests and the "client-test" program itself,
this patch merely changes the data direction in the affected tests
when running with CLIENT_TEST_MODE=server. In that mode, the meaning
of "server" and "client" are thus reverted. Potentially confusing :-/
The sequence number added by createItem() ended up in the UID
of iCalendar 2.0 data. While that isn't strictly forbidden,
it is at least confusing. This patch changes the space into a hyphen.
Google sometimes returns an additional VALARM for an event. There's no
specific pattern to it, sometimes it is included, sometimes not. This
patch allows synccompare to ignore this alarm if the
CLIENT_TEST_GOOGLE_VALARM env variable is set.
client-test ran without redirection stderr, which produces a lot of
output in the console in particular with the mKCal backend. This patch enables
stderr redirection (as usual, only if SYNCEVOLUTION_DEBUG is not set).
All normal output now goes to stdout/std::cout.
Like others, README.qtcontacts tries to summarize limitations
of qtcontacts. Furthermore, it also include issues of interoperability
testing with Google.
To enable it in configuration, use '--enable-buteo-tests'.
To test client-test with buteo to do sync, make sure the
environment variable 'CLIENT_TEST_BUTEO=1' is set.
Three main steps to support buteo testing in client-test,
1) pre-run: storage preparation, clean up buteo logging
files and set corresponding keys in meego-sync-conf.xml.
For qtcontacts, switch backend database files for each
client; For calendar/todo/note, specify different notebook
names for each client.
2) run: re-implement ClientTest.doSync function, which
dbus-calls 'msyncd', the dbus daemon of buteo to do sync.
Listen to needed signals to track its status.
3) post-run: collect its status and summarize its statistics
and convert to internal sync report.
Buteo doesn't support explicit slow-sync mode and refresh-
from-server mode. Add two workarounds for them:
For slow-sync mode, wipe out anchors;
For refresh-from-server, wipe out anchors and clean all local
data and do two-way sync.
To enable buteo testing in your environment, you have to:
1) Add read/write privilege for the directory /etc/sync for
current testing user
2) To test one remote server, make sure the profile xml is put
in $HOME/.sync/profiles/sync. The name of the profile xml should
be set as the server name.
