Created a separate config for the suspend/resume tests in
edsfile, because:
- they are slow and need a larger timeout
- selecting them separately easily is useful
- running too many tests inside the same process under
valgrind still causes problems ("fork: out of memory")
Setting the env variable twice failed in combination with
sudo+schroot as prefix. The value given first on the initial
command line ended up being used by the command. Apparently
one of the involved tools doesn't allow overwriting of
values and keeps the first one it sees.
Fixed by only addding CLIENT_TEST_ALARM to the command
line once, with the final value.
Running additional commands from client-test becomes more
expensive when client-test runs under valgrind with tracing
of children (as needed for local sync testing). It also create
additional output (at least a valgrind header+footer per command).
Avoid calling additional commands as much as possible. Specifically:
- fork/exec synccompare in the simple case (avoids one /bin/sh call)
- avoid setting PATH via env before searching for synccompare (expected
to be in the PATH when client-test is called)
- use that simple case for all servers (appending the synccompare
output again at the end of a test log via CLIENT_TEST_COMPARE_LOG
isn't that useful and was missing from some servers anyway)
- copy files with some C code instead of invoking cp or cat
- do the translation of logs to html in runtests.py
The main goal is to test CalDAV/CardDAV sources as part
of a SyncML client and/or server. A test involving syncevo-http-server
is now named "<client storage><server storage>":
- edsfile = EDS in client, file in server (used to be syncevohttp)
- davfile = CalDAV/CardDAV in client, file in server (new)
- edsdav = EDS in client, CalDAV/CardDAV in server (new)
For this, WebDAVSourceRegister.cpp must be able to create test sources
which match the client 1/2 sync configs. The client "1" or "2" strings
are passed through the abstract ClientTest into the source A/B create
callbacks. WebDAVSourceRegister.cpp cannot get this directly from
ClientTest because it lives in a plugin which is not necessarily
linked against ClientTest.
A conceptual change is that CLIENT_TEST_EVOLUTION_PREFIX/USER/PASSWORD
no longer override existing config properties. That is necessary
because the shared prefix is too simplistic for WebDAV (needs full URL
in "database"); also helps KDE (needs resource URI). The env variables
and the default "SyncEvolution_Test_" value for the database prefix are
still used if the config does not exist. That is useful to prevent
accidentally running client-test against the default databases.
The nightly setup script might (should!?) be made public to simplify
configuring a server.
Another change is the user-configurable part of client-test now lives
entirely in the _1/_2 client sync configs and contexts. From there the
source properties are copied into the Client::Source context each time
client-test runs.
Restarting a sync and sync logic in general is different when
multiple sources are active. Therefore let's also test the eds_contact +
eds_event combination when talking to our own SyncEvolution HTTP
server.
The Client::Sync::*::testTwoWayRestart is the first of several tests
which runs a sync, make changes to the local data after the each cycle
has completed, and requests the sync to continue. Adding, updating and
deleting items are covered with sync sessions which all consist of two
cycles.
The test checks the final sync mode (same as original cycle), all
intermediate reports (captured before each startDataRead slot, at
which time all results from the previous sync are guaranteed to be
recorded) and total number of cycles.
The tests for the other sync modes follow the same pattern and use
the same code.
refresh-from-remote is problematic. Should it try to wipe out all
items added after the initial refresh-from-remote? The
one-way-from-remote sync that happens in later cycles doesn't do
that. The test currently expects that the item doesn't get deleted
and thus reflects the current implementation.
For refresh-from-remote only adding can be tested, because any further
syncs remove that item as it never reaches the server and will be
deleted locally.
Client::Sync::*::testManyCycles makes different changes
(add/modify/remove different numbers of items) before each cycle,
leading to a long sync with 13 cycles altogether.
CLIENT_TEST_PEER_CAN_RESTART must be set if and only if the peer can
restart a sync. When set, the full set of restart tests is enabled and
expected to restart. Otherwise, only testTwoWayRestart is tested and
expected to not do a restart despite the request to do so.
Furthermore, when acting as server no restart tests are done because
that doesn't work in server mode.
Older boost::lambda (as on Ubunty Hardy) has problems with
binds which involve references to classes with pure virtual methods,
like SyncSource. Later Boost releases fixed that:
http://lists.boost.org/boost-users/2006/03/18053.php
As a workaround for older Boost a pointer to SyncSource is
used in boost::lambda::bind.
When compiling with a combination of sudo + schroot to enter the
compile environment, then the "env SYNTHESISSRC=..." prefix got
lost. Setting the variable directly is handled, so use that.
The testDelete404 fails because AkonadiSyncSource::removeItem() doesn't
detect attempts to delete a non-existent item. That is only relevant for
SyncEvolution's virtual calendar+todo storage (used when syncing with
Nokia phones).
All tests involving more advanced test data fail because the test data
was written for EDS. Akonadi uses a different vCard flavor (different
extensions, for example), so different test data has to be created.
--trace-children was still set to "no" in the nightly testing.
Setting it to "yes" showed some deficits in valgrindcheck.sh:
- having multiple processes write into the same log file
garbles the file; now valgrindcheck.sh uses %p and checks
all resulting files one after the other
- a child process might have to be checked (for example,
to check also its child processes) without us actually
caring for the result; valgrindcheck.sh now can filter
out the common potential memory leaks if VALGRIND_LEAK_CHECK_SKIP
containts a *regex* matching the comman name
More control over which errors are ignored might be useful.
Perhaps suppression rules are a better solution (match by
executable with wild cards for stack and repeat for all errors).
The HTTP server and syncevo-dbus-server are started by two nested wrappercheck.sh
invocations. It would have been possible to use --start-syncevolution, but by
starting syncevo-dbus-server directly it is easier to check it with valgrind.
Both valgrindcheck.sh and wrappercheck.sh now support redirecting the output
of the command resp. daemon that they start, without having to also redirect
the output of the scripts themselves. This is used to redirect the output
of both syncevo-dbus-server and syncevo-http-server into syncevohttp.log,
from where client-test copies it into the *.server.log of the test.
wrappercheck.sh now also sends SIGTERM in addition to SIGINT.
Needed by Twisted on Debian Testing.
activesyncd gets compiled completely (which should always succeed) and
then only the required pieces are installed (without the things which
are hard-coded for /usr, because installing those will fail).
Testing is done by running a similar set of client-test tests as for remove
CalDAV/CardDAV servers.
activesyncd is started anew each time client-test is run, done in
the new wrappercheck.sh script. Can be combined with valgrindcheck.sh:
wrappercheck.sh valgrindcheck.sh activesyncd -- valgrindcheck.sh client-test ...
The return code of wrappercheck.sh is the return code of the real command
or the daemon, if the real command succeeded. This way the special 100 return
code from valgrindcheck.sh is returned if the only problems were memory
issues.
Run all three commands in a single invocation, concatenated by &&.
That way it all commands are shown even if they don't all get
executed - useful for fixing problems.
The test prefix needs to be passed through test-dbus.py to the
invocation of syncevo-dbus-server. Done with the new
TEST_DBUS_PREFIX env variable: set by runtests.py, used by test-dbus.py.
Also cleaned up the delay loop for starting syncevo-dbus-server:
the logic for reloading the log file was overly conservative. The file
only needs to be re-read when the file size has changed since the
last check (in contrast to the previous behavior or checkings always
if different from before the loop).
Merge branches are removed locally before fetching from the remote repo.
Ensures that obsolete branched which where removed from the remote repo
also no longer exist locally. By itself, "git fetch" doesn't do that.
Automatic integration ignores branches which were already merged into
the upstream remote branch. Avoids redundant listing of branches
in the integration report which didn't really need to be merged for the
test run.
Unstaged and staged changes ("stash") are included in the patch report.
The patches are sorted from oldest to most recent. Fixed the fallback
code when a patch has no Subject.
The scripts for reporting tests results are taken from the merged
source code. That ensures that the reporting matches the tests
that were run. The versions from the boot-strapping SyncEvolution
code are the fallback, just in case that automatic integration
fails.
(cherry picked from commit 33bbf5df57)
Conflicts:
test/generate-html.xsl
In contrast to the master branch, URLs are still absolute because
backporting those changes would have been more difficult.
Some paths changed as part of the "automatic integrate" patch.
Instead of recalculating the path to the source code, use the
sync.basedir member.
(cherry picked from commit 4c5a27ce66)
Conflicts:
test/runtests.py
"one way" sync also not supported for "from server". Was not skipped
for the new one-way-from-remote sync mode yet.
Also fixed entries for eds_event; due to cut-and-paste error it still
said eds_contact.
Merge branches are removed locally before fetching from the remote repo.
Ensures that obsolete branched which where removed from the remote repo
also no longer exist locally. By itself, "git fetch" doesn't do that.
Automatic integration ignores branches which were already merged into
the upstream remote branch. Avoids redundant listing of branches
in the integration report which didn't really need to be merged for the
test run.
Unstaged and staged changes ("stash") are included in the patch report.
The patches are sorted from oldest to most recent. Fixed the fallback
code when a patch has no Subject.
The scripts for reporting tests results are taken from the merged
source code. That ensures that the reporting matches the tests
that were run. The versions from the boot-strapping SyncEvolution
code are the fallback, just in case that automatic integration
fails.
The URLs in the on-disk HTML are all relative. That allows rsyncing
the results directories without having to translate hrefs. The version
sent via mail still has absolute external URLs (the original motivation
for embedding the full URL).
The HTML version of the .log files links to ClientTest.cpp.html,
colorizes important parts and links to the sync session
directories.
Because it is much more useful than the plain text version, the
nightly.html now links to these .html files.
ClientTest.cpp.html is built with Python pygments if installed,
otherwise some builtin fallback code is used.
Numbering Client::Source::LinkedItems_xxx with xxx being a number is
confusing, in particular when the same number stands for different
test data. Now each set of linked items has an additional, unique name
which is used for Client::Source::LinkedItems<Name>.
Done in combination with adding more linked item tests and slightly
reorganizing the logic for adding them:
- a default set with VTIMEZONE is added in all cases
- some SyncML servers override that default set
- others, in particular peers accessed via their own backend,
enable additional Client::Source tests on a case-by-case basis
Exchange is only tested with its own default set (with "Standard
Timezone" as TZID) and the all-day recurring set (as before).
All other CalDAV servers are now also tested with the all-day
set (previously exclusive to Exchange) and local floating time (= no
TZID, new).
Google CalDAV can't be tested with local time because it converts such
events into the time zone of the current user. All-day events need
special test data because Google adds a time to the UNTIL clause
(http://code.google.com/p/google-caldav-issues/issues/detail?id=63).
synccompare also needs to ignore that Google adds a redundant VTIMEZONE
to the all-day test cases.
Finally, Client::Source tests for updating a child event (with and
without parent) without UID and RECURRENCE-ID inside the payload were
added. These properties are removed via text operations. The
expectation is that the source is able to add them back (if needed)
based on the meta information that it has about the existing item.
The file source is unable to do that. When using it in an HTTP server,
the server will look to peers like a peer which doesn't support the
semantic (which indeed it doesn't) and thus the client will add back
the fields.
Branches committed to the libsynthesis or SyncEvolution git repo with
a name of "for-<branch>/<name>" will be automatically merged into
<branch> before starting the testing:
--sourcedir <dir with existing repos> --workdir <dir with merged repos>
The result can be reused as-is in further runtests.py invocation by using
--sourcedir <merged repos> --no-sourcedir-copy
Any local, *uncommitted* changes will be preserved and applied to
the merged sources (git stash). *Committed* changes will *not* be
applied. Always push them immediately.
Because there is a risk of loosing work when these rules are not
followed, the current state of the tree is stored as a branch created
with "git stash create" under the names
<branch>-nightly-before-<date+time> and <branch>-nightly. The latter
gets overwritten during each run.
These branches can be used to cherry-pick commits or restore the tree
as it was before the automatic merging ("git stash branch <branchname>
<branch>-nightly").
Result reporting includes information about the resulting code
revision. This includes the names of all branches which were
available for merging and whether that succeeded or failed. The output
of "git describe" is also included, always with a hash (even when it
matches a tag, in case that the tag gets moved) and whether the
working tree was dirty. All commits applied to the base branch are
listed in "git am" format.
This is done because the merged code is not pushed. This would clutter
the upstream code repository.
The "for-" prefix (instead of just "<branch>/") was added because git
got confused when <branch> was both a real branch and a directory of
other branches.
(cherry picked from commit 3e9fc042dc)
Branches committed to the libsynthesis or SyncEvolution git repo with
a name of "for-<branch>/<name>" will be automatically merged into
<branch> before starting the testing:
--sourcedir <dir with existing repos> --workdir <dir with merged repos>
The result can be reused as-is in further runtests.py invocation by using
--sourcedir <merged repos> --no-sourcedir-copy
Any local, *uncommitted* changes will be preserved and applied to
the merged sources (git stash). *Committed* changes will *not* be
applied. Always push them immediately.
Because there is a risk of loosing work when these rules are not
followed, the current state of the tree is stored as a branch created
with "git stash create" under the names
<branch>-nightly-before-<date+time> and <branch>-nightly. The latter
gets overwritten during each run.
These branches can be used to cherry-pick commits or restore the tree
as it was before the automatic merging ("git stash branch <branchname>
<branch>-nightly").
Result reporting includes information about the resulting code
revision. This includes the names of all branches which were
available for merging and whether that succeeded or failed. The output
of "git describe" is also included, always with a hash (even when it
matches a tag, in case that the tag gets moved) and whether the
working tree was dirty. All commits applied to the base branch are
listed in "git am" format.
This is done because the merged code is not pushed. This would clutter
the upstream code repository.
The "for-" prefix (instead of just "<branch>/") was added because git
got confused when <branch> was both a real branch and a directory of
other branches.
All-day event syncing only works with vCalendar 1.0 if client
and server use the same timezone. That's because all-day events
have to be represented as 00:00-23:59 events.
Use Europe/Berlin for the Mobical test account. That matches the current
server settings.
"oracle" is used as server name for Oracle. Oracle Beehive also
still exists, but seems less relevant now.
"egroupware-dav" is used for eGroupware when using CalDAV/CardDAV,
to distinguish this from the older eGroupware SyncML testing.
Conflicts:
configure.ac
test/ClientTest.cpp
test/testcases/eds_event.ics.funambol.tem.patch
Conflicts because of version number and updated test cases resp. local
delete optimization.
ActiveSync backend had to be adapted to modified InsertItemResult: now
it requests a merge when it detects duplicates, like the CalDAV backend
already did on the 1.2 branch.
Somehow a PUT which removes the detached recurrence from a meeting has
no effect. Looks like a Google server bug. Keep testing, but ignore
the failure.
The "src" directory no longer contains a Makefile. Therefore "testclean"
and building of client-test (now requested explicitly, instead of using
the obsolete "test" target) have to be done in the top-level directory.
Also stop referencing the global "compile" instance and instead use
the "build" parameter in SyncEvolutionTest.
Memotoo seems to have some problem at the SyncML
level ("[SyncEvolution] Memotoo refresh + empty data"). Keep the
failing tests running for contacts so that we can debug this.
For some strange reason, the ckeck for x86 was broken and can't have
worked like that. Running dpkg-architecture as make argument just
doesn't make sense.
Client::Sync::*::testLinkedItems* cannot be run when testing multiple
sources at once because only event sources have the test data. Limit
testing to single sources.
Memotoo and Mobical need special linked item test case data because
they modify time stamps.
Memotoo refresh semantic seems to have changed. See "[SyncEvolution]
Memotoo refresh + empty data".
The name of the combined sources in _SKIP env variables were wrong.
This removes the change introduced with 2d0da5. Running the command
twice like that does not work when "sudo schroot" is involved because
it is only applied to the first invocation. The workaround for schroot
problems also no longer seems to be necessary.
The parameter for reusing prebuilt binaries points into the build
directory. Now we optionally use installed binaries if found in <build
dir>/../install.
Result checking and running commands must know where to find the
backends. Previously this was handled inconsistently and incorrectly.
Commands with parameters containing single quotes were also quoted
with single quotes, which breaks. Check this and use double quotes
instead in this case. Still fails for parameters with a mixture of
quotes. Does not occur at the moment, though, so the simple quoting is
sufficient.
The valgrindcheck arguments weren't copied from runtest.py's
environment into the constructed command because valgrindcheck.sh is
present with path and thus only a substring of the command as string,
but not a word by itself (and thus not in the word array).
Also added the new VALGRIND_LEAK_CHECK_ONLY_FIRST.
LD_LIBRARY_PATH can be set in the environment *of runtest.py* and
will be forwarded to client-test/syncevolution as part of the
"env LD_LIBRARY_PATH=..." part of the command invocation.
This has to be done like this because sudo+schroot do not pass
through LD_LIBRARY_PATH and runtest.py itself needs to set
it for SyncEvolution libs, which must not overwrite the per-target
setting.
When SyncEvolution forks for local syncing, it doesn't clean
up after the forked process. This caused lots of reports
from valgrind which showed up in the nightly test results.
Now valgrindcheck.sh is used to automatically ignore them.
