Commit graph

125 commits

Author SHA1 Message Date
Patrick Ohly
86cc5f5785 testing: fixes for automatic integration
Some paths changed as part of the "automatic integrate" patch.
Instead of recalculating the path to the source code, use the
sync.basedir member.
(cherry picked from commit 4c5a27ce66)

Conflicts:

	test/runtests.py
2011-11-18 15:15:36 +00:00
Patrick Ohly
81c6d37e1d testing: automatically integrate branches (BMC #735)
Branches committed to the libsynthesis or SyncEvolution git repo with
a name of "for-<branch>/<name>" will be automatically merged into
<branch> before starting the testing:
   --sourcedir <dir with existing repos> --workdir <dir with merged repos>

The result can be reused as-is in further runtests.py invocation by using
   --sourcedir <merged repos> --no-sourcedir-copy

Any local, *uncommitted* changes will be preserved and applied to
the merged sources (git stash). *Committed* changes will *not* be
applied. Always push them immediately.

Because there is a risk of loosing work when these rules are not
followed, the current state of the tree is stored as a branch created
with "git stash create" under the names
<branch>-nightly-before-<date+time> and <branch>-nightly. The latter
gets overwritten during each run.

These branches can be used to cherry-pick commits or restore the tree
as it was before the automatic merging ("git stash branch <branchname>
<branch>-nightly").

Result reporting includes information about the resulting code
revision.  This includes the names of all branches which were
available for merging and whether that succeeded or failed. The output
of "git describe" is also included, always with a hash (even when it
matches a tag, in case that the tag gets moved) and whether the
working tree was dirty. All commits applied to the base branch are
listed in "git am" format.

This is done because the merged code is not pushed. This would clutter
the upstream code repository.

The "for-" prefix (instead of just "<branch>/") was added because git
got confused when <branch> was both a real branch and a directory of
other branches.
(cherry picked from commit 3e9fc042dc)
2011-10-28 09:01:18 +02:00
Patrick Ohly
5bfe6b9a5c Mobical testing: ensure that test account uses the right timezone
All-day event syncing only works with vCalendar 1.0 if client
and server use the same timezone. That's because all-day events
have to be represented as 00:00-23:59 events.

Use Europe/Berlin for the Mobical test account. That matches the current
server settings.
2011-10-25 15:17:05 +00:00
Patrick Ohly
1138f4fb7e Google CalDAV testing: ignore Google bug #61: cannot remove detached recurrence
Somehow a PUT which removes the detached recurrence from a meeting has
no effect. Looks like a Google server bug. Keep testing, but ignore
the failure.
2011-10-17 08:45:31 +00:00
Patrick Ohly
01519b1204 testing: avoid add<->add tests for Funambol, Memotoo, Mobical
All of these servers do not use the UID to find duplicates and
thus duplicate items in these tests, in contrast to what the test
expects.
2011-10-10 08:00:34 +00:00
Patrick Ohly
ee882ca895 Apple testing: no need to avoid complex UID
The VEVENT UID with lots of special characters was unnecessarily
avoided, recent (all?) Apple Calendar Server handles that fine.
2011-10-10 08:00:10 +00:00
Patrick Ohly
9d90b98c9b testing: enable problematic eds_contact tests for Memotoo
Memotoo seems to have some problem at the SyncML
level ("[SyncEvolution] Memotoo refresh + empty data"). Keep the
failing tests running for contacts so that we can debug this.
2011-08-26 08:31:41 +00:00
Patrick Ohly
782f23d247 testing: ignore testExtensions for Funambol
The test fails because we don't get CtCap from the server and don't
have a local copy. Don't run it.
2011-08-26 08:31:41 +00:00
Patrick Ohly
0d6ae7f6dd testing: run all Sync tests for DAViCal
The server runs locally, so enable more testing, just in case.
2011-08-26 08:31:41 +00:00
Patrick Ohly
4e35dc0555 syncevolution.org binaries: enable lpia again
For some strange reason, the ckeck for x86 was broken and can't have
worked like that. Running dpkg-architecture as make argument just
doesn't make sense.
2011-08-26 08:31:41 +00:00
Patrick Ohly
32fec43d6c testing: ignore various problems with servers
Client::Sync::*::testLinkedItems* cannot be run when testing multiple
sources at once because only event sources have the test data. Limit
testing to single sources.

Memotoo and Mobical need special linked item test case data because
they modify time stamps.

Memotoo refresh semantic seems to have changed. See "[SyncEvolution]
Memotoo refresh + empty data".

The name of the combined sources in _SKIP env variables were wrong.
2011-08-17 11:32:09 +00:00
Patrick Ohly
1af59c855f testing: removed hack for running setup command twice
This removes the change introduced with 2d0da5. Running the command
twice like that does not work when "sudo schroot" is involved because
it is only applied to the first invocation. The workaround for schroot
problems also no longer seems to be necessary.
2011-08-17 10:04:21 +00:00
Patrick Ohly
1070f1afaf testing: fixed prebuilt tests
The parameter for reusing prebuilt binaries points into the build
directory. Now we optionally use installed binaries if found in <build
dir>/../install.

Result checking and running commands must know where to find the
backends. Previously this was handled inconsistently and incorrectly.
2011-08-17 10:04:20 +00:00
Patrick Ohly
382d250b66 testing: fixed quoting when running commands
Commands with parameters containing single quotes were also quoted
with single quotes, which breaks. Check this and use double quotes
instead in this case. Still fails for parameters with a mixture of
quotes. Does not occur at the moment, though, so the simple quoting is
sufficient.
2011-08-17 10:04:20 +00:00
Patrick Ohly
72dbb2acc1 runtest.py: fixed valgrindcheck args, added VALGRIND_LEAK_CHECK_ONLY_FIRST
The valgrindcheck arguments weren't copied from runtest.py's
environment into the constructed command because valgrindcheck.sh is
present with path and thus only a substring of the command as string,
but not a word by itself (and thus not in the word array).

Also added the new VALGRIND_LEAK_CHECK_ONLY_FIRST.
2011-08-12 19:17:50 +00:00
Patrick Ohly
ffb216422e testing: preserve LD_LIBRARY_PATH from environment
LD_LIBRARY_PATH can be set in the environment *of runtest.py* and
will be forwarded to client-test/syncevolution as part of the
"env LD_LIBRARY_PATH=..." part of the command invocation.

This has to be done like this because sudo+schroot do not pass
through LD_LIBRARY_PATH and runtest.py itself needs to set
it for SyncEvolution libs, which must not overwrite the per-target
setting.
2011-08-12 19:17:49 +00:00
Patrick Ohly
469041216f testing: ignore leaks in forked process (local sync)
When SyncEvolution forks for local syncing, it doesn't clean
up after the forked process. This caused lots of reports
from valgrind which showed up in the nightly test results.

Now valgrindcheck.sh is used to automatically ignore them.
2011-08-12 19:17:49 +00:00
Patrick Ohly
1c1f2c9ae8 testing: text -> eds_memo renaming
Some test suppressions still used the old "text" name. Renamed
to "eds_memo".
2011-08-12 19:12:58 +00:00
Patrick Ohly
29ef449c68 testing: added DAViCal
Added CalDAV/CardDAV testing against DAViCal. Passes fine with
DAViCal 0.9.9.4-1.
2011-08-12 19:12:58 +00:00
Patrick Ohly
18a28a965e testing: fixed running commands with empty args
An empty argument (like the server list in tests where no server
was involved) must be quoted, too, otherwise it gets dropped.
Broke parsing of some test results.
2011-08-07 14:05:47 +00:00
Patrick Ohly
200476de8c testing: handle sudo as part of shell prefix
When sudo is necessary to enter the target platform, environment
variables no longer can be sent via the environment because
sudo ignores those. They must be be put onto the command line
of sudo, which then puts them into then environment of the invoked
process (depends on SETENV: permission in /etc/sudoers!).

This commit reorders the command line so that sudo comes
first, followed by env variables.
2011-08-07 14:05:47 +00:00
Patrick Ohly
79cdfe8ac3 Revert "runtests.py: avoid ssh access to gitorious.org in nightly checkout"
This reverts commit 4a52bd669f.

Conflicts:

	test/runtests.py

The patch had to be reverted because firewall rules now seem to leave
ssh access as the only means of accessing the repository. Conflict due
to later changes, resolved.
2011-07-15 01:47:27 +08:00
Patrick Ohly
ad003c6fc2 nightly testing: must set install env variables also for setup command
The setup command may also depend on correct paths, for example to
find specific backends.
2011-07-05 21:15:12 +08:00
Patrick Ohly
ce9a77ee5a nightly testing: test-dbus.py + valgrind
Just like resultchecker.py, test-dbus.py also shouldn't be run
under valgrind. In fact, this segfaults under Debian Testing...
2011-07-05 21:15:12 +08:00
Patrick Ohly
956fa25590 nightly testing: resultchecker.py + valgrind fix
resultchecker.py still run under valgrind because valgrindcheck.sh is
part of the test prefix, not the shell. Clean both variables now.
2011-07-05 21:15:12 +08:00
Patrick Ohly
5cd6c39890 D-Bus testing: integrated into nightly testing (BMC #5609)
runtests.py - made the SyncEvolutionTest class configurable so that it
can run test-bus.py, added "--enable=dbus" with it.

resultchecker.py - parse the output of test-dbus.py and split out the
failure reports for linking.
2011-06-30 16:50:36 -07:00
Patrick Ohly
62aab267d4 nightly testing: slightly nicer output
Avoid Python warning when destructing the Python file runs into the
already closed fd by closing the file instead of the low-level fd.
Print some information about the current action.
2011-06-30 15:17:17 -07:00
Patrick Ohly
6c8b5dbfdd WebDAV testing: run sync tests first
Running the sync tests first has the advantage that a successful
sync (one sync is enough, test doesn't have to pass) will set
the final resource URL and thus the source tests (which don't
do that) will run faster because they don't have to repeat
the auto discovery phase over and over again.
2011-06-19 15:04:15 +02:00
Patrick Ohly
b807e55158 testing: use WebDAV specific test cases for Client::Source::*::testImport
google_event.vcf and yahoo_contact.vcf were introduced together
with a configuration mechanism for selecting them because
different peers may need non-default test cases to pass the
testImport test. But the actual testing didn't use that yet - fixed.
2011-06-17 21:00:01 +08:00
Patrick Ohly
79d0feaf3d nightly testing: include configuration without EDS in dist check
When running a "dist" tests, also run "make distcheck" with
configure flags with disable EDS. Might trigger some issues
not seen otherwise, relevant for non-EDS users.
2011-05-16 20:30:29 +08:00
Patrick Ohly
9dfc18356d nightly testing: allow tests to override CLIENT_TEST_ALARM
This is done by inserting the test specific env variables after
the default value. Also reformatted the string and use saner
string formatting.

The Apple test uses that to increase its allowed runtime.
2011-05-05 20:16:45 +08:00
Patrick Ohly
adf061f4db nightly testing: avoid using valgrind in resultchecker
Passing the "valgrind[check.sh]" prefix through to resultchecker
causes unnecessary valgrind output in the runtest .out file because
some of the commands invoked by resultchecker are started under valgrind.

Strip valgrind from the shell prefix, leaving the rest (= schroot + dbus-session)
in place.
2011-05-05 20:16:45 +08:00
Patrick Ohly
b6861ed768 nightly testing: renamed ical20/itodo20/vcard30/text, removed vcard21 from Evolution backend (BMC #14972)
The distinction between vcard21 and vcard30 became mute in the Evolution
backend a while ago. Both tests ended up using the vCard 3.0 Evolution
tests data and the default uri for each server. This patch removes
the vCard 2.1 special case.

It also renames the tests and test data to reflect that they always
were Evolution specific. The new naming convention, also applied
to file, QtContacts, KCalExtended, XMLRPC, Maemo and Akonadi backends, is
now <backend>_contact/event/task/memo, with eds/file/qt/kcal/maemo/kde
as backend names.

The reasoning is:
- results in unique string (in particular no overlap with
  backend type names), easier to search for
- underscore already used before (in contrast to hyphen)
- no plural-s to keep the name shorter

The Akonadi backend should be using its own test data instead of
the Evolution ones.
2011-05-05 20:15:55 +08:00
Patrick Ohly
f29b87283f nightly testing: fixed typo in Google CalDAV testing
The sync testing is done against ical20, not vcard30.
2011-04-30 19:32:26 +08:00
Patrick Ohly
7422a1e2a1 nightly testing: avoid CLIENT_TEST_EVOLUTION_PREFIX=file://
The file:// prefix is no longer supported by EDS 2.32.
It also isn't supported by several other backends, for
example Akonadi. So let's use database names instead,
with a prefix defined via a new command line option.

These databases or directories (for the file backend)
must have been created in advance.
2011-04-30 19:32:26 +08:00
Patrick Ohly
be66091476 nightly testing: handle runs without tests
When the preparation failed and no tests were run, no email
with the HTML test report was sent.

Fixing this at two places:
- resultchecker.py always needs a "server" argument,
  even if it is empty
- the compare style sheet depends on a testing section,
  so write at least an empty one
2011-04-30 19:32:26 +08:00
Patrick Ohly
18d4d22723 nightly testing: fixed splitting of enabled tests
Must use regex, string.split() does not split at character
set.
2011-04-21 18:34:40 +08:00
Patrick Ohly
86ace96ec1 nightly testing: added sync testing with CalDAV/CardDAV
apple, googlecalendar and yahoo include now also Client::Sync
tests. For "apple", this is the full set of tests. For the
others, only testItems is run.
2011-04-21 18:34:40 +08:00
Patrick Ohly
feec9fffc5 nightly testing: support --enable foo=<tests> for all tests
Only the "evolution" test allowed selecting which client-test
tests to run. Now this code is in SyncEvolutionTest class itself.
The parameter to it with the default test list was turned into
a simple string, to match the value of the --enable parameter.
2011-04-20 20:47:57 +08:00
Patrick Ohly
10656b6484 nightly testing: unset http_proxy for client-test runs
This is needed for local CalDAV/CardDAV tests to succeed.
With http_proxy set, all traffic would be attempted via
these proxies, which doesn't work.

All tests which need proxies have to be configured such
that proxyHost/useProxy is set correctly in the server configs.
2011-04-20 20:47:57 +08:00
Patrick Ohly
be95fddac5 nightly testing: added Apple Calendar Server testing
runtests.py now has an entry for Apple Calendar Server,
with both CardDAV and CalDAV. The actual installation
can be local. To use this test, set up source-config@client-test-apple
with the correct syncURL and credentials.
2011-04-12 22:37:09 +08:00
Patrick Ohly
54aadcd6d7 runtest.py: "prebuilt" now only allowed once, replaces compiling
Previously, the --prebuilt option could be specified multiple times.
This was never used and just made its implementation unecessarily complex. Now
this option overrides compilation and leaves the rest of the test
running unmodified.
2011-04-05 18:56:04 +08:00
Patrick Ohly
3ad1911705 runtests.py: added Yahoo and Google CalDAV/CardDAV testing
Added separate Client::Source tests for Yahoo CalDAV/CardDAV
and Google CalDAV. Reporting the results needs to know that these
"servers" are local tests.

These tests can only be run if CLIENT_TEST_WEBDAV is set as
expected by WebDAVSourceRegister.cpp.
2011-04-01 18:26:37 +08:00
Patrick Ohly
b64763ee53 runtest.py: --help was broken without --enable
Not setting --enable caused a Python error because the "enable"
member of the config wasn't set at all. Must set a default.
2011-04-01 18:26:37 +08:00
Patrick Ohly
b9cd6e6f3b nightly testing: include webdav backend in PATH
The webdav directory contains syncevo-webdav-lookup, which is
needed for the WebDAV tests.
2011-03-29 03:54:33 +08:00
Patrick Ohly
eaeebb77e7 nightly testing: run setup command only once
The shell in Ubuntu Hardy was seen to not obey || && priorities.
Using explicit brackets solved that.
2011-03-29 03:54:33 +08:00
Patrick Ohly
59ebdb158b nightly testing: run "make clean" in same environment as testing
This was an attempt to avoid issues with the autotools scripts
and rebuilding their files. It did not solve the problem entirely
(--disable-maintainer-mode eventually did), but the change makes
sense nevertheless, for example of the directory inside a chroot
is different than the one in the host.
2011-03-29 03:54:33 +08:00
Patrick Ohly
54c2204d68 nightly testing: include CLIENT_TEST_WEBDAV in log
Rerunning the commands needs to use the same CLIENT_TEST_WEBDAV
env variable. Include it in the log output, similar to the valgrind
env variables.
2011-03-29 03:54:33 +08:00
Patrick Ohly
6a7a4f2441 nightly testing: updated logging of commands
First, dump information about relevant valgrind parameters
if valgrindcheck.sh is part of the command line. Now
executing the logged command acts like the original invocation.

Second, include the current directory as a "cd" command, to
simplify re-execution of the command.
2011-01-26 15:52:14 +08:00
Patrick Ohly
6fa31270b2 MemoToo: switched to testing vcard30
When using vCard 3.0, the default now with the MemoToo server,
certain problems (like lost NICKNAME) are avoided.
2011-01-21 11:00:19 +01:00