Commit graph

88 commits

Author SHA1 Message Date
Patrick Ohly
86ace96ec1 nightly testing: added sync testing with CalDAV/CardDAV
apple, googlecalendar and yahoo include now also Client::Sync
tests. For "apple", this is the full set of tests. For the
others, only testItems is run.
2011-04-21 18:34:40 +08:00
Patrick Ohly
feec9fffc5 nightly testing: support --enable foo=<tests> for all tests
Only the "evolution" test allowed selecting which client-test
tests to run. Now this code is in SyncEvolutionTest class itself.
The parameter to it with the default test list was turned into
a simple string, to match the value of the --enable parameter.
2011-04-20 20:47:57 +08:00
Patrick Ohly
10656b6484 nightly testing: unset http_proxy for client-test runs
This is needed for local CalDAV/CardDAV tests to succeed.
With http_proxy set, all traffic would be attempted via
these proxies, which doesn't work.

All tests which need proxies have to be configured such
that proxyHost/useProxy is set correctly in the server configs.
2011-04-20 20:47:57 +08:00
Patrick Ohly
be95fddac5 nightly testing: added Apple Calendar Server testing
runtests.py now has an entry for Apple Calendar Server,
with both CardDAV and CalDAV. The actual installation
can be local. To use this test, set up source-config@client-test-apple
with the correct syncURL and credentials.
2011-04-12 22:37:09 +08:00
Patrick Ohly
54aadcd6d7 runtest.py: "prebuilt" now only allowed once, replaces compiling
Previously, the --prebuilt option could be specified multiple times.
This was never used and just made its implementation unecessarily complex. Now
this option overrides compilation and leaves the rest of the test
running unmodified.
2011-04-05 18:56:04 +08:00
Patrick Ohly
3ad1911705 runtests.py: added Yahoo and Google CalDAV/CardDAV testing
Added separate Client::Source tests for Yahoo CalDAV/CardDAV
and Google CalDAV. Reporting the results needs to know that these
"servers" are local tests.

These tests can only be run if CLIENT_TEST_WEBDAV is set as
expected by WebDAVSourceRegister.cpp.
2011-04-01 18:26:37 +08:00
Patrick Ohly
b64763ee53 runtest.py: --help was broken without --enable
Not setting --enable caused a Python error because the "enable"
member of the config wasn't set at all. Must set a default.
2011-04-01 18:26:37 +08:00
Patrick Ohly
b9cd6e6f3b nightly testing: include webdav backend in PATH
The webdav directory contains syncevo-webdav-lookup, which is
needed for the WebDAV tests.
2011-03-29 03:54:33 +08:00
Patrick Ohly
eaeebb77e7 nightly testing: run setup command only once
The shell in Ubuntu Hardy was seen to not obey || && priorities.
Using explicit brackets solved that.
2011-03-29 03:54:33 +08:00
Patrick Ohly
59ebdb158b nightly testing: run "make clean" in same environment as testing
This was an attempt to avoid issues with the autotools scripts
and rebuilding their files. It did not solve the problem entirely
(--disable-maintainer-mode eventually did), but the change makes
sense nevertheless, for example of the directory inside a chroot
is different than the one in the host.
2011-03-29 03:54:33 +08:00
Patrick Ohly
54c2204d68 nightly testing: include CLIENT_TEST_WEBDAV in log
Rerunning the commands needs to use the same CLIENT_TEST_WEBDAV
env variable. Include it in the log output, similar to the valgrind
env variables.
2011-03-29 03:54:33 +08:00
Patrick Ohly
6a7a4f2441 nightly testing: updated logging of commands
First, dump information about relevant valgrind parameters
if valgrindcheck.sh is part of the command line. Now
executing the logged command acts like the original invocation.

Second, include the current directory as a "cd" command, to
simplify re-execution of the command.
2011-01-26 15:52:14 +08:00
Patrick Ohly
6fa31270b2 MemoToo: switched to testing vcard30
When using vCard 3.0, the default now with the MemoToo server,
certain problems (like lost NICKNAME) are avoided.
2011-01-21 11:00:19 +01:00
Patrick Ohly
44ef5a385b nightly testing: log report generation functions
Instead of calling os.system() directly, use "runCommand()". That logs
the command to stdout and checks the result.
2010-09-02 13:49:24 +02:00
Patrick Ohly
97426b498e nightly testing: disable testDeleteAllRefresh for Google
The test depends on "refresh from client" semantic, something that
Google doesn't support.
2010-09-02 13:48:22 +02:00
Patrick Ohly
d20cbf9428 nightly testing: only include services in report which actually were active
The code added all services to the list which it encountered, including
those which were disabled and thus got skipped during testing. This
artificially increased the size of the report; not even sure whether
this was intentional. Now only active services are added.
2010-09-02 13:48:05 +02:00
Patrick Ohly
96722196ef nightly testing: allow enabling valgrind for "prebuilt" test (BMC #1007)
--enable evolution=foo parameters are used for --enable prebuilt-bar,
but without also enabling native Evolution tests.
2010-08-31 15:52:47 +08:00
Patrick Ohly
155a349014 runtests.py: make repos group-writable
The usage model for the nightly repos is that the "nightly" user
updates them and other users modify them to try out changes before
committing them. This relies on group-writable repos.

Despite umask 0002 and drwxrwsr-x permissions of the work area
this didn't work, group only had read permission. This patch adds
an explicit chmod and sets "sharedRepository = group" to fix this
after a fresh clone.
2010-08-23 10:44:50 +02:00
Patrick Ohly
4a52bd669f runtests.py: avoid ssh access to gitorious.org in nightly checkout
The nightly test account only has anonymous read access to
gitorious.org via the git:// protocol. But for normal users
patching the files, access via ssh is needed to commit changes.

Work around this by temporarily patching the URL for origin.
"git remote set-url" would be more elegant than Perl, but
only recent git versions have that.
2010-08-23 10:10:59 +02:00
Patrick Ohly
6abf8c12f0 nightly testing: get sources from meego.gitorious.org 2010-07-12 22:19:45 +08:00
Patrick Ohly
da4bab20df nightly testing: updated suppression of tests for Memotoo and Funambol
Funambol doubles one email address during twinning.

Memotoo doesn't support refresh-from-client.
2010-06-09 11:58:15 +02:00
Chen Congwu
6ded33e01d Ovi nightly testing: set serverName to 'Ovi' instead of 'ovi' 2010-04-28 09:17:26 +02:00
Chen Congwu
25de8f2baf Enable Ovi test 2010-04-15 11:15:36 +02:00
Patrick Ohly
d741059975 runtest.py: check Synthesis source
We check out the Synthesis source first, then the SyncEvolution source.
That let's us use SyncEvolution's autogen.sh/gen-autotools.sh scripts
for checking whether both sources are clean and properly tagged.

For Synthesis, we have to set the SYNTHESISSRC env variable.
2010-03-16 13:24:20 +01:00
Patrick Ohly
73195a730b testing: insert valgrind before syncevolution (MB #6457)
Previously, the valgrind command was inserted in front of "env",
which always succeeded because child processes were not checked
(would include EDS). Now we run exactly the right process under
valgrind.
2010-01-22 16:34:54 +01:00
Chen Congwu
837c0429c8 Testing: Fix no html report error (MB#8391)
Do not write 'parameter' property, as the value sometimes fails the XML parser.
Generate a default HTML report if the report failed to generate.
2010-01-22 16:34:54 +01:00
Patrick Ohly
210a2656f7 nightly testing: git fetch fixes
Instead of fetching a specific repo, use just "git fetch" to
pull updates into an existing repo. Specifying the repo
should have worked, but apparently is not done correctly (not
yet?) by the git in Debian Stable.

Using "git fetch" assumes that the URL of the repo does not
change between updates. That should be the case for us.
2009-12-01 21:45:34 +01:00
tester
2d0da511f0 test/runtest.py: git checkout debugging
Added both 'set -x' and 'git show-ref' to get more information
because it seems that sometimes the latest source is not checked
out.
2009-12-01 21:45:34 +01:00
Patrick Ohly
70329670bf runtests.py: run setup command twice, just to be sure
In a setup where it was run inside schroot we got error
messages from schroot because gconfd got stuck and prevented
a clean unmount of /tmp. Even "kill -9" wouldn't get rid of
such a hanging gconfd.

As a workaround, running the setup command twice seems to
succeed, including getting rid of the handing gconfd.
2009-12-01 21:45:34 +01:00
Zhu, Yongsheng
ca5a4d76f4 Testing: clean log/test files before a new testing
Previously do cleanup operation after implementing
a syncevolution setup. But if this setup raises
exceptions, we might get incorrect test results.
so clean all these files before doing any other
preparations
2009-12-01 21:45:34 +01:00
Patrick Ohly
653c1986b9 runtest.py: fixed checking out branch from git
Checking out a branch which only exists remotely requires
a bit more work. Checking out tags might also have failed
due to an unnecessary "git pull". Now "git tag" and "git branch"
are used to check what the requested revision is and do the
necessary opertions.
2009-12-01 21:45:33 +01:00
Patrick Ohly
8e6bfc24e5 runtest.py: point prebuilt binaries to correct location of backends
The "prebuilt" test ran, but because it used the build directory
of the current platform it didn't find any backends, or the wrong
ones. Fixed by only using the backend setting from the previous
testing if the directory really exists and falling back to the
relative "backends" directory otherwise.

It might be possible to always use "backends". In order to not
change the running testing this wasn't done.
2009-12-01 21:45:33 +01:00
Zhu, Yongsheng
280e67bddc Testing: refine return code checkings(bug#6457)
when memory leaks occur, the return code should be '100'
but python on unix will re-encode it and put them in a high byte
in a 16-bit number.
2009-12-01 21:45:33 +01:00
Zhu, Yongsheng
46a3ed30ee Mobical: skip unit test testOneWayFromClient(MB#8121)
Mobical has a known server issue that it initiates a
slow sync when one client syncs with it in one-way-from-client
mode from the second time.

Because Mobical has no schedule on this, we skip this
test for Mobical interoperability test.
2009-12-01 21:42:41 +01:00
Congwu Chen
6d2be17e6f runtests.py: added configuration of Resend test
This change was originally committed a while back to the
runtest.py used in the nightly testing. Now we are merging
it back to the public runtest.py.

This patch adds the "Resend" test to various _SKIP and
_FAILURES lists and adds CLIENT_TEST_RESEND_TIMEOUT and
CLIENT_TEST_INTERRUPT_AT.
2009-10-16 14:25:32 +02:00
Patrick Ohly
00cb014971 runtests.py: reformatted source code to split up long lines
The long and unreadable environment strings for client-test
are now in the Python code as multiple independent strings
which get concatenated by Python:
function("foo"
         "bar")
2009-10-16 14:25:32 +02:00
Patrick Ohly
ddaf76ec1b runtests.py: build RPMs in addition to DEBs
checkinstall *does* work on a Debian system, even under fakeroot.
The trick is to make /usr/src/rpm/RPMS writable for normal users.
The Makefile then copies the resulting .rpm back to the current
working directory.
2009-10-16 14:25:32 +02:00
Patrick Ohly
1cffe8fe6f runtests.py: fixed "prebuilt" tests
The invocation of the Client::Source tests for prebuilts binaries
was incorrect, the new source arraty was missing.
2009-10-16 14:25:32 +02:00
Patrick Ohly
fd40f4f87a runtests.py: fixed building lpia .deb
The "lpia" (Low Power Intel Architecture) .debs are our normal
x86 binaries packaged with a different architecture name. The check
whether runtests.py was building for x86 broke when moving to an
64 bit bost. It must be run inside the build environment.
2009-10-16 14:25:32 +02:00
Zhu, Yongsheng
a2cd73595b Testing: add nightly test for memotoo
Memotoo is ready to test so enable it
2009-10-16 14:25:31 +02:00
Chen Congwu
a2b079298e Testing: add time info for mail title
add time info for mail title for the purpose
of clear report
2009-10-16 14:25:31 +02:00
Chen Congwu
d1e9e24721 Testing: change know failures to skip for google testing
Change CLIENT_TEST_FAIURES to CLIENT_TEST_SKIP for google
testing
2009-10-16 14:25:31 +02:00
Chen Congwu
06fd773fd9 Testing: add valgrind check result and backend scan dir
1)Get return code from valgrind check result
2)Add environment variable SYNCEVOLUION_BACKEND_DIR when starting testing
2009-10-16 14:25:31 +02:00
Chen Congwu
950e9251c2 Testing: add valgrind check result and backend scan dir
Change failure to skip for google known failures
Change report title to have more acurate time information
2009-10-16 14:25:31 +02:00
Chen Congwu
015e1d3d9c Testing: a fix to avoid null servers list if compile error 2009-08-27 10:21:58 +02:00
Zhu, Yongsheng
b338591bae Testing: added Mobical.net in nightly test
Mobical.net supports 4 kinds of PIMs, e.g.,
contacts, calendar, task and memo.
Enable them in nightly test
2009-08-24 15:18:56 +02:00
Zhu, Yongsheng
c234306473 The main purpose is to refine nightly test report.
1) extract test results from many result files and combine them
into a xml format file to summary each unit test status
2) use 'compare.xsl' to compare two days' result and generate
comparison result
3) generate more visiable html output file by nightly result
and comparison result to make tracking regressions easily
2009-08-24 15:18:47 +02:00
Patrick Ohly
9c8eb81c45 testing: removed incorrect CLIENT_TEST_FAILURES=EvolutionCalendarTest::testOpenDefaultMemo
First, the test really shouldn't fail. Need to investigate why it does on
one test platform. Second, it was applied to all test runs, overwriting
the desired value of CLIENT_TEST_FAILURE for that run.
2009-07-24 17:00:50 +02:00
Patrick Ohly
89d1ae3b35 testing: don't compress .out files
These files used to contain the complete sync session
output. Now they contain just the user-visible output,
so don't compress anymore because that doesn't save much
space and makes the files harder to read.
2009-07-22 11:54:30 +02:00
Patrick Ohly
b69430fb98 testing: added --sanity-checks mode
In this mode, Client::Sync is replaced with
just Client::Sync::<first source>::testItems.
This assumes that all servers support WBXML
with our test data.
2009-07-22 10:53:01 +02:00