This is the result of 2to3, with the interpreter path updated manually,
tabs replaced by spaces and some obsolete packages removed.
Required because the current nightly build host no longer has Python2 installed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The current HTTP server for nightly.syncevolution.org reports the
content type for .txt files as plain text, but not for .log
files. Plain text is desirable for easy viewing in a web
browser. While the .log suffix is nicer, getting the HTTP server
reconfigured is hard and might have to be repeated again in the
future, so let's just use .txt.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
"evolution" tests used to be a traditional name for all Client::Source
and SyncEvolution unit tests. This caused problems when adding
Akonadi (crash in Akonadi client lib when EDS was used before) and
was not nice in the test reporting (multiple different tests all mixed
into one summary line). It also prevented running these tests in parallel.
All of this gets fixed by splitting "evolution" into "eds", "kde",
"file" and "unittests". To avoid updating the nightly test run scripts,
--enable evolution=.... is still understood.
Don't assume that each server actually has an entry in the result
list. In case of a failure to write that, treat the server as failed
to highlight the problem instead of failing in the resultchecker.py
script.
"dist" must complete before being able to install and test packages on
different platforms. By running "make distcheck" separately, we can
start with parallel testing sooner and separately enable/disable
distcheck testing.
Compilation and actual testing will be split into two runtest.py invocations
to support parallelizing of testing. resultchecker.py must handle the
resulting output.txt, which will have multiple entries for for each action:
first compilation followed by skipped testing, then skipped compilation
followed by testing. resultcheckper.py must report the non-skipped
results and only fall back to "skipped" if none are found.
In addition, make it clear that the initial steps are about the source code by
renaming the step in the report with a "-source" suffix.
When testing pre-compiled binaries it may happen that we cannot
determine library versions with pkg-config because development
files are not installed. Ignore these errors and skip adding
library version numbers.
The resultchecker.py script must not rely on being able to cd into
a specific directory and then having the chroot use that same directory,
because paths may be different. Instead always use absolute paths inside
the schroot commands.
In the future, nightly testing will use chroots which copy their root
filesystem and all additional files, instead of bind mounting them as it is
done now. The advantage of that setup is that the test runs become independent
from each other and thus can be parallelized. The downside is that paths to
files created inside the chroot has to be accessed via a different path from
the outside.
runtests.py will be invoked with those outside paths and needs to translate
them to the shorter paths used inside the chroot when invoking commands that
run inside it. This affects the current directory and any paths in the
argument list.
This enables interoperability testing with owndrive.com, a hosted OwnCloud
provider. owndrive.com was chosen because a user picked it and support was
able to help with some problems. It's also a bit simpler than running OwnCloud
ourselves. But it may also be slower, so perhaps ultimately a local
installation will be needed.
This adds testing of Google Contacts syncing via CardDAV. It passes
only thanks to the simplified EDS contacts test cases, with one exception:
removing extensions is known to fail because the server keeps them even
when receiving an update without them.
Start testing the new org._01.pim.contacts D-Bus API, as implemented
by SyncEvolution with EDS as storage.
Currently EDS has a race condition where an ESource becomes
unremovable when reusing the UID. Therefore the tests temporarily
avoid doing that.
The sleeps are necessary to prevent problems caused by creating and
removing an ESource before the file monitoring code in
evolution-source-registry can process the file events.
Testing shows that compared to Exchange, some additional problems
exist, which are ignored via synccompare:
- FN gets overwritten.
- CATGEORIES are lost.
Not sure whether anything can be done, so let's consider this
permanently broken.
BDAY also gets modified. Perhaps that can be fixed, so a synccompare
workaround will be commit separately so that it can be reverted later.
Suspend/Resend/Retry are sub-groups of Client::Sync. They need to be enabled
when gathering the list of all sync tests. Finding the right logs needs to
be tweaked because of different naming and several logs per tests.
resultchecker.py must configure client-test so that it enables
the restart tests when asked for a list of existing tests. The
nightly HTML overview was incomplete without this change.
activesyncd gets compiled completely (which should always succeed) and
then only the required pieces are installed (without the things which
are hard-coded for /usr, because installing those will fail).
Testing is done by running a similar set of client-test tests as for remove
CalDAV/CardDAV servers.
activesyncd is started anew each time client-test is run, done in
the new wrappercheck.sh script. Can be combined with valgrindcheck.sh:
wrappercheck.sh valgrindcheck.sh activesyncd -- valgrindcheck.sh client-test ...
The return code of wrappercheck.sh is the return code of the real command
or the daemon, if the real command succeeded. This way the special 100 return
code from valgrindcheck.sh is returned if the only problems were memory
issues.
Instead of hard-coding a set of .log files which need to be skipped,
only do it for files which match the expected naming pattern. Also
only match the base name of the file instead of the full path,
to avoid mismatches.
Some tests were incorrectly marked as "okay" because the there was
a line in the error message matching the "okay" regex. Now the pattern
also expects "okay" or "ok" at the end of the line.
The logic for detecting the start and end of an error report did not handle
multiple errors for a test. Now it does not overwrite an already opened
.log file anymore and keeps writing into it until some line really
indicates the beginning of another error.
As a new feature, the .html version of the .log file is written at the
same time as the .log file. It contains minor color coding. Its main
advantage is separation of D-Bus and stdout output and anchors with quick
links at the top of the page (can be used for skipping the D-Bus output).
On Debian Testing with Python 2.7 the nightly report did not properly
report failed tests. That's because now python-unit writes error
reports with the failed function on the error line, instead of using
the doc string. The change is useful (because doc strings are not
necessarily enough to identify the test, which SyncEvolution had to
work around by imposing a specific format of the doc strings), but the
change broke output parsing. Now both old and new output is matched.
Merge branches are removed locally before fetching from the remote repo.
Ensures that obsolete branched which where removed from the remote repo
also no longer exist locally. By itself, "git fetch" doesn't do that.
Automatic integration ignores branches which were already merged into
the upstream remote branch. Avoids redundant listing of branches
in the integration report which didn't really need to be merged for the
test run.
Unstaged and staged changes ("stash") are included in the patch report.
The patches are sorted from oldest to most recent. Fixed the fallback
code when a patch has no Subject.
The scripts for reporting tests results are taken from the merged
source code. That ensures that the reporting matches the tests
that were run. The versions from the boot-strapping SyncEvolution
code are the fallback, just in case that automatic integration
fails.
(cherry picked from commit 33bbf5df57)
Conflicts:
test/generate-html.xsl
In contrast to the master branch, URLs are still absolute because
backporting those changes would have been more difficult.
Merge branches are removed locally before fetching from the remote repo.
Ensures that obsolete branched which where removed from the remote repo
also no longer exist locally. By itself, "git fetch" doesn't do that.
Automatic integration ignores branches which were already merged into
the upstream remote branch. Avoids redundant listing of branches
in the integration report which didn't really need to be merged for the
test run.
Unstaged and staged changes ("stash") are included in the patch report.
The patches are sorted from oldest to most recent. Fixed the fallback
code when a patch has no Subject.
The scripts for reporting tests results are taken from the merged
source code. That ensures that the reporting matches the tests
that were run. The versions from the boot-strapping SyncEvolution
code are the fallback, just in case that automatic integration
fails.
The URLs in the on-disk HTML are all relative. That allows rsyncing
the results directories without having to translate hrefs. The version
sent via mail still has absolute external URLs (the original motivation
for embedding the full URL).
Numbering Client::Source::LinkedItems_xxx with xxx being a number is
confusing, in particular when the same number stands for different
test data. Now each set of linked items has an additional, unique name
which is used for Client::Source::LinkedItems<Name>.
Done in combination with adding more linked item tests and slightly
reorganizing the logic for adding them:
- a default set with VTIMEZONE is added in all cases
- some SyncML servers override that default set
- others, in particular peers accessed via their own backend,
enable additional Client::Source tests on a case-by-case basis
Exchange is only tested with its own default set (with "Standard
Timezone" as TZID) and the all-day recurring set (as before).
All other CalDAV servers are now also tested with the all-day
set (previously exclusive to Exchange) and local floating time (= no
TZID, new).
Google CalDAV can't be tested with local time because it converts such
events into the time zone of the current user. All-day events need
special test data because Google adds a time to the UNTIL clause
(http://code.google.com/p/google-caldav-issues/issues/detail?id=63).
synccompare also needs to ignore that Google adds a redundant VTIMEZONE
to the all-day test cases.
Finally, Client::Source tests for updating a child event (with and
without parent) without UID and RECURRENCE-ID inside the payload were
added. These properties are removed via text operations. The
expectation is that the source is able to add them back (if needed)
based on the meta information that it has about the existing item.
The file source is unable to do that. When using it in an HTTP server,
the server will look to peers like a peer which doesn't support the
semantic (which indeed it doesn't) and thus the client will add back
the fields.
Branches committed to the libsynthesis or SyncEvolution git repo with
a name of "for-<branch>/<name>" will be automatically merged into
<branch> before starting the testing:
--sourcedir <dir with existing repos> --workdir <dir with merged repos>
The result can be reused as-is in further runtests.py invocation by using
--sourcedir <merged repos> --no-sourcedir-copy
Any local, *uncommitted* changes will be preserved and applied to
the merged sources (git stash). *Committed* changes will *not* be
applied. Always push them immediately.
Because there is a risk of loosing work when these rules are not
followed, the current state of the tree is stored as a branch created
with "git stash create" under the names
<branch>-nightly-before-<date+time> and <branch>-nightly. The latter
gets overwritten during each run.
These branches can be used to cherry-pick commits or restore the tree
as it was before the automatic merging ("git stash branch <branchname>
<branch>-nightly").
Result reporting includes information about the resulting code
revision. This includes the names of all branches which were
available for merging and whether that succeeded or failed. The output
of "git describe" is also included, always with a hash (even when it
matches a tag, in case that the tag gets moved) and whether the
working tree was dirty. All commits applied to the base branch are
listed in "git am" format.
This is done because the merged code is not pushed. This would clutter
the upstream code repository.
The "for-" prefix (instead of just "<branch>/") was added because git
got confused when <branch> was both a real branch and a directory of
other branches.
(cherry picked from commit 3e9fc042dc)
Branches committed to the libsynthesis or SyncEvolution git repo with
a name of "for-<branch>/<name>" will be automatically merged into
<branch> before starting the testing:
--sourcedir <dir with existing repos> --workdir <dir with merged repos>
The result can be reused as-is in further runtests.py invocation by using
--sourcedir <merged repos> --no-sourcedir-copy
Any local, *uncommitted* changes will be preserved and applied to
the merged sources (git stash). *Committed* changes will *not* be
applied. Always push them immediately.
Because there is a risk of loosing work when these rules are not
followed, the current state of the tree is stored as a branch created
with "git stash create" under the names
<branch>-nightly-before-<date+time> and <branch>-nightly. The latter
gets overwritten during each run.
These branches can be used to cherry-pick commits or restore the tree
as it was before the automatic merging ("git stash branch <branchname>
<branch>-nightly").
Result reporting includes information about the resulting code
revision. This includes the names of all branches which were
available for merging and whether that succeeded or failed. The output
of "git describe" is also included, always with a hash (even when it
matches a tag, in case that the tag gets moved) and whether the
working tree was dirty. All commits applied to the base branch are
listed in "git am" format.
This is done because the merged code is not pushed. This would clutter
the upstream code repository.
The "for-" prefix (instead of just "<branch>/") was added because git
got confused when <branch> was both a real branch and a directory of
other branches.
"oracle" is used as server name for Oracle. Oracle Beehive also
still exists, but seems less relevant now.
"egroupware-dav" is used for eGroupware when using CalDAV/CardDAV,
to distinguish this from the older eGroupware SyncML testing.
With Google Calendar, having a VALARM leads to different communication
compared to the case without VALARM (requires resending). Test both
cases by providing multiple sets of linked items.
Client::Source now contains LinkedItems_1 and LinkedItems_2
sub-groups. This required changes in resultchecker.py, to keep these
tests as listed in the same table as the other source tests.
ClientTestConfig becomes more complicated: it used to be a plain C
struct which could be copied/cleaned with memcpy/memset. This approach
is kept by adding a pointer to a std::vectore. A nicer solution would
be to turn all "const char *" into std::string and int/bool values
with wrapper classes which initialize them.
First, the list of tests was incomplete because there is no single
source which has all tests enabled. Must create a union of all enabled
tests. Do it so that non-standard tests are listed last.
Second, don't break that order when creating the HTML output. Order as
run is better than alphabetical order.
runtests.py - made the SyncEvolutionTest class configurable so that it
can run test-bus.py, added "--enable=dbus" with it.
resultchecker.py - parse the output of test-dbus.py and split out the
failure reports for linking.
An empty "server" parameter still wasn't handled
correctly, leading to missing result emails.
Empty string must be treated like an empty list.
resultchecker.py incorrectly stored the result of
googlecalendar for google because it was only checking
the line prefix. Now the check ensures that the line
starts with a complete word matching the server name.
The distinction between vcard21 and vcard30 became mute in the Evolution
backend a while ago. Both tests ended up using the vCard 3.0 Evolution
tests data and the default uri for each server. This patch removes
the vCard 2.1 special case.
It also renames the tests and test data to reflect that they always
were Evolution specific. The new naming convention, also applied
to file, QtContacts, KCalExtended, XMLRPC, Maemo and Akonadi backends, is
now <backend>_contact/event/task/memo, with eds/file/qt/kcal/maemo/kde
as backend names.
The reasoning is:
- results in unique string (in particular no overlap with
backend type names), easier to search for
- underscore already used before (in contrast to hyphen)
- no plural-s to keep the name shorter
The Akonadi backend should be using its own test data instead of
the Evolution ones.
The script assumed that "schroot" was always used as "schroot -c <chroot>".
That broke when using "schroot -v -c <chroot>" for debugging.
This patch removes the assumption by inserting -i into the
schroot invocation and retrieving the name from the output.
When the preparation failed and no tests were run, no email
with the HTML test report was sent.
Fixing this at two places:
- resultchecker.py always needs a "server" argument,
even if it is empty
- the compare style sheet depends on a testing section,
so write at least an empty one
runtests.py now has an entry for Apple Calendar Server,
with both CardDAV and CalDAV. The actual installation
can be local. To use this test, set up source-config@client-test-apple
with the correct syncURL and credentials.
A previous fix for calendar+todo in the source name introduced
escaping of the special + character. This escaping also replaced _
with __. Unescaping was missing in several places, leading to
google_caldav being shown as google__caldav with links to non-existent
files Client_Source_google_caldav_testOpen.log.
This test is a bit special because it is only run for the first source
and doesn't really talk to the server. The logic in resultchecker.py
missed that test because the invocation of "client-test --help" didn't
print that test for "vcard21". Fixed by ensuring that only "vcard21"
is active during the --help call.
The Client::Source tests were sorted by name. This was confusing
because it hides when a peer like a CalDAV server stops responding
after a certain test.
It is more intuitive to present the results in the order in which they
were run. This is accomplished by sorting by the sub-second modification
time of the corresponding .log files (in resultschecker.py) and keeping
that order (in generate-html.xsl).
A similar approach might also be useful for Client::Sync. Not done yet.
Added separate Client::Source tests for Yahoo CalDAV/CardDAV
and Google CalDAV. Reporting the results needs to know that these
"servers" are local tests.
These tests can only be run if CLIENT_TEST_WEBDAV is set as
expected by WebDAVSourceRegister.cpp.
This column is useful to determine at a glance whether tests
failed because of something outside of our control. The value
is also shown for each single test.
When a failure is known and it should not be treated as
a 'failed' case. Instead, 'knownfailure' is used to indicate
it. Thus, a new testing status for each case is shown in
the nightly testing outputs.
If the source type is 'calendar+todo' type, XML spec doesn't allow
'+' in the element tag. Thus the current solution is to replace
'calendar+todo' with 'calendar_todo'.
But this is not a good choice. Instead, escaped strings are used.
"_" is escaped to "__". "+" is escaped to "_-".
So firstly escape these 2 characters when generating nightly xml
documents and then unescape them when generating nightly html page.
Patrick Ohly
Zhu, Yongsheng
Chen Congwu