* Add a space after the "@comment" PLIST keyword inserted in the
case where "tests" option is disabled.
* Only create the "${PREFIX}/tests" directory and install the
Kyuafile there if the "tests" option is enabled.
**Released on August 26th, 2016.**
* Fixed execution of test cases as an unprivileged user, at least under
NetBSD 7.0. Kyua-level failures were probably a regression introduced
in Kyua 0.12, but the underlying may have existed for much longer:
test cases might have previously failed for mysterious reasons when
running under an unprivileged user.
* Issue #134: Fixed metadata test broken on 32-bit platforms.
* Issue #139: Added per-test case start/end timestamps to all reports.
* Issue #156: Fixed crashes due to the invalid handling of cleanup
routine data and triggered by the reuse of PIDs in long-running Kyua
instances.
* Issue #159: Fixed TAP parser to ignore case while matching `TODO` and
`SKIP` directives, and to also recognize `Skipped`.
* Fixed potential crash due to a race condition in the unprogramming of
timers to control test deadlines.
**Released on November 22nd, 2015.**
This is a huge release and marks a major milestone for Kyua as it finally
implements a long-standing feature request: the ability to execute test
cases in parallel. This is a big deal because test cases are rarely
CPU-bound: running them in parallel yields much faster execution times for
large test suites, allowing faster iteration of changes during development.
As an example: the FreeBSD test suite as of this date contains 3285 test
cases. With sequential execution, a full test suite run takes around 12
minutes to complete, whereas on a 4-core machine with a high level of
parallelism it takes a little over 1 minute.
Implementing parallel execution required rewriting most of Kyua's core and
partly explains explains why there has not been a new release for over a
year. The current implementation is purely subprocess-based, which works
but has some limitations and has resulted in a core that is really complex
and difficult to understand. Future versions will investigate the use of
threads instead for a simplified programming model and additional
parallelization possibilities.
* Issue #2: Implemented support to execute test cases in parallel when
invoking `kyua test`. Parallel execution is *only* enabled when the new
`parallelism` configuration variable is set to a value greater than `1`.
The default behavior is still to run tests sequentially because some test
suites contain test cases with side-effects that might fail when run in
parallel. To resolve this, the new metadata property `is_exclusive` can
be set to `true` on a test basis to indicate that the test must be run on
its own.
* Known regression: Running `kyua debug` on a TAP-based test program does
not currently report the output in real time. The output will only be
displayed once the test program completes. This is a shortcoming of
the new parallel execution engine and will be resolved.
* Removed the external C-based testers code in favor of the new built-in
implementations. The new approach feels significantly faster than the
previous one.
* Fixed the handling of relative paths in the `fs.*` functions available
in `Kyuafile`s. All paths are now resolved relative to the location of
the caller `Kyuafile`. `Kyuafile.top` has been updated with these
changes and you should update custom copies of this file with the new
version.
* Changed temporary directory creation to always grant search
permissions on temporary directories. This is to prevent potential
problems when running Kyua as root and executing test cases that require
dropping privileges (as they may later be unable to use absolute paths
that point inside their work directory).
* The cleanup of work directories does not longer attempt to deal with
mount points. If a test case mounts a file system and forgets to unmount
it, the mount point will be left behind. It is now the responsibility of
the test case to clean after itself. The reasons for this change are
simplicity and clarity: there are many more things that a test case can
do that have side-effects on the system and Kyua cannot protect against
them all, so it is better to just have the test undo anything it might
have done.
* Improved `kyua report --verbose` to properly handle environment
variables with continuation lines in them, and fixed the integration
tests for this command to avoid false negatives.
* Changed the configuration file format to accept the definition of
unknown variables without declaring them local. The syntax version
number remains at 2. This is to allow configuration files for newer Kyua
versions to work on older Kyua versions, as there is no reason to forbid
this.
* Fixed stacktrace gathering with FreeBSD's ancient version of GDB.
GDB 6.1.1 (circa 2004) does not have the `-ex` flag so we need to
generate a temporary GDB script and feed it to GDB with `-x` instead.
* Issue #136: Fixed the XML escaping in the JUnit output so that
non-printable characters are properly handled when they appear in the
process's stdout or stderr.
* Issue #141: Improved reporting of errors triggered by sqlite3. In
particular, all error messages are now tagged with their corresponding
database filename and, if they are API-level errors, the name of the
sqlite3 function that caused them.
* Issue #144: Improved documentation on the support for custom properties
in the test metadata.
* Converted the `INSTALL`, `NEWS`, and `README` distribution documents to
Markdown for better formatting online.
Issues found with existing distfiles:
distfiles/eclipse-sourceBuild-srcIncluded-3.0.1.zip
distfiles/fortran-utils-1.1.tar.gz
distfiles/ivykis-0.39.tar.gz
distfiles/enum-1.11.tar.gz
distfiles/pvs-3.2-libraries.tgz
distfiles/pvs-3.2-linux.tgz
distfiles/pvs-3.2-solaris.tgz
distfiles/pvs-3.2-system.tgz
No changes made to these distinfo files.
Otherwise, existing SHA1 digests verified and found to be the same on
the machine holding the existing distfiles (morden). All existing
SHA1 digests retained for now as an audit trail.
Changes in version 0.11
=======================
Released on October 23rd, 2014.
* Added support to print the details of all test cases (metadata and
their output) to 'report'. This is via a new '--verbose' flag which
replaces the previous '--show-context'.
* Added support to specify the amount of physical disk space required
by a test case. This is in the form of a new "required_disk_space"
metadata property, which can also be provided by ATF test cases as
"require.diskspace".
* Assimilated the contents of all the kyua-*-tester(1) and
kyua-*-interface(7) manual pages into more relevant places. In
particular, added more details on test program registration and their
metadata to kyuafile(5), and added kyua-test-isolation(7) describing
the isolation features of the test execution.
* Assimilated the contents of all auxiliary manual pages, including
kyua-build-root(7), kyua-results-files(7), kyua-test-filters(7) and
kyua-test-isolation(7), into the relevant command-specific manual
pages. This is for easier discoverability of relevant information
when reading how specific Kyua commands work.
* Issue 30: Plumbed through support to query configuration variables
from ATF's test case heads. This resolves the confusing situation
where test cases could only do this from their body and cleanup
routines.
* Issue 49: Extended 'report' to support test case filters as
command-line arguments. Combined with '--verbose', this allows
inspecting the details of a test case failure after execution.
* Issue 55: Deprecated support for specifying test_suite overrides on
a test program basis. This idiom should not be used but support for
it remains in place.
* Issue 72: Added caching support to the getcwd(3) test in configure so
that the result can be overriden for cross-compilation purposes.
* Issue 83: Changed manual page headings to include a "kyua" prefix in
their name. This prevents some possible confusion when displaying,
for example, the "kyua-test" manual page with a plain name of "test".
* Issue 84: Started passing test-suite configuration variables to plain
and TAP test programs via the environment. The name of the
environment variables set this way is prefixed by TEST_ENV_, so a
configuration variable of the form
'test_suites.some_name.allow_unsafe_ops=yes' in kyua.conf becomes
'TEST_ENV_allow_unsafe_ops=YES' in the environment.
* Issue 97 and 116: Fixed the build on Illumos.
* Issue 102: Set TMPDIR to the test case's work directory when running
the test case. If the test case happens to use the mktemp(3) family
of functions (due to misunderstandings on how Kyua works or due to
the reuse of legacy test code), we don't want it to easily escape the
automanaged work directory.
* Issue 103: Started being more liberal in the parsing of TAP test
results by treating the number in 'ok' and 'not ok' lines as optional.
* Issue 105: Started using tmpfs instead of md as a temporary file
system for tests in FreeBSD so that we do not leak md(4) devices.
* Issue 109: Changed the privilege dropping code to start properly
dropping group privileges when unprivileged_user is set. Also fixes
testers/run_test:fork_wait__unprivileged_group.
* Issue 110: Changed 'help' to display version information and clarified
the purpose of the 'about' command in its documentation.
* Issue 111: Fixed crash when defining a test program in a Kyuafile that
has not yet specified the test suite name.
* Issue 114: Improved the kyuafile(5) manual page by clarifying the
restrictions of the include() directive and by adding abundant
examples.
This is a "new" import that unifies all the previous Kyua subpackages
(kyua-atf-compat, kyua-cli and kyua-testers) under a single package.
The change has been made upstream mainly for simplicity reasons; the
old modular scheme was unnecessary.
Given that this new import should be really considered to be an update,
the following are the changes since kyua-cli-0.9 and kyua-testers-0.3,
both released last week:
Experimental version released on August 14th, 2014.
* Merged 'kyua-cli' and 'kyua-testers' into a single 'kyua' package.
* Dropped the 'kyua-atf-compat' package.
* Issue 100: Do not try to drop privileges to unprivileged_user when we
are already running as an unprivileged user. Doing so is not possible
and thus causes spurious test failures when the current user is not
root and the current user and unprivileged_user do not match.
* Issue 79: Mention kyua.conf(5) in the "see also" section of kyua(1).
* Issue 75: Change the rewrite__expected_signal__bad_arg test in
testers/atf_result_test to use a different signal value. This is to
prevent triggering a core dump that made the test fail in some platforms.
