111 lines
3.8 KiB
XML
111 lines
3.8 KiB
XML
<!-- $NetBSD: regression.xml,v 1.2 2006/12/15 13:22:14 martti Exp $ -->
|
|
|
|
<chapter id="regression"> <?dbhtml filename="regression.html"?>
|
|
<title>Regression tests</title>
|
|
|
|
<para>The pkgsrc infrastructure consists of a large codebase,
|
|
and there are many corners where every little bit of a file is
|
|
well thought out, making pkgsrc likely to fail as soon as
|
|
anything is changed near those parts. To prevent most changes
|
|
from breaking anything, a suite of regression tests should go
|
|
along with every important part of the pkgsrc infrastructure.
|
|
This chapter describes how regression tests work in pkgsrc and
|
|
how you can add new tests.</para>
|
|
|
|
<sect1 id="regression.descr">
|
|
<title>The regression tests framework</title>
|
|
|
|
<para></para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="regression.run">
|
|
<title>Running the regression tests</title>
|
|
|
|
<para>You first need to install the <filename
|
|
role="pkg">pkgtools/pkg_regress</filename> package, which
|
|
provides the <command>pkg_regress</command> command. Then you
|
|
can simply run that command, which will run all tests in the
|
|
<filename>regress</filename> category.</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="regression.new">
|
|
<title>Adding a new regression test</title>
|
|
|
|
<para>Every directory in the <filename>regress</filename>
|
|
category that contains a file called <filename>spec</filename>
|
|
is considered a regression test. This file is a shell program
|
|
that is included by the <command>pkg_regress</command> command.
|
|
The following functions can be overridden to suit your
|
|
needs.</para>
|
|
|
|
<sect2 id="regression.fun.override">
|
|
<title>Overridable functions</title>
|
|
|
|
<para>These functions do not take any parameters. They are all
|
|
called in <quote>set -e</quote> mode, so you should be careful
|
|
to check the exitcodes of any commands you run in the
|
|
test.</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry><term><varname>do_setup()</varname></term>
|
|
<listitem><para>This function prepares the environment for the
|
|
test. By default it does nothing.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>do_test()</varname></term>
|
|
<listitem><para>This function runs the actual test. By default,
|
|
it calls <varname>TEST_MAKE</varname> with the arguments
|
|
<varname>MAKEARGS_TEST</varname> and writes its output including
|
|
error messages into the file
|
|
<varname>TEST_OUTFILE</varname>.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>check_result()</varname></term>
|
|
<listitem><para>This function is run after the test and is
|
|
typically used to compare the actual output from the one that is
|
|
expected. It can make use of the various helper functions from
|
|
the next section.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>do_cleanup()</varname></term>
|
|
<listitem><para>This function cleans everything up after the
|
|
test has been run. By default it does nothing.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
</sect2>
|
|
|
|
<sect2 id="regression.fun.helper">
|
|
<title>Helper functions</title>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry><term><varname>exit_status(expected)</varname></term>
|
|
<listitem><para>This function compares the exitcode of the
|
|
<command>do_test()</command> function with its first parameter.
|
|
If they differ, the test will fail.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>output_require(regex...)</varname></term>
|
|
<listitem><para>This function checks for each of its parameters
|
|
if the output from <command>do_test()</command> matches the
|
|
extended regular expression. If it does not, the test will
|
|
fail.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>output_prohibit(regex...)</varname></term>
|
|
<listitem><para>This function checks for each of its parameters
|
|
if the output from <command>do_test()</command> does
|
|
<emphasis>not</emphasis> match the extended regular expression.
|
|
If any of the regular expressions matches, the test will
|
|
fail.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</sect2>
|
|
</sect1>
|
|
</chapter>
|