141 lines
4.7 KiB
XML
141 lines
4.7 KiB
XML
<!-- $NetBSD: regression.xml,v 1.10 2020/05/01 14:22:12 rillig Exp $ -->
|
|
|
|
<chapter id="regression"> <?dbhtml filename="regression.html"?>
|
|
<title>Regression tests</title>
|
|
|
|
<para>The pkgsrc infrastructure consists of a large codebase,
|
|
and there are many corners where every little bit of a file is
|
|
well thought out, making pkgsrc likely to fail as soon as
|
|
anything is changed near those parts. To prevent most changes
|
|
from breaking anything, a suite of regression tests should go
|
|
along with every important part of the pkgsrc infrastructure.
|
|
This chapter describes how regression tests work in pkgsrc and
|
|
how you can add new tests.</para>
|
|
|
|
<sect1 id="regression.run">
|
|
<title>Running the regression tests</title>
|
|
|
|
<para>You first need to install the <filename
|
|
role="pkg">pkgtools/pkg_regress</filename> package, which
|
|
provides the <command>pkg_regress</command> command. Then you
|
|
can simply run that command, which will run all tests in the
|
|
<filename>regress/</filename> directory.</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="regression.new">
|
|
<title>Adding a new regression test</title>
|
|
|
|
<para>Every directory in the <filename>regress/</filename>
|
|
directory that contains a file called <filename>spec</filename>
|
|
is considered a regression test. This file is a shell program
|
|
that is included by the <command>pkg_regress</command> command.
|
|
The following functions can be overridden to suit your
|
|
needs.</para>
|
|
|
|
<sect2 id="regression.fun.override">
|
|
<title>Overridable functions</title>
|
|
|
|
<para>These functions do not take any parameters. Although they
|
|
are called in <quote>set -e</quote> mode, they don't stop at the
|
|
first failing command. See <ulink
|
|
url="https://stackoverflow.com/q/4072984">this Stack Overflow
|
|
question</ulink> for details.</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry><term><varname>do_setup</varname></term>
|
|
<listitem><para>This function prepares the environment for the
|
|
test. By default it does nothing.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>do_test</varname></term>
|
|
<listitem><para>This function runs the actual test. By default,
|
|
it calls <varname>TEST_MAKE</varname> with the arguments
|
|
<varname>MAKEARGS_TEST</varname> and writes its output including
|
|
error messages into the file
|
|
<varname>TEST_OUTFILE</varname>.</para>
|
|
|
|
<para>When defining this function, make sure that all output that
|
|
needs to be checked is written to the correct output file.
|
|
Example:</para>
|
|
|
|
<programlisting>
|
|
do_test() {
|
|
echo "Example output"
|
|
} 1>$TEST_OUTFILE 2>&1
|
|
</programlisting>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>check_result</varname></term>
|
|
<listitem><para>This function is run after the test and is
|
|
typically used to compare the actual output from the one that is
|
|
expected. It can make use of the various helper functions from
|
|
the next section. Example:</para>
|
|
|
|
<programlisting>
|
|
check_result() {
|
|
exit_status 0
|
|
output_require "Example"
|
|
output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
|
|
output_prohibit "no such file or directory"
|
|
regress_fail "expected $expected but got $actual for input $input"
|
|
}
|
|
</programlisting>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>do_cleanup</varname></term>
|
|
<listitem><para>This function cleans everything up after the
|
|
test has been run. By default it does nothing.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
</sect2>
|
|
|
|
<sect2 id="regression.fun.helper">
|
|
<title>Helper functions</title>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><varname>regress_fail <replaceable>message...</replaceable></varname></term>
|
|
<listitem><para>This function makes the test fail with the given error message.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>exit_status expected</varname></term>
|
|
<listitem><para>This function compares the exitcode of the
|
|
<command>do_test</command> function with its first parameter.
|
|
If they differ, the test will fail.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>output_require regex...</varname></term>
|
|
<listitem><para>This function checks for each of its parameters
|
|
if the output from <command>do_test</command> matches the
|
|
extended regular expression. If it does not, the test will
|
|
fail. Example:</para>
|
|
|
|
<programlisting>
|
|
output_require "looks fine"
|
|
output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
|
|
</programlisting>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry><term><varname>output_prohibit regex...</varname></term>
|
|
<listitem><para>This function checks for each of its parameters
|
|
if the output from <command>do_test()</command> does
|
|
<emphasis>not</emphasis> match the extended regular expression.
|
|
If any of the regular expressions matches, the test will
|
|
fail.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</sect2>
|
|
</sect1>
|
|
</chapter>
|