pkgsrc/doc/guide/files/buildlink.xml
2013-07-30 07:14:09 +00:00

594 lines
24 KiB
XML

<!-- $NetBSD: buildlink.xml,v 1.36 2013/07/30 07:14:09 wiz Exp $ -->
<chapter id="buildlink">
<title>Buildlink methodology</title>
<para>Buildlink is a framework in pkgsrc that controls what headers and libraries
are seen by a package's configure and build processes. This is implemented
in a two step process:</para>
<orderedlist>
<listitem>
<para>Symlink headers and libraries for dependencies into
<varname>BUILDLINK_DIR</varname>, which by default is a subdirectory
of <varname>WRKDIR</varname>.</para>
</listitem>
<listitem>
<para>Create wrapper scripts that are used in place of the normal compiler
tools that translate <option>-I${LOCALBASE}/include</option> and
<option>-L${LOCALBASE}/lib</option> into references to
<varname>BUILDLINK_DIR</varname>. The wrapper scripts also make
native compiler on some operating systems look like GCC, so that
packages that expect GCC won't require modifications to build with
those native compilers.</para>
</listitem>
</orderedlist>
<para>This normalizes the environment in which a package is built so that the
package may be built consistently despite what other software may be
installed. Please note that the normal system header and library paths,
e.g. <filename>/usr/include</filename>,
<filename>/usr/lib</filename>, etc., are always searched -- buildlink3 is
designed to insulate the package build from non-system-supplied
software.</para>
<sect1 id="converting-to-buildlink3">
<title>Converting packages to use buildlink3</title>
<para>The process of converting packages to use the buildlink3
framework (<quote>bl3ifying</quote>) is fairly straightforward.
The things to keep in mind are:</para>
<orderedlist>
<listitem>
<para>Ensure that the build always calls the wrapper scripts
instead of the actual toolchain. Some packages are tricky,
and the only way to know for sure is the check
<filename>${WRKDIR}/.work.log</filename> to see if the
wrappers are being invoked.</para>
</listitem>
<listitem>
<para>Don't override <varname>PREFIX</varname> from within
the package Makefile, e.g. Java VMs, standalone shells,
etc., because the code to symlink files into
<filename>${BUILDLINK_DIR}</filename> looks for files
relative to <quote>pkg_info -qp <replaceable>pkgname</replaceable></quote>.
</para>
</listitem>
<listitem>
<para>Remember that <emphasis>only</emphasis> the
<filename>buildlink3.mk</filename> files that you list in a
package's Makefile are added as dependencies for that package.
</para>
</listitem>
</orderedlist>
<para>If a dependency on a particular package is required for its libraries and
headers, then we replace:</para>
<programlisting>
DEPENDS+= foo>=1.1.0:../../category/foo
</programlisting>
<para>with</para>
<programlisting>
.include "../../category/foo/buildlink3.mk"
</programlisting>
<para>The buildlink3.mk files usually define the required dependencies.
If you need a newer version of the dependency when using buildlink3.mk
files, then you can define it in your Makefile; for example:</para>
<programlisting>
BUILDLINK_API_DEPENDS.foo+= foo>=1.1.0
.include "../../category/foo/buildlink3.mk"
</programlisting>
<para>There are several <filename>buildlink3.mk</filename>
files in <filename>pkgsrc/mk</filename>
that handle special package issues:</para>
<itemizedlist>
<listitem>
<para><filename>bdb.buildlink3.mk</filename> chooses either
the native or a pkgsrc Berkeley DB implementation based on
the values of <varname>BDB_ACCEPTED</varname> and
<varname>BDB_DEFAULT</varname>.</para>
</listitem>
<listitem>
<para><filename>curses.buildlink3.mk</filename>: If the system
comes with neither Curses nor NCurses, this will take care
to install the <filename
role="pkg">devel/ncurses</filename> package.</para>
</listitem>
<listitem>
<para><filename>krb5.buildlink3.mk</filename> uses the value
of <varname>KRB5_ACCEPTED</varname> to choose between
adding a dependency on Heimdal or MIT-krb5 for packages that
require a Kerberos 5 implementation.</para>
</listitem>
<listitem>
<para><filename>motif.buildlink3.mk</filename> checks for a
system-provided Motif installation or adds a dependency on
<filename role="pkg">x11/lesstif</filename>, <filename
role="pkg">x11/motif</filename> or <filename
role="pkg">x11/openmotif</filename>. The user can set
<varname>MOTIF_TYPE</varname> to <quote>dt</quote>,
<quote>lesstif</quote>, <quote>motif</quote> or
<quote>openmotif</quote> to choose
which Motif version will be used.</para>
</listitem>
<listitem>
<para><filename>readline.buildlink3.mk</filename> checks for a
system-provided GNU readline or editline (libedit) installation,
or adds a dependency on <filename role="pkg">devel/readline</filename>,
<filename role="pkg">devel/editline</filename>. The user can set
<varname>READLINE_DEFAULT</varname> to choose readline implementation.
If your package really needs GNU readline library, its Makefile
should include <filename>devel/readline/buildlink3.mk</filename>
instead of <filename>readline.buildlink3.mk</filename>.</para>
</listitem>
<listitem>
<para><filename>oss.buildlink3.mk</filename> defines several
variables that may be used by packages that use the
Open Sound System (OSS) API.</para>
</listitem>
<listitem>
<para><filename>pgsql.buildlink3.mk</filename> will accept
any of the Postgres versions in the variable
<varname>PGSQL_VERSIONS_ACCEPTED</varname> and default to
the version <varname>PGSQL_VERSION_DEFAULT</varname>. See
the file for more information.</para>
</listitem>
<listitem>
<para><filename>pthread.buildlink3.mk</filename> uses the value of
<varname>PTHREAD_OPTS</varname> and checks for native pthreads or adds
a dependency on <filename
role="pkg">devel/pth</filename> as needed.</para>
</listitem>
<listitem>
<para><filename>xaw.buildlink3.mk</filename> uses the value of
<varname>XAW_TYPE</varname> to choose a particular Athena widgets
library.</para>
</listitem>
</itemizedlist>
<para>The comments in those <filename>buildlink3.mk</filename>
files provide a more complete
description of how to use them properly.</para>
</sect1>
<sect1 id="creating-buildlink3.mk">
<title>Writing <filename>buildlink3.mk</filename> files</title>
<anchor id="buildlink3.mk"/>
<para>A package's <filename>buildlink3.mk</filename> file is
included by Makefiles to indicate the need to compile and link
against header files and libraries provided by the package. A
<filename>buildlink3.mk</filename> file should always provide
enough information to add the correct type of dependency
relationship and include any other
<filename>buildlink3.mk</filename> files that it needs to find
headers and libraries that it needs in turn.</para>
<para>To generate an initial <filename>buildlink3.mk</filename>
file for further editing, Rene Hexel's <filename
role="pkg">pkgtools/createbuildlink</filename>
package is highly recommended. For most packages, the following
command will generate a good starting point for
<filename>buildlink3.mk</filename> files:</para>
<screen>
&cprompt; <userinput>cd pkgsrc/<replaceable>category</replaceable>/<replaceable>pkgdir</replaceable>
&cprompt; createbuildlink &gt;buildlink3.mk</userinput>
</screen>
<sect2 id="anatomy-of-bl3">
<title>Anatomy of a buildlink3.mk file</title>
<para>The following real-life example
<filename>buildlink3.mk</filename> is taken
from <filename>pkgsrc/graphics/tiff</filename>:</para>
<programlisting>
# &#36;NetBSD: buildlink3.mk,v 1.16 2009/03/20 19:24:45 joerg Exp &#36;
BUILDLINK_TREE+= tiff
.if !defined(TIFF_BUILDLINK3_MK)
TIFF_BUILDLINK3_MK:=
BUILDLINK_API_DEPENDS.tiff+= tiff>=3.6.1
BUILDLINK_ABI_DEPENDS.tiff+= tiff>=3.7.2nb1
BUILDLINK_PKGSRCDIR.tiff?= ../../graphics/tiff
.include "../../devel/zlib/buildlink3.mk"
.include "../../graphics/jpeg/buildlink3.mk"
.endif # TIFF_BUILDLINK3_MK
BUILDLINK_TREE+= -tiff
</programlisting>
<para>The header and footer manipulate
<varname>BUILDLINK_TREE</varname>, which is common across all
<filename>buildlink3.mk</filename> files and is used to track
the dependency tree.</para>
<para>The main section is protected from multiple inclusion
and controls how the dependency on <replaceable>pkg</replaceable> is
added. Several important variables are set in the section:</para>
<itemizedlist>
<listitem>
<para><varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
is the actual dependency recorded in the installed
package; this should always be set using
<command>+=</command> to ensure that
we're appending to any pre-existing list of values. This
variable should be set to the first version of the
package that had an backwards-incompatible API change.
</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_PKGSRCDIR.<replaceable>pkg</replaceable></varname>
is the location of the <replaceable>pkg</replaceable>
pkgsrc directory.</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_DEPMETHOD.<replaceable>pkg</replaceable></varname>
(not shown above) controls whether we use
<varname>BUILD_DEPENDS</varname> or
<varname>DEPENDS</varname> to add the dependency on
<replaceable>pkg</replaceable>. The build dependency is
selected by setting
<varname>BUILDLINK_DEPMETHOD.<replaceable>pkg</replaceable></varname>
to <quote>build</quote>. By default, the full dependency is
used.</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_INCDIRS.<replaceable>pkg</replaceable></varname>
and
<varname>BUILDLINK_LIBDIRS.<replaceable>pkg</replaceable></varname>
(not shown above) are lists of subdirectories of
<filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
to add to the header and library search paths. These
default to <quote>include</quote> and <quote>lib</quote>
respectively.</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_CPPFLAGS.<replaceable>pkg</replaceable></varname>
(not shown above) is the list of preprocessor flags to add
to <varname>CPPFLAGS</varname>, which are passed on to the
configure and build phases. The <quote>-I</quote> option
should be avoided and instead be handled using
<varname>BUILDLINK_INCDIRS.<replaceable>pkg</replaceable></varname> as
above.</para>
</listitem>
</itemizedlist>
<para>The following variables are all optionally defined within
this second section (protected against multiple inclusion) and
control which package files are symlinked into
<filename>${BUILDLINK_DIR}</filename> and how their names are
transformed during the symlinking:</para>
<itemizedlist>
<listitem>
<para><varname>BUILDLINK_FILES.<replaceable>pkg</replaceable></varname>
(not shown above) is a shell glob pattern relative to
<filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
to be symlinked into
<filename>${BUILDLINK_DIR}</filename>,
e.g. <filename>include/*.h</filename>.</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_FILES_CMD.<replaceable>pkg</replaceable></varname>
(not shown above) is a shell pipeline that
outputs to stdout a list of files relative to
<filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>.
The resulting files are to be symlinked
into <filename>${BUILDLINK_DIR}</filename>. By default,
this takes the <filename>+CONTENTS</filename> of a
<replaceable>pkg</replaceable> and filters it through
<varname>${BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable>}</varname>.</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable></varname>
(not shown above) is a filter command that filters
<filename>+CONTENTS</filename> input into a list of files
relative to
<filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
on stdout. By default for overwrite packages,
<varname>BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable></varname>
outputs the contents of the <filename>include</filename>
and <filename>lib</filename> directories in the package
<filename>+CONTENTS</filename>, and for pkgviews packages,
it outputs any libtool archives in
<filename>lib</filename> directories.</para>
</listitem>
<listitem>
<para><varname>BUILDLINK_FNAME_TRANSFORM.<replaceable>pkg</replaceable></varname>
(not shown above) is a list of sed arguments used to
transform the name of the source filename into a
destination filename, e.g. <command>-e
"s|/curses.h|/ncurses.h|g"</command>.</para>
</listitem>
</itemizedlist>
<para>This section can additionally include any
<filename>buildlink3.mk</filename> needed for
<replaceable>pkg</replaceable>'s library dependencies.
Including these <filename>buildlink3.mk</filename> files
means that the headers and libraries for these
dependencies are also symlinked into
<filename>${BUILDLINK_DIR}</filename>
whenever the <replaceable>pkg</replaceable>
<filename>buildlink3.mk</filename>
file is included. Dependencies are only added for directly
include <filename>buildlink3.mk</filename> files.</para>
<para>When providing a <filename>buildlink3.mk</filename> and
including other <filename>buildlink3.mk</filename> files in it,
please only add necessary ones, i.e., those whose libraries or
header files are automatically exposed when the package is
use.</para>
<para>In particular, if only an executable
(<filename>bin/foo</filename>) is linked against a library, that
library does not need to be propagated in the
<filename>buildlink3.mk</filename> file.</para>
<para>The following steps should help you decide if a
<filename>buildlink3.mk</filename> file needs to be included:
<itemizedlist>
<listitem><para>Look at the installed header files: What
headers do they include? The packages providing these files
must be buildlinked.</para></listitem>
<listitem><para>Run <filename>ldd</filename> on all installed
libraries and look against what other libraries they link.
Some of the packages providing these probably need to be
buildlinked; however, it's not automatic, since e.g. GTK on
some systems pulls in the X libraries, so they will show up in
the <filename>ldd</filename> output, while on others (like OS
X) it won't. <filename>ldd</filename> output can thus only be
used as a hint.</para></listitem>
</itemizedlist>
</para>
</sect2>
<sect2 id="updating-buildlink-depends">
<title>Updating
<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
and
<varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
in <filename>buildlink3.mk</filename> files</title>
<para>These two variables differ in that one describes source
compatibility (API) and the other binary compatibility (ABI).
The difference is that a change in the API breaks compilation of
programs while changes in the ABI stop compiled programs from
running.</para>
<para>Changes to the
<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
variable in a <filename>buildlink3.mk</filename> file happen
very rarely. One possible reason is that all packages depending
on this already need a newer version. In case it is bumped see
the description below.</para>
<para>The most common example of an ABI change is that the major
version of a shared library is increased. In this case,
<varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
should be adjusted to require at least the new package version.
Then the packages that depend on this package need their
<varname>PKGREVISION</varname>s increased and, if they have
<filename>buildlink3.mk</filename> files, their
<varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
adjusted, too. This is needed so pkgsrc will require the correct
package dependency and not settle for an older one when building
the source.</para>
<para>See <xref linkend="dependencies"/> for
more information about dependencies on other packages,
including the <varname>BUILDLINK_ABI_DEPENDS</varname> and
<varname>ABI_DEPENDS</varname> definitions.</para>
<para>Please take careful consideration before adjusting
<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
or
<varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
as we don't want to cause unneeded package deletions and
rebuilds. In many cases, new versions of packages work just
fine with older dependencies.</para>
<para>Also it is not needed to set
<varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
when it is identical to
<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>. </para>
</sect2>
</sect1>
<sect1 id="writing-builtin.mk">
<title>Writing <filename>builtin.mk</filename> files</title>
<para>Some packages in pkgsrc install headers and libraries that
coincide with headers and libraries present in the base system.
Aside from a <filename>buildlink3.mk</filename> file, these
packages should also include a <filename>builtin.mk</filename>
file that includes the necessary checks to decide whether using
the built-in software or the pkgsrc software is
appropriate.</para>
<para>The only requirements of a builtin.mk file for
<replaceable>pkg</replaceable> are:</para>
<orderedlist>
<listitem>
<para>It should set
<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
to either <quote>yes</quote> or <quote>no</quote>
after it is included.</para>
</listitem>
<listitem>
<para>It should <emphasis>not</emphasis> override any
<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
which is already set before the
<filename>builtin.mk</filename> file is included.</para>
</listitem>
<listitem>
<para>It should be written to allow multiple inclusion. This
is <emphasis>very</emphasis> important and takes careful
attention to <filename>Makefile</filename> coding.</para>
</listitem>
</orderedlist>
<sect2 id="anatomy-of-builtin.mk">
<title>Anatomy of a <filename>builtin.mk</filename> file</title>
<para>The following is the recommended template for builtin.mk
files:</para>
<programlisting>
.if !defined(IS_BUILTIN.foo)
#
# IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
# genuinely exists in the system or not.
#
IS_BUILTIN.foo?= no
# BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
# version can be determined.
#
. if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
BUILTIN_PKG.foo?= foo-1.0
. endif
.endif # IS_BUILTIN.foo
.if !defined(USE_BUILTIN.foo)
USE_BUILTIN.foo?= ${IS_BUILTIN.foo}
. if defined(BUILTIN_PKG.foo)
. for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
. if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
USE_BUILTIN.foo!= \
${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo} \
&amp;&amp; ${ECHO} "yes" || ${ECHO} "no"
. endif
. endfor
. endif
.endif # USE_BUILTIN.foo
CHECK_BUILTIN.foo?= no
.if !empty(CHECK_BUILTIN.foo:M[nN][oO])
#
# Here we place code that depends on whether USE_BUILTIN.foo is set to
# "yes" or "no".
#
.endif # CHECK_BUILTIN.foo
</programlisting>
<para>The first section sets
<varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
depending on if <replaceable>pkg</replaceable> really exists
in the base system. This should not be a base system software
with similar functionality to <replaceable>pkg</replaceable>;
it should only be <quote>yes</quote> if the actual package is
included as part of the base system. This variable is only
used internally within the <filename>builtin.mk</filename>
file.</para>
<para>The second section sets
<varname>BUILTIN_PKG.<replaceable>pkg</replaceable></varname>
to the version of <replaceable>pkg</replaceable> in the base
system if it exists (if
<varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
is <quote>yes</quote>). This variable is only used internally
within the <filename>builtin.mk</filename> file.</para>
<para>The third section sets
<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
and is <emphasis>required</emphasis> in all
<filename>builtin.mk</filename> files. The code in this
section must make the determination whether the built-in
software is adequate to satisfy the dependencies listed in
<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.
This is typically done by comparing
<varname>BUILTIN_PKG.<replaceable>pkg</replaceable></varname>
against each of the dependencies in
<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.
<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
<emphasis>must</emphasis> be set to the correct value by the
end of the <filename>builtin.mk</filename> file. Note that
<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
may be <quote>yes</quote> even if
<varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
is <quote>no</quote> because we may make the determination
that the built-in version of the software is similar enough to
be used as a replacement.</para>
<para>The last section is guarded by
<varname>CHECK_BUILTIN.<replaceable>pkg</replaceable></varname>,
and includes code that uses the value of
<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
set in the previous section. This typically includes, e.g.,
adding additional dependency restrictions and listing additional
files to symlink into <filename>${BUILDLINK_DIR}</filename> (via
<varname>BUILDLINK_FILES.<replaceable>pkg</replaceable></varname>).</para>
</sect2>
<sect2 id="native-or-pkgsrc-preference">
<title>Global preferences for native or pkgsrc software</title>
<para>When building packages, it's possible to choose whether to set
a global preference for using either the built-in (native)
version or the pkgsrc version of software to satisfy a
dependency. This is controlled by setting
<varname>PREFER_PKGSRC</varname> and
<varname>PREFER_NATIVE</varname>. These variables take values
of either <quote>yes</quote>, <quote>no</quote>, or a list of
packages. <varname>PREFER_PKGSRC</varname> tells pkgsrc to
use the pkgsrc versions of software, while
<varname>PREFER_NATIVE</varname> tells pkgsrc to use the
built-in versions. Preferences are determined by the most
specific instance of the package in either
<varname>PREFER_PKGSRC</varname> or
<varname>PREFER_NATIVE</varname>. If a package is specified
in neither or in both variables, then
<varname>PREFER_PKGSRC</varname> has precedence over
<varname>PREFER_NATIVE</varname>. For example, to require
using pkgsrc versions of software for all but the most basic
bits on a NetBSD system, you can set:</para>
<programlisting>
PREFER_PKGSRC= yes
PREFER_NATIVE= getopt skey tcp_wrappers
</programlisting>
<para>A package <emphasis>must</emphasis> have a
<filename>builtin.mk</filename>
file to be listed in <varname>PREFER_NATIVE</varname>,
otherwise it is simply ignored in that list.</para>
</sect2>
</sect1>
</chapter>