190 lines
8.4 KiB
XML
190 lines
8.4 KiB
XML
<!-- $NetBSD: bulk.xml,v 1.13 2014/05/31 20:49:16 asau Exp $ -->
|
|
|
|
<chapter id="bulk">
|
|
<title>Creating binary packages for everything in pkgsrc (bulk
|
|
builds)</title>
|
|
|
|
<para>When you have multiple machines that should run the same packages,
|
|
it is wasted time if they all build their packages themselves from
|
|
source. There is a ways of getting a set of binary packages:
|
|
The bulk build system, or pbulk ("p" stands for "parallel).
|
|
This chapter describes how to set it up so that the packages
|
|
are most likely to be usable later.</para>
|
|
|
|
<sect1 id="bulk.pre">
|
|
<title>Think first, build later</title>
|
|
|
|
<para>Since a bulk build takes several days or even weeks to finish, you
|
|
should think about the setup before you start everything. Pay attention
|
|
to at least the following points:</para>
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para>If you want to upload the binary packages to
|
|
ftp.NetBSD.org, make sure the setup complies to the requirements for binary
|
|
packages:</para>
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para>To end up on ftp.NetBSD.org, the packages must be built
|
|
by a NetBSD developer on a trusted machine (that is, where you and only
|
|
you have root access).</para></listitem>
|
|
|
|
<listitem><para>Packages on ftp.NetBSD.org should only be created from
|
|
the stable branches (like 2009Q1), so that users browsing the available
|
|
collections can see at a glance how old the packages
|
|
are.</para></listitem>
|
|
|
|
<listitem><para>The packages must be built as root, since some packages
|
|
require set-uid binaries at runtime, and creating those packages as
|
|
unprivileged user doesn't work well at the moment.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
</listitem>
|
|
|
|
<listitem><para>Make sure that the bulk build cannot break anything in
|
|
your system. Most bulk builds run as root, so they should be run at least
|
|
in a chroot environment or something even more restrictive, depending on
|
|
what the operating system provides. There have been numerous cases where
|
|
certain packages tried to install files outside the
|
|
<filename>LOCALBASE</filename> or wanted to edit some files in
|
|
<filename>/etc</filename>. Furthermore, the bulk builds install and
|
|
deinstall packages in <filename>/usr/pkg</filename> (or whatever
|
|
<filename>LOCALBASE</filename> is) during their operation, so be sure
|
|
that you don't need any package during the build.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
</sect1>
|
|
|
|
<sect1 id="bulk.req">
|
|
<title>Requirements of a bulk build</title>
|
|
|
|
<para>A complete bulk build requires lots of disk space. Some of the
|
|
disk space can be read-only, some other must be writable. Some can be on
|
|
remote filesystems (such as NFS) and some should be local. Some can be
|
|
temporary filesystems, others must survive a sudden reboot.</para>
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para>40 GB for the distfiles (read-write, remote, temporary)</para></listitem>
|
|
|
|
<listitem><para>30 GB for the binary packages (read-write, remote, permanent)</para></listitem>
|
|
|
|
<listitem><para>1 GB for the pkgsrc tree (read-only, remote, permanent)</para></listitem>
|
|
|
|
<listitem><para>5 GB for <filename>LOCALBASE</filename> (read-write, local, temporary for pbulk, permanent for old-bulk)</para></listitem>
|
|
|
|
<listitem><para>10 GB for the log files (read-write, remote, permanent)</para></listitem>
|
|
|
|
<listitem><para>5 GB for temporary files (read-write, local, temporary)</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="bulk.pbulk">
|
|
<title>Running a pbulk-style bulk build</title>
|
|
|
|
<para>Running a pbulk-style bulk build works roughly as follows:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem><para>First, build the pbulk infrastructure in a fresh pkgsrc location.</para></listitem>
|
|
<listitem><para>Then, build each of the packages from a clean installation directory using the infrastructure.</para></listitem>
|
|
</itemizedlist>
|
|
|
|
<sect2 id="bulk.pbulk.prepare">
|
|
<title>Preparation</title>
|
|
|
|
<para>First, you need to create a pkgsrc installation for the pbulk infrastructure. No matter on which platform you are (even on NetBSD), you should bootstrap into its own directory. Let's take the directory <filename>/usr/pbulk</filename> or <filename>$HOME/pbulk</filename> for it. This installation will be bootstrapped and all the tools that are required for the bulk build will be installed there.</para>
|
|
|
|
<screen>
|
|
$ <userinput>cd /usr/pkgsrc</userinput>
|
|
$ <userinput>./bootstrap/bootstrap --prefix=/usr/pbulk --varbase=/usr/pbulk/var --workdir=/tmp/pbulk-bootstrap</userinput>
|
|
$ <userinput>rm -rf /tmp/pbulk-bootstrap</userinput>
|
|
</screen>
|
|
|
|
<para>Now the basic environment for the pbulk infrastructure is installed. The specific tools are still missing. This is a good time to edit the pkgsrc configuration file <filename>/usr/pbulk/etc/mk.conf</filename> to fit your needs. Typical things you might set now are:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem><para><literal><varname>PKG_DEVELOPER</varname>=yes</literal>, to enable many consistency checks,</para></listitem>
|
|
<listitem><para><literal><varname>WRKOBJDIR</varname>=/tmp/pbulk-outer</literal>, to keep <filename>/usr/pkgsrc</filename> free from any modifications,</para></listitem>
|
|
<listitem><para><literal><varname>DISTDIR</varname>=/distfiles</literal>, to have only one directory in which all distfiles (for the infrastructure and for the actual packages) are downloaded,</para></listitem>
|
|
<listitem><para><literal><varname>ACCEPTABLE_LICENSES</varname>+=...</literal>, to select some licenses additional to the usual Free/Open Source licenses that are acceptable to you,</para></listitem>
|
|
<listitem><para><literal><varname>SKIP_LICENSE_CHECK</varname>=yes</literal>, to bypass the license checks.</para></listitem>
|
|
</itemizedlist>
|
|
|
|
<para>Now you are ready to build the rest of the pbulk infrastructure.</para>
|
|
|
|
<screen>
|
|
$ <userinput>cd pkgtools/pbulk</userinput>
|
|
$ <userinput>/usr/pbulk/bin/bmake install</userinput>
|
|
$ <userinput>rm -rf /tmp/pbulk-outer</userinput>
|
|
</screen>
|
|
|
|
<para>Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.</para>
|
|
</sect2>
|
|
|
|
<sect2 id="bulk.pbulk.conf">
|
|
<title>Configuration</title>
|
|
|
|
<para>TODO; see pkgsrc/doc/HOWTO-pbulk for more information.</para>
|
|
|
|
<para>TODO: continue writing</para>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="creating-cdroms">
|
|
<title>Creating a multiple CD-ROM packages collection</title>
|
|
|
|
<para>After your pkgsrc bulk-build has completed, you may wish to
|
|
create a CD-ROM set of the resulting binary packages to assist
|
|
in installing packages on other machines. The
|
|
<filename role="pkg">pkgtools/cdpack</filename> package provides
|
|
a simple tool for creating the ISO 9660 images.
|
|
<command>cdpack</command> arranges the packages on the CD-ROMs in a
|
|
way that keeps all the dependencies for a given package on the same
|
|
CD as that package.</para>
|
|
|
|
<sect2 id="cdpack-example">
|
|
<title>Example of cdpack</title>
|
|
|
|
<para>Complete documentation for cdpack is found in the cdpack(1)
|
|
man page. The following short example assumes that the binary
|
|
packages are left in
|
|
<filename>/usr/pkgsrc/packages/All</filename> and that
|
|
sufficient disk space exists in <filename>/u2</filename> to
|
|
hold the ISO 9660 images.</para>
|
|
|
|
<screen>
|
|
&rprompt; <userinput>mkdir /u2/images</userinput>
|
|
&rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
|
|
&rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput>
|
|
</screen>
|
|
|
|
<para>If you wish to include a common set of files
|
|
(<filename>COPYRIGHT</filename>, <filename>README</filename>,
|
|
etc.) on each CD in the collection, then you need to create a
|
|
directory which contains these files. e.g.</para>
|
|
|
|
<screen>
|
|
&rprompt; <userinput>mkdir /tmp/common</userinput>
|
|
&rprompt; <userinput>echo "This is a README" > /tmp/common/README</userinput>
|
|
&rprompt; <userinput>echo "Another file" > /tmp/common/COPYING</userinput>
|
|
&rprompt; <userinput>mkdir /tmp/common/bin</userinput>
|
|
&rprompt; <userinput>echo "#!/bin/sh" > /tmp/common/bin/myscript</userinput>
|
|
&rprompt; <userinput>echo "echo Hello world" >> /tmp/common/bin/myscript</userinput>
|
|
&rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput>
|
|
</screen>
|
|
|
|
<para>Now create the images:</para>
|
|
|
|
<screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>
|
|
|
|
<para>Each image will contain <filename>README</filename>,
|
|
<filename>COPYING</filename>, and <filename>bin/myscript</filename>
|
|
in their root directories.</para>
|
|
</sect2>
|
|
</sect1>
|
|
</chapter>
|