to ease the use of the sandbox.
$sandbox/sandbox umount
Umount the null mounts from the sandbox
$sandbox/sandbox mount
Mount the null mounts required by the sandbox (useful across reboots, ...).
$sandbox/sandbox whatever
Runs the command passed as arguments, chrooted into the sandbox.
Typical usage would be `$sandbox/sandbox /my/favourite/shell'.
$sandbox/sandbox
Same as `$sandbox/sandbox /bin/sh'
Approved by agc.
This prevent surprise when /etc/localtime -previously copied over in the
sandbox by pax/tar- is a symlink to /usr/share/zoneinfo/whatever.
- Improve somewhat some messages.
- Make sure $sandbox/$packages and $sandbox/$distfiles are created instead of
$packages and $distfiles.
environment, and pass them down to the bulk build. This means that the
/etc/mk.conf (or $MAKECONF) can be shared between ordinary builds and
bulk builds.
environment variable, which contains the name of the make(1) program to
invoke - suggestion by Julien Letessier some time ago, the confusion mine,
since I thought he was referring to something else completely.
Don't rely on there being a POSIX tr(1) in the path by default - test
explicitly for both "yes" and "YES".
Set the default for PRUNEDISTFILES to "no", since we can't assume that
the user wants us to delete something which he may have been keeping
around, and there are other ways of accomplishing this aim (lintpkgsrc
-o, for example).
Clean up some superfluous white space at the end of lines.
OBJMACHINE and OBJHOSTNAME to work. Also don't hardcode /usr/pkg and
/usr/X11R6 for LOCALBASE and X11BASE. Finally, fix a few typos in messages
while here.
The primary purpose is so when a package is broken due to broken depends,
there are links to the build logs of the broken dependencies.
Suggested by Brook Milligan on tech-pkg.
When OBJHOSTNAME is set, the various cache files and log files will include
the hostname in their names. Useful when multiple machines of the same
architecture share pkgsrc.
the cache files used during a bulk pkgsrc build.
- replace the code in the build script that used to create the cache
files with a 'make bulk-cache' call.
references of the pkglint package.
_PKGSRCDIR is an internal definition in bsd.pkg.mk, and a few packages
which would like to refer to other packages in the build tree. It should
not be set by users, but neither should it stop a user from building a
package if it is defined, so make it obvious that this is the case.
whitespeace, followed by it's required pkgs. Make sure to look for that
whitespace, to not catch any wrong positives, e.g.
egrep "x11/kdebase.*" $DEPENDSFILE
will catch depends for both KDE1 and KDE2 and do weird things then (among
other things, think that kdelibs-2.2.1 is ok as a dependency when building
in x11/kdebase, which is certainly not!)
Eight broken pkgs less in the bulk builds by adding a single space!
(Now if adding some more would help further... :-)
matches one of the DEPENDS, the bulk-build system looked up the
package in the INDEXFILE, then used the pkgdir returned to check
against the DEPENDS. If for some reason the package was not found
in the index file the package was left installed as 'requires
installed package'.
GROUP_SPECIFIC_PKGS into a top pkgsrc level call to make. The new way,
uses a shell 'for pkgdir in ....' and then calls a make in each pkg
directory.
This does 2 things. The first thing is that a 'build restart' starts
up _much_ faster because instead of calling 'make' for each package we've
already built and finding out they're already build, we just grep through
the list of already built pkgs. The second item (a side benefit) is that
now a progress meter is simple.
- add a progress meter to the build.
- add even more error checking.
- if something fails while extracting the DEPENDS for a particular package,
mark that package as broken and drop the DEPENDS info for it. This allows
the build to continue and properly marks the package as broken.
- if something fails while extracting the PKGNAME, also mark the pkg as broken.
- be more careful about seeing if directories exist before cd'ing to them.
Lack of such a test has caused problems elsewhere, so hopefully this will
make the system more robust.
to remove it, then try again with 'pkg_delete -r'. This can happen when
some other out date pacakges are installed and we're trying to bulk build
a single package.
them in the right place. For example, a collection of binary packages can
now put uploaded with most going to the major.minor directory and the
OSVERSION_SPECIFIC ones going to the major.minor.patch directory.
still todo: the destination settings should probably be moved out of
build.conf so the same build.conf can be used by multiple machines of
different MACHINE_ARCH's at the same time. Perhaps these should be
a command line option to 'upload' instead.
Nothing fancy, probably needs adjusting to other people's environments.
XXX the list of things that need to be setup in the sandbox should
XXX probably be moved to Packages.txt...
- use the PKGPATH variable instead of getting it indirectly
- don't use `foo` when we only want the return value, not the output.
Also, in the non-cache case, we need to remove un-needed packages as noted
in the most recent non-cached bulk build.
Patches provided by Hubert and slightly modified by me.
then run it at the end of pre-build. This lets users do things like:
echo "I do not have enough disk space to build this pig." \
> games/crafty-book-enormous/$BROKENF
even if they are not enabled in the category makefiles. This catches,
in particular, pkgtools/xpkgwedge which was missing before.
Thanks to Christoph Badura for noting this.
of packages which must stay installed during the build, but are not pulled
in by the DEPENDS for each package. Currently, BULK_PREREQ will always include
pkgtools/digest. At this time, the primary use will be to add
BULK_PREREQ+=pkgtools/xpkgwedge
in /etc/mk.conf to do an xpkgwedge'd bulk build. It is up to the user to make
sure that the list of packages in BULK_PREREQ is a flattened list (ie all the
DEPENDS are listed too). Again, at this time, xpkgwedge is really the only
package that should be added to the list.
-add an ADMINSIG build.conf variable. This is the signature at the end of the
email report. Maybe now I'll quit forwarding reports signed as "-Hubert".
-while here, eliminate grep|awk lines and `grep >/dev/null` replacing them
with pure awk and grep -q.
the old behaviour. This is done because unless we're in the middle of a
bulk build, we don't really know that the cache files are up to date. These
are fairly time consuming (relative to a single smallish package build) to
generate and depend on all of the pkgsrc makefiles. During a bulk build, the
overhead is far outweighed by the savings.
In particular, 'make bulk-install' will now work correctly outside of a bulk
build (useful for debugging broken packages).
Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
fail as a result. Then report this number in the generated email and
html summary. The goal is to help the pkgsrc crew focus our efforts
on the broken packages which have the largest impact. Thanks to
Christoph Badura for suggesting that I do this and Hubert Feyrer who
suggested an easier to read output format.
over several bulk builds on an alpha:
- At the start of a bulk build, 4 files are created to allow fast lookup of
various dependency tree things. These files are
.index == maps package name (foo-2.3) to directory (bar/foo)
.dependstree == contains the complete pkgsrc dependency tree in a tsort(1)
compatible format.
.depends == contains 1 line per package and lists all the build depends
for the package.
.supports == contains 1 line per package and lists all packages which
depend on this package.
- When a package fails to build, the list of all packages which depend upon
it is read from .supports. Each of those packages is immediately marked
as broken. This prevents us from trying to build those packages which can
have significant overhead if lots of other depends are installed before the
system notices the broken one.
In addition, the post-build postprocessing will now indicate that a package
is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which
is broken because it depends on a failed package. This assists in determining
where to focus our efforts in fixing broken packages.
- In the old approach, all packages are removed after each one is built. The
purpose was to a) conserve disk space, b) verify that all required dependencies
are in fact listed, and c) prevent conflicts. The problem was that often times
several packages in a row have similar depends. For example we might want to
build several perl packages in a row. In the old approach, we would install perl,
build the pkg, deinstall perl and continue with the next one.
In the new approach, when it is decided that a pkg is out of date and should be
rebuilt, the .depends file is used to obtain a list of pkgs we require. Then
and pkgs which are no longer needed are removed. This helps to minimize the total
number of pkg_add/pkg_delete's in a bulk build.
- Since the order of the build is controlled by the depends tree, all depends for a
given package will have been built by the time the pkg in question needs them.
If any of the depends failed to build, then the pkg which needs the failed one will
have been marked as broken. Given this, the complete depends list for a pkg is
read from .depends and the depends installed via pkg_add rather than relying on
recursive make calls to install the depends.
- while here, fix a few minor bugs
x - don't leave .make log files around when the build succeeds
x - make sure we refer to the correct report file in the email
x - use '.order' for the build order file instead of '.l'
x - use 'grep -c' instead of 'grep | wc -l'
been deleted.
- when removing the '.start' file, don't remove them all, only the one we
created. Prevents clashes when pkgsrc is shared among multiple machines.
tflat is a small awk script used to flatten a dependency tree. It can
process a tsort(1) compatible input file and produce a flattened
list showing all packages which depend on a particular package or
all packages which are depended upon by a package. This is used by
the bulk pkgsrc build system.
Written by Dan McMahill after careful study of a perl program that
does the same function in a nicer way written by Thomas Klausner
<wiz@netbsd.org>. The reason for rewriting it without perl was to
make it work with only in-tree utilities.
printindex is a small shell script run from /usr/pkgsrc. It generates
an index file which associates package directory (foo/bar) to package name
(bar-3.2) for the entire pkgsrc tree. The index file is useful for
processing of some of the dependencies during a bulk build.
Written by Dan McMahill using a little bit of the code from printdepends.
of the top level build script and into the {pre,post}-build scripts. This
ensures that either of those scripts may be run directly and not rely on
a variable being set by the top level script.
Thanks to Hubert Feyrer for pointing out the problem in a private email.
These changes do not require any user changes to the build.conf file and
should be transparent to the user.
- set BROKENF and BLDLOG in the 'build' script and pass those variables
down to {pre,post}-build in the environment to make sure we only set them
in one place. The values are determined by a
make show-var VARNAME=BROKENFILE
This causes the default (set in bsd.bulk-pkg.mk) or the user overridden
value from /etc/mk.conf or the environment to be correctly determined. This
is more robust that relying on a build.conf setting which may or may not be
correctly set.
- have pre-build only clean up BROKENF and BLDLOG files instead of
.broken* and .make*
This avoids conflicts when pkgsrc is shared among different machines.
pkgsrc is shared via nfs where multiple machines may want to run the script
at the same time. Also avoid conflicts if multiple copies are run on the
same machine at once.
hard coding it.
- explicitly only add $BROKENFILE's to the output rather than .broken.* This makes
things work correctly when /usr/pkgsrc is shared among several machines which might
use .broken.`hostname` for its broken file logs.
