- Removed the perl5_dir variable, as it has only been used in one place.
- Added error checking for the installation of the prerequisite packages
after the main build.
"from the beginning".
- Moved the code to get the value of BULK_PREREQ some lines up, as there
are also 19 other variables using the same code. This saves one extra
check if the pkglint directory exists.
- Added BULK_PREREQ to the variables whose value is printed before starting
the actual build.
- When installing prerequisite packages, print a nice sentence instead of
a simple "echo $pkgdir".
The default function prepends the well-known "date, progress, pkgdir" to
each line.
- Check that the environment variable USR_PKGSRC is given and is an
absolute pathname.
- Introduced variables for the various directories. Their names all end in
_dir, and they contain absolute pathnames. Changed all instances of
${USR_PKGSRC}/pkgtools/pkglint to ${pkglint_dir}. Likewise for the other
directories.
- Added more error checking. The bulk build is terminated if the
pkg_install package cannot be updated or the prerequisite packages
cannot be installed.
a package.
This functionality consumed an enormous amount and time, and did not
do much useful work:
* It is not needed when doing a bulk build of the full pkgrsrc tree,
since the pre-requisite packages already have been built and cleaned.
* It does not work when doing a bulk build of a subset of the packages,
since it only cleans the direct dependencies, and not the full list.
- All error messages go to stderr instead of stdout.
- Added variable quoting.
- Reindented some code for better readability.
- Changed "Build Temporary Files" caption to "Some variables used in the
bulk build", which is more appropriate.
- Now it starts with "set -e", preventing unwanted continuing after
errors have occured.
- Added more error checking and handling.
- All variables are properly quoted (if at all possible).
- Merged the two loops for checking packages into one loop.
- Instead of first checking if a directory exists and then changing
into it, the tool now tries to cd, and handles failure.
- Fixed the check for duplicates. Some packages might have been skipped
unintentionally.
build.conf file. The only way to do this before had been to set the
(undocumented) environment variable BULK_BUILD_CONF. This is a
conservative extension.
Make use of find(1) instead of ls(1) to find the broken files.
This prevents the generation of an empty report due to 'too many arguments'
to ls on certain platforms.
with the results to ADMIN.
Now the file will be created if the option is not defined and the
email will be sent too, otherwise only the file will be created.
The file with the results will be created in
${FTP}/pkgsrc-results-${DATE}.txt.
Can't use an undefined value as an ARRAY reference at mk/bulk/post-build line 470.
In the case that no packages were broken, getBroken returns a hash
without the key "topten". When Perl tries to use that undefined value
as an array reference, it croaks.
vulnerable packages are uploaded directly into the vulnerable subdir.
While here:
quote variables
better handling of the temporary directory
remove some backwards compatibility code that's been here long enough
opsys-specific package handling was doing the same as non-opsys
specific, so fold them together.
Written together with dillo.
mount in the sandbox. Then in the individual OS section, turn it back
on if needed. Right now, just turn on NetBSD. At some point we should
probably fill out the other supported OS's or figure out a way to pull
in the information from the makefiles which already define much of whats
needed.
If they existed, it cd to that package database directory.
This broke the code later which assumed to be running from
the pkgsrc directory.
So cd to the USR_PKGSRC after this.
because pkginstall encodes it into its scripts. sigh.
Hack around this. For bulk builds on Interix only, use gsort for "tflat";
use the system-supplied sort for the rest of the build.
on some systems, e.g. solaris, will simply drop long lines. These longs
lines are found in the flattened up and down dependency lists in the bulk
cache files. Instead, use sed to deal with this.
Fixes problems noted in Solaris bulk builds. Patch tested on NetBSD-2.0
and Solaris-2.9
packages they upload (by setting the optional variable MKSUMS=yes) and,
also optionally, PGP signing them (by setting SIGN_AS=username@NetBSD.org,
for example).
binpkg timestamps, and ${PKGDIR} file timestamps, will be used to
determine if a rebuild is necessary. If changed to "no", then rebuilds
will only happen when PKGNAME actually changes.
* Add LINTPKGSRC_DB (and companion USE_LINTPKGSRC_CACHE, default "no") to
make pre-build generate a reusable database with "lintpkgsrc -E"; helps
with nfs-mounted pkgsrc. A future change will allow use of this in the
upload script.
* Add entry for PRUNEPACKAGES in build.conf-example (this was missing).
grepping, run make to get MAINTAINER's value. I've tested this with
a script that tries to "simulate" this one without problems, but I
can't test it "in place".
* Add optional verbosity to show most commands as they are being executed.
* "use strict" and "use warnings".
* Slurp in the varables from build.conf and bmake in one shot (amazing
startup speed boost with nfs pkgsrc); put them in a hash to make
"use strict" much happier with the namespace.
* Fix a bunch of undef-dereference errors evidenced by "use warnings".
* Exclude PKG_DBDIR from leftovers list if it is inside LOCALBASE.
* Convert some <a name="..."/> constructs to <a name="..."></a> to make
non-XHTML-compliant browsers happier.
containing constructs of the form
{perl>=5.8.3nb1,perl-thread>=5.8.3nb1}
is that a Bourne shell "for" interprets this as two separate items in the
list, nuking the {}s. The above will never succeed and thus always cause
Perl-dependent packages to be rebuilt unconditionally.
Just adding more \s isn't enough -- the parsing of the depends list has to
be moved out of the "for" clause. So, echo them one per line and use "read"
to pull them in without allowing the shell to peek at those characters.
in the package directory, and add a link to it from .broken.html at
the point where the build fails. Also adjust the auxilliary scripts
to handle/cleanup .broken.work.html files.
This should enhance the ability of developers to debug broken builds
by providing important information about what is happening as a build
progresses and fails.
as it's only used internally by bsd.prefs.mk.
* Make _PKGSRCDIR a public variable by renaming it to PKGSRCDIR.
Also, generate its value from ${_PKGSRC_TOPDIR} so it's less fragile
than the old method of stripping off the last two components of
${.CURDIR}. PKGSRCDIR may now be used after bsd.prefs.mk is defined.
* Change all references to _PKGSRCDIR to PKGSRCDIR.
this fixes a problem where bulk builds with PKG_SUFX=.tbz would
needlessly, repeatedly rebuild dependency packages because
${PKGNAME}.tgz did not exist.
Portability note: unlike GNU grep, traditional grep did not con-
form to POSIX.2, because traditional grep lacked a -q option and
its -s option behaved like GNU grep's -q option. Shell scripts
intended to be portable to traditional grep should avoid both -q
and -s and should redirect output to /dev/null instead.
And, guess what, that's exactly what we run into when we call IRIX'
egrep(1) with '-q' (which it doesn't know). So let's follow this
suggestion and use neither '-q' nor '-s' and redirect output to
/dev/null instead.
- If it's a full sentence, start capitalize first letter and end with a
period.
- Break comments at ~80 chars so it fits into a terminal.[*]
(* This should be done for the code as well, but then it'll need testing.
I'll do this at another point in time.)
setting SPECIFIC_PKGS=1, this is a convenient way to make sure you have
all required distfiles before starting a build. In particular, you will
get all distfiles for all required dependencies of the explicitly listed
packages. As part of this make the option processing part of this script
a bit more general purpose and add a -h|--help option.
In particular, when SPECIFIC_PKGS is set in /etc/mk.conf, you can now do
sh mk/bulk/build
and have the right thing happen. Only those packages explicitly listed
and those which are depended upon are considered for the build. Other
than the restricted list of packages, the bulk build works the same way
as a full bulk build.
does, so rearrange the arguments so that "-f -" is always at the end.
noted by krister@.
while I'm here, make sure we use the same tar(1) program pkgsrc does.
- move the build summary to the top and include build start/end
times.
- break the report up into three sections for easier navigation:
Broken, Broken dependencies, and Not packaged.
- HTML report now uses a subset of the www.NetBSD.org stylesheet.
- improve some of the text.
some shells can set MAIL to be the mailbox of the user, and
environment variables will override assignments in make when using
conditional assignments.
definition is not intended to be set by the user, and so has a leading
'_', and is undocumented.
If set to "yes", then a "make clean" of the package will not be done,
thereby preserving the working directory. This is useful for
debugging problems in bulk builds.
the "cvs" command inside the su returns an error, that should be ignored.
(Aparently cvs returns "1" even if it's done a successful update but if
there were some files removed on purpose).
1) When checking if any of the required binary pkgs is newer, it's
not good to look into the (already existing) binary pkg, as that
might be unchanged. Instead, look at the DEPENDS.
In the context of the recent jpeg changes, the gd package itself was
not changed, but the DEPENDS were (via buildlink files). Now looking
into the existing gd binary pkg still said it wanted jpeg-6b instead
of the now-wanted jpeg>=6b, which was only available via the DEPENDS.
That's the first chunk of the patch below.
2) While debugging this, I found that the change in rev. 1.48 was
wrong, as can be seen throughout the last bulk build, search for errors
like:
find: "/usr/cvs.local/pkgsrc/packages/i386/All/gd-2.0.15.tgz": No such file or directory
As the whole operation is really on two files (as assured by "pkg_admin
lsbest" for pkg and REFS by definition), the quotes can be ommitted.
Why this wasn't caught when that change was tested is beyond me - maybe
different sh(1) behaviour? (The error happened on 1.6.1_STABLE, see
e.g. http://smaug.fh-regensburg.de/~feyrer/ftp/pub/NetBSD/pkgstat-i386/last/www/p5-Template-Toolkit/.broken.yui.html).
Anyways, that's addressed in the second part of the patch below, too.
3) Use ${FIND} while there.
Introduce sandboxEmptyFiles a list of files to create empty in the
sandbox if they exist on the hosting system. Hence put /var/run in
sandboxEmptyDirs list.
Use $cppprog instead of cp.
in the sandbox if they exist on the hosting system: put /var/spool/mqueue
as it was already created before and add /var/log for now (needed
for various packages, like security/ssh2).
Only create /var/run/utmp(x) if they exist on the hosting system.
XXX this may better be an opsys dependent action.
care not to blow away our bootstrap-pkgsrc stuff in the initial phase.
Also mark devel/bmake and devel/mk-files as broken on non-NetBSD so as not
to blow away our precious files from the bootstrap process in the middle
of a bulk-build. Now let's see if bulk-building works on Linux...
Provided that I copy a working gcc and the binaries from the bootstrap kit
into the sandbox manually, this gets me as far as having a pkgsrc
sandbox that can build pkg_tools/pkg_install.
can stop builds when a dependency is broken, yet continue builds
when a dependency is merely skipped (usually because it duplicates
functionality in the base system). Thus IGNORE_FAIL, the
oft-misunderstood and naught-documented option, goes away.
This addresses many people's complaints on tech-pkg@ and other
lists, as well as PR pkg/18157.
back to the current page, just list it as regular text.
- Put each dependency hyperlink on a separate line in the README.html file.
These were both noted by Thomas Klausner in a private email.
avoid simply appending to it.
- add some additional error checks when using the -r/--restart options to
make sure that the database exists and give a useful error message if it
doesn't
directory of a single package (e.g. graphics/gimp) for which a README.html
is to be generated. This provides the required hook to be able to use this
script to generate a README.html file for a single package as well as
for all packages.
don't exist, exit with a fatal error and don't try and make empty README.html
files.
- fix a bug which caused some duplication in category README.html files.
Problem noted on netbsd-help@ by Nicolas Saurbier ( Nicolas dot Saurbier at biodata dot de)
way than the previous recursive make approach. The 'mkreadme' script is
the top level script. 'mkreadme -h' or 'mkreadme --help' for complete
documentation on its use. Generates README.html files more than two orders
of magnitude faster than the recursive make approach by only calling make(1)
once per pkg.
to ease the use of the sandbox.
$sandbox/sandbox umount
Umount the null mounts from the sandbox
$sandbox/sandbox mount
Mount the null mounts required by the sandbox (useful across reboots, ...).
$sandbox/sandbox whatever
Runs the command passed as arguments, chrooted into the sandbox.
Typical usage would be `$sandbox/sandbox /my/favourite/shell'.
$sandbox/sandbox
Same as `$sandbox/sandbox /bin/sh'
Approved by agc.
This prevent surprise when /etc/localtime -previously copied over in the
sandbox by pax/tar- is a symlink to /usr/share/zoneinfo/whatever.
- Improve somewhat some messages.
- Make sure $sandbox/$packages and $sandbox/$distfiles are created instead of
$packages and $distfiles.
environment, and pass them down to the bulk build. This means that the
/etc/mk.conf (or $MAKECONF) can be shared between ordinary builds and
bulk builds.
environment variable, which contains the name of the make(1) program to
invoke - suggestion by Julien Letessier some time ago, the confusion mine,
since I thought he was referring to something else completely.
Don't rely on there being a POSIX tr(1) in the path by default - test
explicitly for both "yes" and "YES".
Set the default for PRUNEDISTFILES to "no", since we can't assume that
the user wants us to delete something which he may have been keeping
around, and there are other ways of accomplishing this aim (lintpkgsrc
-o, for example).
Clean up some superfluous white space at the end of lines.
OBJMACHINE and OBJHOSTNAME to work. Also don't hardcode /usr/pkg and
/usr/X11R6 for LOCALBASE and X11BASE. Finally, fix a few typos in messages
while here.
The primary purpose is so when a package is broken due to broken depends,
there are links to the build logs of the broken dependencies.
Suggested by Brook Milligan on tech-pkg.
When OBJHOSTNAME is set, the various cache files and log files will include
the hostname in their names. Useful when multiple machines of the same
architecture share pkgsrc.
the cache files used during a bulk pkgsrc build.
- replace the code in the build script that used to create the cache
files with a 'make bulk-cache' call.
