that the bulk builds would only work properly with the audit-packages
package installed at all times.
I am obviously hideously mistaken in this.
By popular request, revert the change to add audit-packages, since I
am told this is the incorrect way of doing this.
cd's into pkgtools/pkglint and does a "make fetch".
Since the audit-packages check was introduced to bsd.pkg.mk, this
"make fetch" will fail with an error message which looks like:
===> do-fetch [pkglint-4.42.1] ===> Checking for vulnerabilities in pkglint-4.42.1
===> check-vulnerable [pkglint-4.42.1] ===> *** The audit-packages package must be at least version 1.40
===> check-vulnerable [pkglint-4.42.1] ===> *** Please install the security/audit-packages package and run
===> check-vulnerable [pkglint-4.42.1] ===> *** '/usr/pkg/sbin/download-vulnerability-list'.
*** Error code 1
since there is no audit-packages package installed at this point (we
are still doing the pre-installation checks).
So call "make fetch" with an additional SKIP_AUDIT_PACKAGES=yes
argument at this point.
However, this does not help people who are doing bulk builds
in sandboxes via read-only nullfs mounts.
A correct, long-term solution to this needs to be found.
formline(). This prevents long package names from being trimmed.
- Removed the "-p e" command line option from the pax(1) command line.
This makes it possible to run mk/bulk/post-build as different user
from the one who did the actual build.
- Added section headings to make reading the code more efficient.
- Added bp_die() for uniform error messages.
- Added sanity checks:
- $BULK_BUILD_CONF must be a regular file.
- Protect against spurious output from the bulk.conf file.
- Check that configuration variables are defined, non-empty and
do not contain newlines.
- Properly extract MAKECONF from the bulk.conf file when getting the
mk.conf variables.
- Protect against spurious output from BMAKE show-vars.
- Fixed the use of undefined values ($startdate).
- Changed some calls to my_system() to a more secure form.
- Check if the calls to chdir() actually work.
- Look for the BROKENFILEs in BULKFILESDIR instead of USR_PKGSRC.
- Changed postfix conditions to prefix conditions. Example:
do_foo() if $bar; ==> if ($bar) { do_foo(); }
nullfs mounts are used (on NetBSD).
Only attempt to install new pkg_install tools if the bulk build is not
taking place in a sandbox.
This is a quick and dirty hack to allow sandboxed bulk builds (the majority
of cases, I suspect) to operate properly once again.
- Added a bunch of internal variables that determine the correct places
for the build log files of the current package.
- Added section headings.
- Moved a sanity check out of the configuration section to not confuse
the reader.
- mk/bulk/print{depends,index} are called with two arguments, which allows
separation of BULKFILESDIR and PKGSRCDIR.
- Replaced uses of BUILDLOG, BROKENFILE and BROKENWRKLOG with their
internal counterparts, which use ${BULKFILESDIR} instead of ${PKGSRCDIR}.
- The main .make file is stored in ${BULKFILESDIR} instead of ${PKGSRCDIR}.
- The PKGLIST variable from the configuration file is passed to the
bulk-cache command in the environment. There are very few variables left
that have to be exported via export_config_vars(). Maybe that function
can be avoided alltogether.
pkgsrc packages, collecting different information.
- BMAKE is taken from the environment instead of querying `uname -s`.
- package-specific log files can be saved in a directory tree different
from PKGSRCDIR.
- Added sanity checks for BMAKE, bulkfilesdir and brokenfile.
- The .pkglist cache is saved in ${BULKFILESDIR} instead of ${PKGSRCDIR}.
- Reformatted the diagnostics.
- Changed the implementation how package-specific log files are generated.
No user-visible changes.
- Stricter preconditions:
- bulkfilesdir must be an absolute path.
- brokenfile must not contain slashes.
- The .pkglist file is taken from BULKFILESDIR, not from PKGSRCDIR.
- Error messages use the prefix "[printindex]" instead of "===>", as the
latter is already reserved for pkgsrc. Everything else would be confusing.
- If a "cd" into a package directory fails, an error message is written to
${BULKFILESDIR}/${BROKENFILE}.
- Renamed the second parameter from "brokenbasedir" to "bulkfilesdir". The
latter name is already used in bsd.bulk-pkg.mk for the same purpose.
- Renamed "cwd" to "pkgsrcdir" for the same reason.
Whoever had written such unusual code should have left a note in the
printindex script that the trailing white-space was required. It is such
an uncommon assumption that I treated it as a typo and removed it. Now
it's fixed. I also replaced the use of grep|awk with a simple awk, as
grep would have interpreted regular expressions in the package name.
In general, please think more about _appropriate_ ways to handle data.
grep(1) is not a tool for searching strings. It's a tool for searching
regular expressions.
if present. By design, it is not cleaned at the start of a bulk build; it
is intended to allow a bulk build admin to mark a time-consuming package as
not buildable, but which shouldn't be marked permanently as such in the
package's Makefile.
an empty MAKECONF...
$ echo ".include <bsd.own.mk>" > Makefile
$ env MAKECONF= make
make: "/usr/share/mk/" line 1: Need an operator
make: "/usr/share/mk/" line 2: Need an operator
...
- Added file post-build-conf, which provides functions to print and
export the config variables from build.conf.
- All config variables are shown at the start of a bulk build.
- Added a configuration variable MAKECONF with the obvious meaning.
- When printindex is called with two arguments, the second argument
specifies the base directory of the <brokenfile>. The complete path is
then <brokenbasedir>/${pkgdir}/<brokenfile>. This is useful for creating
the log files in a directory outside the pkgsrc root.
- Removed the perl5_dir variable, as it has only been used in one place.
- Added error checking for the installation of the prerequisite packages
after the main build.
"from the beginning".
- Moved the code to get the value of BULK_PREREQ some lines up, as there
are also 19 other variables using the same code. This saves one extra
check if the pkglint directory exists.
- Added BULK_PREREQ to the variables whose value is printed before starting
the actual build.
- When installing prerequisite packages, print a nice sentence instead of
a simple "echo $pkgdir".
The default function prepends the well-known "date, progress, pkgdir" to
each line.
- Check that the environment variable USR_PKGSRC is given and is an
absolute pathname.
- Introduced variables for the various directories. Their names all end in
_dir, and they contain absolute pathnames. Changed all instances of
${USR_PKGSRC}/pkgtools/pkglint to ${pkglint_dir}. Likewise for the other
directories.
- Added more error checking. The bulk build is terminated if the
pkg_install package cannot be updated or the prerequisite packages
cannot be installed.
a package.
This functionality consumed an enormous amount and time, and did not
do much useful work:
* It is not needed when doing a bulk build of the full pkgrsrc tree,
since the pre-requisite packages already have been built and cleaned.
* It does not work when doing a bulk build of a subset of the packages,
since it only cleans the direct dependencies, and not the full list.
- All error messages go to stderr instead of stdout.
- Added variable quoting.
- Reindented some code for better readability.
- Changed "Build Temporary Files" caption to "Some variables used in the
bulk build", which is more appropriate.
- Now it starts with "set -e", preventing unwanted continuing after
errors have occured.
- Added more error checking and handling.
- All variables are properly quoted (if at all possible).
- Merged the two loops for checking packages into one loop.
- Instead of first checking if a directory exists and then changing
into it, the tool now tries to cd, and handles failure.
- Fixed the check for duplicates. Some packages might have been skipped
unintentionally.
build.conf file. The only way to do this before had been to set the
(undocumented) environment variable BULK_BUILD_CONF. This is a
conservative extension.
Make use of find(1) instead of ls(1) to find the broken files.
This prevents the generation of an empty report due to 'too many arguments'
to ls on certain platforms.
with the results to ADMIN.
Now the file will be created if the option is not defined and the
email will be sent too, otherwise only the file will be created.
The file with the results will be created in
${FTP}/pkgsrc-results-${DATE}.txt.
Can't use an undefined value as an ARRAY reference at mk/bulk/post-build line 470.
In the case that no packages were broken, getBroken returns a hash
without the key "topten". When Perl tries to use that undefined value
as an array reference, it croaks.
vulnerable packages are uploaded directly into the vulnerable subdir.
While here:
quote variables
better handling of the temporary directory
remove some backwards compatibility code that's been here long enough
opsys-specific package handling was doing the same as non-opsys
specific, so fold them together.
Written together with dillo.
mount in the sandbox. Then in the individual OS section, turn it back
on if needed. Right now, just turn on NetBSD. At some point we should
probably fill out the other supported OS's or figure out a way to pull
in the information from the makefiles which already define much of whats
needed.
If they existed, it cd to that package database directory.
This broke the code later which assumed to be running from
the pkgsrc directory.
So cd to the USR_PKGSRC after this.
because pkginstall encodes it into its scripts. sigh.
Hack around this. For bulk builds on Interix only, use gsort for "tflat";
use the system-supplied sort for the rest of the build.
on some systems, e.g. solaris, will simply drop long lines. These longs
lines are found in the flattened up and down dependency lists in the bulk
cache files. Instead, use sed to deal with this.
Fixes problems noted in Solaris bulk builds. Patch tested on NetBSD-2.0
and Solaris-2.9
packages they upload (by setting the optional variable MKSUMS=yes) and,
also optionally, PGP signing them (by setting SIGN_AS=username@NetBSD.org,
for example).
binpkg timestamps, and ${PKGDIR} file timestamps, will be used to
determine if a rebuild is necessary. If changed to "no", then rebuilds
will only happen when PKGNAME actually changes.
* Add LINTPKGSRC_DB (and companion USE_LINTPKGSRC_CACHE, default "no") to
make pre-build generate a reusable database with "lintpkgsrc -E"; helps
with nfs-mounted pkgsrc. A future change will allow use of this in the
upload script.
* Add entry for PRUNEPACKAGES in build.conf-example (this was missing).
grepping, run make to get MAINTAINER's value. I've tested this with
a script that tries to "simulate" this one without problems, but I
can't test it "in place".
* Add optional verbosity to show most commands as they are being executed.
* "use strict" and "use warnings".
* Slurp in the varables from build.conf and bmake in one shot (amazing
startup speed boost with nfs pkgsrc); put them in a hash to make
"use strict" much happier with the namespace.
* Fix a bunch of undef-dereference errors evidenced by "use warnings".
* Exclude PKG_DBDIR from leftovers list if it is inside LOCALBASE.
* Convert some <a name="..."/> constructs to <a name="..."></a> to make
non-XHTML-compliant browsers happier.
containing constructs of the form
{perl>=5.8.3nb1,perl-thread>=5.8.3nb1}
is that a Bourne shell "for" interprets this as two separate items in the
list, nuking the {}s. The above will never succeed and thus always cause
Perl-dependent packages to be rebuilt unconditionally.
Just adding more \s isn't enough -- the parsing of the depends list has to
be moved out of the "for" clause. So, echo them one per line and use "read"
to pull them in without allowing the shell to peek at those characters.
in the package directory, and add a link to it from .broken.html at
the point where the build fails. Also adjust the auxilliary scripts
to handle/cleanup .broken.work.html files.
This should enhance the ability of developers to debug broken builds
by providing important information about what is happening as a build
progresses and fails.
as it's only used internally by bsd.prefs.mk.
* Make _PKGSRCDIR a public variable by renaming it to PKGSRCDIR.
Also, generate its value from ${_PKGSRC_TOPDIR} so it's less fragile
than the old method of stripping off the last two components of
${.CURDIR}. PKGSRCDIR may now be used after bsd.prefs.mk is defined.
* Change all references to _PKGSRCDIR to PKGSRCDIR.
this fixes a problem where bulk builds with PKG_SUFX=.tbz would
needlessly, repeatedly rebuild dependency packages because
${PKGNAME}.tgz did not exist.
Portability note: unlike GNU grep, traditional grep did not con-
form to POSIX.2, because traditional grep lacked a -q option and
its -s option behaved like GNU grep's -q option. Shell scripts
intended to be portable to traditional grep should avoid both -q
and -s and should redirect output to /dev/null instead.
And, guess what, that's exactly what we run into when we call IRIX'
egrep(1) with '-q' (which it doesn't know). So let's follow this
suggestion and use neither '-q' nor '-s' and redirect output to
/dev/null instead.
- If it's a full sentence, start capitalize first letter and end with a
period.
- Break comments at ~80 chars so it fits into a terminal.[*]
(* This should be done for the code as well, but then it'll need testing.
I'll do this at another point in time.)
setting SPECIFIC_PKGS=1, this is a convenient way to make sure you have
all required distfiles before starting a build. In particular, you will
get all distfiles for all required dependencies of the explicitly listed
packages. As part of this make the option processing part of this script
a bit more general purpose and add a -h|--help option.
In particular, when SPECIFIC_PKGS is set in /etc/mk.conf, you can now do
sh mk/bulk/build
and have the right thing happen. Only those packages explicitly listed
and those which are depended upon are considered for the build. Other
than the restricted list of packages, the bulk build works the same way
as a full bulk build.
does, so rearrange the arguments so that "-f -" is always at the end.
noted by krister@.
while I'm here, make sure we use the same tar(1) program pkgsrc does.
- move the build summary to the top and include build start/end
times.
- break the report up into three sections for easier navigation:
Broken, Broken dependencies, and Not packaged.
- HTML report now uses a subset of the www.NetBSD.org stylesheet.
- improve some of the text.
