if it occurs as the condition of a "while" or "if", or in combination
with && or ||. This, of course, introduces a race condition if someone
removes the directory between the two "cd"s. The result would be that
the bulk build terminates, but that's not a big issue, as in this case
there must be something much more severely broken.
of successful builds are not removed, only renamed. This variable will
be useful for examining the warnings that appear during the bulk builds
but are thrown away because the package builds fine. This feature is
experimental and thus not enabled by default.
call to the mk/bulk/build script. This is to allow restarting the bulk
build without getting confused by the various timestamps that occur
during a bulk build. It also fixes some problems where the text/plain
report had ended up in a directory different from the other files.
- Changed the default filename for the text/plain report from
"pkgsrc-results-${BUILDDATE}.txt" to simply "report.txt", as the
${BUILDDATE} variable didn't give a hint to when the bulk build had been
started, which is often more important than when the bulk build stopped.
- Changed the way to specify the directory and the URL where the bulk build
reports are made available. See
http://mail-index.netbsd.org/tech-pkg/2005/12/05/0002.html
for details. Specifying the directories using the FTPHOST and FTPURL
variables will be possible for some months, after which it support for
it will be dropped.
1. Ensure that it exists before trying to use anything inside of it.
2. Prefix files correctly with BULKFILESDIR to match the rest of
the tree.
3. Most importantly, create the .broken.html files for bootstrap tools
in BULKFILESDIR, otherwise e.g. bmake would be removed in the middle of
a bulk build on non-NetBSD.
Discussed-with: rillig@
PKG_DELETE and PKG_INFO. This is more straigt-forward and adds the
proper `-K' command line option to the calls, which in turn fixes bulk
builds that don't use the default PKG_DBDIR.
regenerate. This change makes printdepends and printindex more
independent, because before this change, printindex could use the
cache provided by printdepends, but not the other way round.
- Fixed the section "Sanity checks, part 2" by allowing newline
characters in the list of packages from which the index is built.
This bug just hasn't been noticed because the list taken from the
cache did not have newlines at all.
The '-' character is not included in the $allowed variable because it
is handled specially and differently by sh(1) and tr(1).
- Updated the printindex man page.
2. Reindented two "if" statements that looked like this:
if long_long_command; then :; else
else_commands
fi
By just scanning the beginnings of the lines, which is a common way of
reading source code, one may easily miss the "hidden" "then" branch at
the end of the line. I have changed it to:
if long_long_command; then :;
else
else_commands
fi
mostly a cosmetic change. The motives for this change are that the HTML
files only consisted of "<pre>", the properly quoted text/plain contents
and "</pre>". There has been no documentation of reasons for encoding plain
text in HTML unnecessarily, so this change will produce text/plain files
instead.
Another change is that instead of appending to the BROKENWRKSRC, the file is
overwritten each time. In a single bulk build this will not change anything.
Only when the package is built multiple times in a single bulk build, the
behavior will differ.
Ok'ed by agc.
point where it is actually used, which is when mk/bulk/pre-build.local
is executed for allowing user-specific modifications before the build
begins. That way it's easier to see why this command is necessary at
all.
create log files in it.
- Fail early if the ORDERFILE does not exist. (As this program is not in
"set -e" mode, it had continued without error checking, which resulted
in _many_ error messages.)
that the bulk builds would only work properly with the audit-packages
package installed at all times.
I am obviously hideously mistaken in this.
By popular request, revert the change to add audit-packages, since I
am told this is the incorrect way of doing this.
cd's into pkgtools/pkglint and does a "make fetch".
Since the audit-packages check was introduced to bsd.pkg.mk, this
"make fetch" will fail with an error message which looks like:
===> do-fetch [pkglint-4.42.1] ===> Checking for vulnerabilities in pkglint-4.42.1
===> check-vulnerable [pkglint-4.42.1] ===> *** The audit-packages package must be at least version 1.40
===> check-vulnerable [pkglint-4.42.1] ===> *** Please install the security/audit-packages package and run
===> check-vulnerable [pkglint-4.42.1] ===> *** '/usr/pkg/sbin/download-vulnerability-list'.
*** Error code 1
since there is no audit-packages package installed at this point (we
are still doing the pre-installation checks).
So call "make fetch" with an additional SKIP_AUDIT_PACKAGES=yes
argument at this point.
However, this does not help people who are doing bulk builds
in sandboxes via read-only nullfs mounts.
A correct, long-term solution to this needs to be found.
formline(). This prevents long package names from being trimmed.
- Removed the "-p e" command line option from the pax(1) command line.
This makes it possible to run mk/bulk/post-build as different user
from the one who did the actual build.
- Added section headings to make reading the code more efficient.
- Added bp_die() for uniform error messages.
- Added sanity checks:
- $BULK_BUILD_CONF must be a regular file.
- Protect against spurious output from the bulk.conf file.
- Check that configuration variables are defined, non-empty and
do not contain newlines.
- Properly extract MAKECONF from the bulk.conf file when getting the
mk.conf variables.
- Protect against spurious output from BMAKE show-vars.
- Fixed the use of undefined values ($startdate).
- Changed some calls to my_system() to a more secure form.
- Check if the calls to chdir() actually work.
- Look for the BROKENFILEs in BULKFILESDIR instead of USR_PKGSRC.
- Changed postfix conditions to prefix conditions. Example:
do_foo() if $bar; ==> if ($bar) { do_foo(); }
nullfs mounts are used (on NetBSD).
Only attempt to install new pkg_install tools if the bulk build is not
taking place in a sandbox.
This is a quick and dirty hack to allow sandboxed bulk builds (the majority
of cases, I suspect) to operate properly once again.
- Added a bunch of internal variables that determine the correct places
for the build log files of the current package.
- Added section headings.
- Moved a sanity check out of the configuration section to not confuse
the reader.
- mk/bulk/print{depends,index} are called with two arguments, which allows
separation of BULKFILESDIR and PKGSRCDIR.
- Replaced uses of BUILDLOG, BROKENFILE and BROKENWRKLOG with their
internal counterparts, which use ${BULKFILESDIR} instead of ${PKGSRCDIR}.
- The main .make file is stored in ${BULKFILESDIR} instead of ${PKGSRCDIR}.
- The PKGLIST variable from the configuration file is passed to the
bulk-cache command in the environment. There are very few variables left
that have to be exported via export_config_vars(). Maybe that function
can be avoided alltogether.
pkgsrc packages, collecting different information.
- BMAKE is taken from the environment instead of querying `uname -s`.
- package-specific log files can be saved in a directory tree different
from PKGSRCDIR.
- Added sanity checks for BMAKE, bulkfilesdir and brokenfile.
- The .pkglist cache is saved in ${BULKFILESDIR} instead of ${PKGSRCDIR}.
- Reformatted the diagnostics.
- Changed the implementation how package-specific log files are generated.
No user-visible changes.
- Stricter preconditions:
- bulkfilesdir must be an absolute path.
- brokenfile must not contain slashes.
- The .pkglist file is taken from BULKFILESDIR, not from PKGSRCDIR.
- Error messages use the prefix "[printindex]" instead of "===>", as the
latter is already reserved for pkgsrc. Everything else would be confusing.
- If a "cd" into a package directory fails, an error message is written to
${BULKFILESDIR}/${BROKENFILE}.
- Renamed the second parameter from "brokenbasedir" to "bulkfilesdir". The
latter name is already used in bsd.bulk-pkg.mk for the same purpose.
- Renamed "cwd" to "pkgsrcdir" for the same reason.
Whoever had written such unusual code should have left a note in the
printindex script that the trailing white-space was required. It is such
an uncommon assumption that I treated it as a typo and removed it. Now
it's fixed. I also replaced the use of grep|awk with a simple awk, as
grep would have interpreted regular expressions in the package name.
In general, please think more about _appropriate_ ways to handle data.
grep(1) is not a tool for searching strings. It's a tool for searching
regular expressions.
if present. By design, it is not cleaned at the start of a bulk build; it
is intended to allow a bulk build admin to mark a time-consuming package as
not buildable, but which shouldn't be marked permanently as such in the
package's Makefile.
an empty MAKECONF...
$ echo ".include <bsd.own.mk>" > Makefile
$ env MAKECONF= make
make: "/usr/share/mk/" line 1: Need an operator
make: "/usr/share/mk/" line 2: Need an operator
...
- Added file post-build-conf, which provides functions to print and
export the config variables from build.conf.
- All config variables are shown at the start of a bulk build.
- Added a configuration variable MAKECONF with the obvious meaning.
- When printindex is called with two arguments, the second argument
specifies the base directory of the <brokenfile>. The complete path is
then <brokenbasedir>/${pkgdir}/<brokenfile>. This is useful for creating
the log files in a directory outside the pkgsrc root.
- Removed the perl5_dir variable, as it has only been used in one place.
- Added error checking for the installation of the prerequisite packages
after the main build.
"from the beginning".
- Moved the code to get the value of BULK_PREREQ some lines up, as there
are also 19 other variables using the same code. This saves one extra
check if the pkglint directory exists.
- Added BULK_PREREQ to the variables whose value is printed before starting
the actual build.
- When installing prerequisite packages, print a nice sentence instead of
a simple "echo $pkgdir".
The default function prepends the well-known "date, progress, pkgdir" to
each line.
- Check that the environment variable USR_PKGSRC is given and is an
absolute pathname.
- Introduced variables for the various directories. Their names all end in
_dir, and they contain absolute pathnames. Changed all instances of
${USR_PKGSRC}/pkgtools/pkglint to ${pkglint_dir}. Likewise for the other
directories.
- Added more error checking. The bulk build is terminated if the
pkg_install package cannot be updated or the prerequisite packages
cannot be installed.
a package.
This functionality consumed an enormous amount and time, and did not
do much useful work:
* It is not needed when doing a bulk build of the full pkgrsrc tree,
since the pre-requisite packages already have been built and cleaned.
* It does not work when doing a bulk build of a subset of the packages,
since it only cleans the direct dependencies, and not the full list.
- All error messages go to stderr instead of stdout.
- Added variable quoting.
- Reindented some code for better readability.
- Changed "Build Temporary Files" caption to "Some variables used in the
bulk build", which is more appropriate.
- Now it starts with "set -e", preventing unwanted continuing after
errors have occured.
- Added more error checking and handling.
- All variables are properly quoted (if at all possible).
- Merged the two loops for checking packages into one loop.
- Instead of first checking if a directory exists and then changing
into it, the tool now tries to cd, and handles failure.
- Fixed the check for duplicates. Some packages might have been skipped
unintentionally.
build.conf file. The only way to do this before had been to set the
(undocumented) environment variable BULK_BUILD_CONF. This is a
conservative extension.
Make use of find(1) instead of ls(1) to find the broken files.
This prevents the generation of an empty report due to 'too many arguments'
to ls on certain platforms.
with the results to ADMIN.
Now the file will be created if the option is not defined and the
email will be sent too, otherwise only the file will be created.
The file with the results will be created in
${FTP}/pkgsrc-results-${DATE}.txt.
Can't use an undefined value as an ARRAY reference at mk/bulk/post-build line 470.
In the case that no packages were broken, getBroken returns a hash
without the key "topten". When Perl tries to use that undefined value
as an array reference, it croaks.
vulnerable packages are uploaded directly into the vulnerable subdir.
While here:
quote variables
better handling of the temporary directory
remove some backwards compatibility code that's been here long enough
opsys-specific package handling was doing the same as non-opsys
specific, so fold them together.
Written together with dillo.
mount in the sandbox. Then in the individual OS section, turn it back
on if needed. Right now, just turn on NetBSD. At some point we should
probably fill out the other supported OS's or figure out a way to pull
in the information from the makefiles which already define much of whats
needed.
