on some systems, e.g. solaris, will simply drop long lines. These longs
lines are found in the flattened up and down dependency lists in the bulk
cache files. Instead, use sed to deal with this.
Fixes problems noted in Solaris bulk builds. Patch tested on NetBSD-2.0
and Solaris-2.9
binpkg timestamps, and ${PKGDIR} file timestamps, will be used to
determine if a rebuild is necessary. If changed to "no", then rebuilds
will only happen when PKGNAME actually changes.
* Add LINTPKGSRC_DB (and companion USE_LINTPKGSRC_CACHE, default "no") to
make pre-build generate a reusable database with "lintpkgsrc -E"; helps
with nfs-mounted pkgsrc. A future change will allow use of this in the
upload script.
* Add entry for PRUNEPACKAGES in build.conf-example (this was missing).
containing constructs of the form
{perl>=5.8.3nb1,perl-thread>=5.8.3nb1}
is that a Bourne shell "for" interprets this as two separate items in the
list, nuking the {}s. The above will never succeed and thus always cause
Perl-dependent packages to be rebuilt unconditionally.
Just adding more \s isn't enough -- the parsing of the depends list has to
be moved out of the "for" clause. So, echo them one per line and use "read"
to pull them in without allowing the shell to peek at those characters.
in the package directory, and add a link to it from .broken.html at
the point where the build fails. Also adjust the auxilliary scripts
to handle/cleanup .broken.work.html files.
This should enhance the ability of developers to debug broken builds
by providing important information about what is happening as a build
progresses and fails.
as it's only used internally by bsd.prefs.mk.
* Make _PKGSRCDIR a public variable by renaming it to PKGSRCDIR.
Also, generate its value from ${_PKGSRC_TOPDIR} so it's less fragile
than the old method of stripping off the last two components of
${.CURDIR}. PKGSRCDIR may now be used after bsd.prefs.mk is defined.
* Change all references to _PKGSRCDIR to PKGSRCDIR.
this fixes a problem where bulk builds with PKG_SUFX=.tbz would
needlessly, repeatedly rebuild dependency packages because
${PKGNAME}.tgz did not exist.
Portability note: unlike GNU grep, traditional grep did not con-
form to POSIX.2, because traditional grep lacked a -q option and
its -s option behaved like GNU grep's -q option. Shell scripts
intended to be portable to traditional grep should avoid both -q
and -s and should redirect output to /dev/null instead.
And, guess what, that's exactly what we run into when we call IRIX'
egrep(1) with '-q' (which it doesn't know). So let's follow this
suggestion and use neither '-q' nor '-s' and redirect output to
/dev/null instead.
- If it's a full sentence, start capitalize first letter and end with a
period.
- Break comments at ~80 chars so it fits into a terminal.[*]
(* This should be done for the code as well, but then it'll need testing.
I'll do this at another point in time.)
In particular, when SPECIFIC_PKGS is set in /etc/mk.conf, you can now do
sh mk/bulk/build
and have the right thing happen. Only those packages explicitly listed
and those which are depended upon are considered for the build. Other
than the restricted list of packages, the bulk build works the same way
as a full bulk build.
definition is not intended to be set by the user, and so has a leading
'_', and is undocumented.
If set to "yes", then a "make clean" of the package will not be done,
thereby preserving the working directory. This is useful for
debugging problems in bulk builds.
1) When checking if any of the required binary pkgs is newer, it's
not good to look into the (already existing) binary pkg, as that
might be unchanged. Instead, look at the DEPENDS.
In the context of the recent jpeg changes, the gd package itself was
not changed, but the DEPENDS were (via buildlink files). Now looking
into the existing gd binary pkg still said it wanted jpeg-6b instead
of the now-wanted jpeg>=6b, which was only available via the DEPENDS.
That's the first chunk of the patch below.
2) While debugging this, I found that the change in rev. 1.48 was
wrong, as can be seen throughout the last bulk build, search for errors
like:
find: "/usr/cvs.local/pkgsrc/packages/i386/All/gd-2.0.15.tgz": No such file or directory
As the whole operation is really on two files (as assured by "pkg_admin
lsbest" for pkg and REFS by definition), the quotes can be ommitted.
Why this wasn't caught when that change was tested is beyond me - maybe
different sh(1) behaviour? (The error happened on 1.6.1_STABLE, see
e.g. http://smaug.fh-regensburg.de/~feyrer/ftp/pub/NetBSD/pkgstat-i386/last/www/p5-Template-Toolkit/.broken.yui.html).
Anyways, that's addressed in the second part of the patch below, too.
3) Use ${FIND} while there.
can stop builds when a dependency is broken, yet continue builds
when a dependency is merely skipped (usually because it duplicates
functionality in the base system). Thus IGNORE_FAIL, the
oft-misunderstood and naught-documented option, goes away.
This addresses many people's complaints on tech-pkg@ and other
lists, as well as PR pkg/18157.
OBJMACHINE and OBJHOSTNAME to work. Also don't hardcode /usr/pkg and
/usr/X11R6 for LOCALBASE and X11BASE. Finally, fix a few typos in messages
while here.
The primary purpose is so when a package is broken due to broken depends,
there are links to the build logs of the broken dependencies.
Suggested by Brook Milligan on tech-pkg.
When OBJHOSTNAME is set, the various cache files and log files will include
the hostname in their names. Useful when multiple machines of the same
architecture share pkgsrc.
the cache files used during a bulk pkgsrc build.
- replace the code in the build script that used to create the cache
files with a 'make bulk-cache' call.
references of the pkglint package.
_PKGSRCDIR is an internal definition in bsd.pkg.mk, and a few packages
which would like to refer to other packages in the build tree. It should
not be set by users, but neither should it stop a user from building a
package if it is defined, so make it obvious that this is the case.
whitespeace, followed by it's required pkgs. Make sure to look for that
whitespace, to not catch any wrong positives, e.g.
egrep "x11/kdebase.*" $DEPENDSFILE
will catch depends for both KDE1 and KDE2 and do weird things then (among
other things, think that kdelibs-2.2.1 is ok as a dependency when building
in x11/kdebase, which is certainly not!)
Eight broken pkgs less in the bulk builds by adding a single space!
(Now if adding some more would help further... :-)
matches one of the DEPENDS, the bulk-build system looked up the
package in the INDEXFILE, then used the pkgdir returned to check
against the DEPENDS. If for some reason the package was not found
in the index file the package was left installed as 'requires
installed package'.
to remove it, then try again with 'pkg_delete -r'. This can happen when
some other out date pacakges are installed and we're trying to bulk build
a single package.
- use the PKGPATH variable instead of getting it indirectly
- don't use `foo` when we only want the return value, not the output.
Also, in the non-cache case, we need to remove un-needed packages as noted
in the most recent non-cached bulk build.
Patches provided by Hubert and slightly modified by me.
of packages which must stay installed during the build, but are not pulled
in by the DEPENDS for each package. Currently, BULK_PREREQ will always include
pkgtools/digest. At this time, the primary use will be to add
BULK_PREREQ+=pkgtools/xpkgwedge
in /etc/mk.conf to do an xpkgwedge'd bulk build. It is up to the user to make
sure that the list of packages in BULK_PREREQ is a flattened list (ie all the
DEPENDS are listed too). Again, at this time, xpkgwedge is really the only
package that should be added to the list.
-add an ADMINSIG build.conf variable. This is the signature at the end of the
email report. Maybe now I'll quit forwarding reports signed as "-Hubert".
-while here, eliminate grep|awk lines and `grep >/dev/null` replacing them
with pure awk and grep -q.
the old behaviour. This is done because unless we're in the middle of a
bulk build, we don't really know that the cache files are up to date. These
are fairly time consuming (relative to a single smallish package build) to
generate and depend on all of the pkgsrc makefiles. During a bulk build, the
overhead is far outweighed by the savings.
In particular, 'make bulk-install' will now work correctly outside of a bulk
build (useful for debugging broken packages).
Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
fail as a result. Then report this number in the generated email and
html summary. The goal is to help the pkgsrc crew focus our efforts
on the broken packages which have the largest impact. Thanks to
Christoph Badura for suggesting that I do this and Hubert Feyrer who
suggested an easier to read output format.
over several bulk builds on an alpha:
- At the start of a bulk build, 4 files are created to allow fast lookup of
various dependency tree things. These files are
.index == maps package name (foo-2.3) to directory (bar/foo)
.dependstree == contains the complete pkgsrc dependency tree in a tsort(1)
compatible format.
.depends == contains 1 line per package and lists all the build depends
for the package.
.supports == contains 1 line per package and lists all packages which
depend on this package.
- When a package fails to build, the list of all packages which depend upon
it is read from .supports. Each of those packages is immediately marked
as broken. This prevents us from trying to build those packages which can
have significant overhead if lots of other depends are installed before the
system notices the broken one.
In addition, the post-build postprocessing will now indicate that a package
is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which
is broken because it depends on a failed package. This assists in determining
where to focus our efforts in fixing broken packages.
- In the old approach, all packages are removed after each one is built. The
purpose was to a) conserve disk space, b) verify that all required dependencies
are in fact listed, and c) prevent conflicts. The problem was that often times
several packages in a row have similar depends. For example we might want to
build several perl packages in a row. In the old approach, we would install perl,
build the pkg, deinstall perl and continue with the next one.
In the new approach, when it is decided that a pkg is out of date and should be
rebuilt, the .depends file is used to obtain a list of pkgs we require. Then
and pkgs which are no longer needed are removed. This helps to minimize the total
number of pkg_add/pkg_delete's in a bulk build.
- Since the order of the build is controlled by the depends tree, all depends for a
given package will have been built by the time the pkg in question needs them.
If any of the depends failed to build, then the pkg which needs the failed one will
have been marked as broken. Given this, the complete depends list for a pkg is
read from .depends and the depends installed via pkg_add rather than relying on
recursive make calls to install the depends.
- while here, fix a few minor bugs
x - don't leave .make log files around when the build succeeds
x - make sure we refer to the correct report file in the email
x - use '.order' for the build order file instead of '.l'
x - use 'grep -c' instead of 'grep | wc -l'
- quote arguments to pkg_admin so the shell doesn't try to expand any *'s.
- be sure that all required binary packages exist. The previous version
made sure the required packages were not newer, but missed catching missing
required packages.
Still todo:
When examining required binary packages to see if they have changed, the test
only looks at the modification time of the binary package. It does not verify
that the required binary package is also up to date with respect to its
pkgsrc files.
