kpriv/pkgsrc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
pkgsrc/mk/bulk/bsd.bulk-pkg.mk

332 lines
14 KiB
Makefile
Raw Normal View History

Add date when a buid for a pkg started, and comment a place that has some side effecs
2001-05-09 04:54:38 +02:00
# $NetBSD: bsd.bulk-pkg.mk,v 1.27 2001/05/09 02:54:38 hubertf Exp $
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
#
# Copyright (c) 1999, 2000 Hubert Feyrer <hubertf@netbsd.org>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. All advertising materials mentioning features or use of this software
# must display the following acknowledgement:
# This product includes software developed by Hubert Feyrer for
# the NetBSD Project.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
##
## For this to work, put
## DEPENDS_TARGET= bulk-install
## BATCH= yes
## into your /etc/mk.conf, then cd /usr/pkgsrc; make bulk-package
##
## bulk-install will only compile pkgs if necessary, and install binary
## pkgs if possible/available.
## bulk-package will build binary packages, and remove them (together with
## any depending packages) immediately after that, leaving only the
## binary package back. Broken packages are marked (``.broken'').
##
LS?= ls
WC?= wc
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
# This variable is set to 'yes' by the pkgsrc/mk/bulk/build script. It enables
# the use of several cache files (DEPENDSTREEFILE, DEPENDSFILE, SUPPORTSFILE,
# and INDEXFILE) for speeding up the processing of dependencies. If we're not
# actively in the middle of a bulk build, however, we don't know that these files
# exist or are up to date and they take quite a while to rebuild. So unless
# they're known to exist and be up to date, don't use them.
USE_BULK_CACHE?= no
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
# Shall we remove any packages which are installed, but not required
# to build this package
PRECLEAN?= yes
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
.ifdef OBJMACHINE
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
# This file exists to mark a package as broken
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
BROKENFILE?= .broken.${MACHINE}
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
# This file is where the log of the build goes
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
BUILDLOG?= .make.${MACHINE}
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
# This is a top level file which lists the entire pkgsrc depends tree in the format:
# foo/bar devel/libfoo
# meaning 'foo/bar' is requied to build 'devel/libfoo'
# this is in the format needed by tsort(1)
DEPENDSTREEFILE?= ${PKGSRCDIR}/.dependstree.${MACHINE}
# This is a top level file which lists the entire pkgsrc depends tree in the format:
# foo/bar depends on: devel/libfoo devel/libbar devel/baz .....
# ie, to build foo/bar we need devel/libfoo devel/libbar devel/baz ... installed
DEPENDSFILE?= ${PKGSRCDIR}/.depends.${MACHINE}
# This is a top level file which lists the entire pkgsrc depends tree in the format:
# devel/libfoo is depended upon by: foo/bar graphics/gtkfoo ...
# ie, to build foo/bar we need devel/libfoo to be installed.
# to build graphics/gtkfoo we need devel/libfoo to be installed
SUPPORTSFILE?= ${PKGSRCDIR}/.supports.${MACHINE}
# This is a top level file which cross-references each package name and pkg directory
# in the format:
# devel/libfoo libfoo-1.3
INDEXFILE?= ${PKGSRCDIR}/.index.${MACHINE}
# file containing a list of all the packages in the correct order for a bulk build.
# the correct order is one where packages that are required by others are built
# before the packages which require them.
ORDERFILE?= ${PKGSRCDIR}/.order.${MACHINE}
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
.else
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
BROKENFILE?= .broken
BUILDLOG?= .make
DEPENDSTREEFILE?= ${PKGSRCDIR}/.dependstree
DEPENDSFILE?= ${PKGSRCDIR}/.depends
SUPPORTSFILE?= ${PKGSRCDIR}/.supports
INDEXFILE?= ${PKGSRCDIR}/.index
ORDERFILE?= ${PKGSRCDIR}/.order
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
.endif
-add support for a BULK_PREREQ make variable. BULK_PREREQ contains a list of packages which must stay installed during the build, but are not pulled in by the DEPENDS for each package. Currently, BULK_PREREQ will always include pkgtools/digest. At this time, the primary use will be to add BULK_PREREQ+=pkgtools/xpkgwedge in /etc/mk.conf to do an xpkgwedge'd bulk build. It is up to the user to make sure that the list of packages in BULK_PREREQ is a flattened list (ie all the DEPENDS are listed too). Again, at this time, xpkgwedge is really the only package that should be added to the list. -add an ADMINSIG build.conf variable. This is the signature at the end of the email report. Maybe now I'll quit forwarding reports signed as "-Hubert". -while here, eliminate grep|awk lines and `grep >/dev/null` replacing them with pure awk and grep -q.
2001-03-19 12:25:39 +01:00
# a list of pkgs which we should _never_ delete during a build. The primary use is for digest
# and also for xpkgwedge. Add pkgtools/xpkgwedge in /etc/mk.conf to do an xpkgwedged bulk build.
BULK_PREREQ+= pkgtools/digest
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
# check if the $REF file is uptodate, i.e. is newer than any of
# the pkg files; prints "1" if upto date, "0" if not.
bulk-check-uptodate:
@uptodate=1 ; \
if [ -f "${REF}" ]; then \
${SHCOMMENT} "Check files of this package" ; \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
newfiles="`find . -type f -newer "${REF}" -print | ${EGREP} -v -e ./work -e pkg/COMMENT -e pkg/DESCR -e README.html -e CVS -e '^\./\.' || true`" ; \
nnewfiles="`find . -type f -newer "${REF}" -print | ${EGREP} -v -e ./work -e pkg/COMMENT -e pkg/DESCR -e README.html -e CVS -e '^\./\.' | ${WC} -l`" ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
if [ "$$nnewfiles" -gt 0 ]; then \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} >&2 "BULK> Package ${PKGNAME} ($$newfiles) modified since last 'make package' re-packaging..." ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
uptodate=0 ; \
else \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} >&2 "BULK> ${REF} is up to date." ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
fi ; \
else \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} >&2 "BULK> Package ${PKGNAME} not built yet, packaging..." ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
uptodate=0 ; \
fi ; \
if [ "$$uptodate" = "1" ]; then \
${SHCOMMENT} "Check required binary packages" ; \
deps=`${PKG_INFO} -qf ${REF} \
| ${GREP} ^@pkgdep \
| ${SED} 's,@pkgdep.,,g'`; \
for dep in $$deps ; do \
${SHCOMMENT} "check against the binary pkg that pkg_add would pick, too:" ; \
${SHCOMMENT} "(Only one should be returned here, really...)" ; \
a couple of minor changes to the bulk-check-uptodate target: - quote arguments to pkg_admin so the shell doesn't try to expand any *'s. - be sure that all required binary packages exist. The previous version made sure the required packages were not newer, but missed catching missing required packages. Still todo: When examining required binary packages to see if they have changed, the test only looks at the modification time of the binary package. It does not verify that the required binary package is also up to date with respect to its pkgsrc files.
2000-08-25 14:43:28 +02:00
pkg=`${PKG_ADMIN} lsbest "${PACKAGES}/All/$$dep"` ; \
if [ -z $$pkg ]; then \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} >&2 "BULK> Required binary package $$dep does not exist, rebuilding... " ; \
a couple of minor changes to the bulk-check-uptodate target: - quote arguments to pkg_admin so the shell doesn't try to expand any *'s. - be sure that all required binary packages exist. The previous version made sure the required packages were not newer, but missed catching missing required packages. Still todo: When examining required binary packages to see if they have changed, the test only looks at the modification time of the binary package. It does not verify that the required binary package is also up to date with respect to its pkgsrc files.
2000-08-25 14:43:28 +02:00
uptodate=0 ; \
elif [ "$$pkg" -nt "${REF}" ]; then \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} >&2 "BULK> Required binary package $$dep (`basename $$pkg`) is newer, rebuilding... " ; \
a couple of minor changes to the bulk-check-uptodate target: - quote arguments to pkg_admin so the shell doesn't try to expand any *'s. - be sure that all required binary packages exist. The previous version made sure the required packages were not newer, but missed catching missing required packages. Still todo: When examining required binary packages to see if they have changed, the test only looks at the modification time of the binary package. It does not verify that the required binary package is also up to date with respect to its pkgsrc files.
2000-08-25 14:43:28 +02:00
uptodate=0 ; \
else \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} >&2 "BULK> Required binary package $$dep (`basename $$pkg`) is usable. " ; \
a couple of minor changes to the bulk-check-uptodate target: - quote arguments to pkg_admin so the shell doesn't try to expand any *'s. - be sure that all required binary packages exist. The previous version made sure the required packages were not newer, but missed catching missing required packages. Still todo: When examining required binary packages to see if they have changed, the test only looks at the modification time of the binary package. It does not verify that the required binary package is also up to date with respect to its pkgsrc files.
2000-08-25 14:43:28 +02:00
fi ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
done ; \
fi ; \
${ECHO_MSG} $$uptodate
# rebuild binpkg if any of the pkg files is newer than the binary archive
# set DO to ":" to not actually do anything (debugging, ...)
bulk-package:
log more stuff to buildlog
2000-09-02 15:53:06 +02:00
@if [ -f ${BROKENFILE} ]; then \
${ECHO_MSG} "BULK> *** Package ${PKGNAME} seems broken and needs attention:" ; \
${LS} -la ${BROKENFILE}; \
exit 1; \
fi
@( \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
if [ "${PRECLEAN}" = "yes" ]; then \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
${ECHO_MSG} '' ; \
${ECHO_MSG} '' ; \
${ECHO_MSG} '###' ; \
Add date when a buid for a pkg started, and comment a place that has some side effecs
2001-05-09 04:54:38 +02:00
${ECHO_MSG} '###' `date`: ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
${ECHO_MSG} '### ${MAKE} ${.TARGET} for ${PKGNAME}' ; \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} '### Current pkg count: ' `${LS} -l ${PKG_DBDIR} | ${GREP} ^d | ${WC} -l` installed packages: `${LS} ${PKG_DBDIR} | ${GREP} -v pkgdb.byfile.db`; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
${ECHO_MSG} '###' ; \
log more stuff to buildlog
2000-09-02 15:53:06 +02:00
fi \
) 2>&1 | tee -a ${BUILDLOG}
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
@uptodate=`${MAKE} ${MAKEFLAGS} bulk-check-uptodate REF=${PKGFILE}` ; \
if ${PKG_INFO} -qe "${PKGNAME:C/-[^-]*$/-[0-9]*/}" ; then \
installed=1; \
else \
installed=0; \
fi ; \
if [ $$uptodate = 1 ]; then \
log more stuff to buildlog
2000-09-02 15:53:06 +02:00
( if [ $$installed = 1 ]; then \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} "BULK> Package ${PKGNAME} is upto-date, and still installed" ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
else \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} "BULK> Nothing to be done." ; \
log more stuff to buildlog
2000-09-02 15:53:06 +02:00
fi \
) 2>&1 | tee -a ${BUILDLOG}; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
else \
log more stuff to buildlog
2000-09-02 15:53:06 +02:00
( if [ $$installed = 1 ]; then \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} "BULK> Removing outdated (installed) package ${PKGNAME} first." ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${ECHO_MSG} ${MAKE} deinstall ; \
${DO} ${MAKE} deinstall ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
fi ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
if [ -f ${PKGFILE} ]; then \
Back out last commit: It nuked pkgs that we compile in several versions, e.g. for "normal" and -current pkgs.
2000-08-24 02:52:23 +02:00
${ECHO_MSG} "BULK> Removing old binary package..." ; \
${ECHO_MSG} ${RM} -f ${PKGFILE} ; \
${DO} ${RM} -f ${PKGFILE} ; \
when removing out of date binary packages from ${PACKAGES}/All, also remove the links from the package category subdirectories so we don't leave bad links around.
2001-02-15 21:13:28 +01:00
for cat in ${CATEGORIES} ;\
do \
${ECHO_MSG} ${RM} -f ${PACKAGES}/$$cat/${PKGNAME}${PKG_SUFX}; \
${DO} ${RM} -f ${PACKAGES}/$$cat/${PKGNAME}${PKG_SUFX}; \
done ;\
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
fi; \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
${ECHO_MSG} "BULK> Full rebuild in progress..." ; \
${ECHO_MSG} "BULK> Cleaning package and its depends" ;\
if [ "${USE_BULK_CACHE}" = "yes" ]; then \
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
for pkgdir in ${PKGPATH} `${GREP} "^${PKGPATH} " ${DEPENDSFILE} | ${SED} -e 's;^.*:;;g'`; do \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${DO} (cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} clean) ; \
done ;\
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
else \
${ECHO_MSG} ${MAKE} clean CLEANDEPENDS=YES;\
${DO} ${MAKE} clean CLEANDEPENDS=YES;\
fi; \
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
if [ "${PRECLEAN}" = "yes" ]; then \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${ECHO_MSG} "BULK> Removing installed packages which are not needed to build ${PKGNAME}" ; \
get rid of an unneeded awk invocation
2001-03-31 02:03:03 +02:00
for pkgname in `${PKG_INFO} -e \*` ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
do \
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
if [ "${USE_BULK_CACHE}" = "yes" ]; then \
pkgdir=`${GREP} " $$pkgname " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\
if ${PKG_INFO} -qe $$pkgname ; then \
${SHCOMMENT} "Remove only unneeded pkgs" ; \
if ! ${EGREP} -q "^${PKGPATH}.* $$pkgdir( |$$)" ${DEPENDSFILE} ; then \
case "${BULK_PREREQ}" in \
*$$pkgdir* ) \
${ECHO_MSG} "BULK> Keeping BULK_PREREQ: $$pkgname ($$pkgdir)" ;\
;; \
* ) \
${ECHO_MSG} ${PKG_DELETE} -r $$pkgname ; \
${DO} ${PKG_DELETE} -r $$pkgname || true ; \
if ${PKG_INFO} -qe $$pkgname ; then \
${DO} ${PKG_DELETE} -f $$pkgname || true ; \
fi ;\
;; \
esac ; \
else \
${ECHO_MSG} "BULK> ${PKGNAME} requires installed package $$pkgname ($$pkgdir) to build." ;\
fi ;\
fi ;\
else \
${SHCOMMENT} "Remove all pkgs" ; \
${ECHO_MSG} ${PKG_DELETE} -r $$pkgname ; \
${DO} ${PKG_DELETE} -r $$pkgname || true ; \
if ${PKG_INFO} -qe $$pkgname ; then \
${DO} ${PKG_DELETE} -f $$pkgname || true ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
fi ;\
fi ;\
done ; \
fi ;\
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
if [ "${USE_BULK_CACHE}" = "yes" ]; then \
Add date when a buid for a pkg started, and comment a place that has some side effecs
2001-05-09 04:54:38 +02:00
${SHCOMMENT} "Install required depends via binarypkgs XXX" ; \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
${ECHO_MSG} "BULK> Installing packages which are required to build ${PKGNAME}." ;\
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
for pkgdir in `${GREP} "^${PKGPATH} " ${DEPENDSFILE} | ${SED} -e 's;^.*:;;g'` ${BULK_PREREQ} ; do \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
pkgname=`${GREP} "^$$pkgdir " ${INDEXFILE} | ${AWK} '{print $$2}'` ; \
pkgfile=${PACKAGES}/All/$${pkgname}.tgz ;\
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
if ! ${PKG_INFO} -qe $$pkgname ; then \
do some extra checks. In particular, don't try to pkg_add a package if it doesn't exist (doh!).
2001-05-02 15:51:43 +02:00
if [ -f $$pkgfile ]; then \
${ECHO_MSG} "BULK> ${PKG_ADD} $$pkgfile"; \
don't exit if a depends can't be pkg_add'ed. In this case it will be rebuilt from sources.
2001-05-09 04:39:31 +02:00
${DO} ${PKG_ADD} $$pkgfile || ${ECHO_MSG} "warning: could not add $$pkgfile." ; \
do some extra checks. In particular, don't try to pkg_add a package if it doesn't exist (doh!).
2001-05-02 15:51:43 +02:00
else \
${ECHO_MSG} "BULK> warning: $$pkgfile does not exist. It will be rebuilt." ;\
fi ;\
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
else \
${ECHO_MSG} "BULK> Required package $$pkgname ($$pkgdir) is already installed" ; \
fi ;\
done ;\
fi ;\
log more stuff to buildlog
2000-09-02 15:53:06 +02:00
${ECHO_MSG} ${MAKE} package '(${PKGNAME})' 2>&1 ; \
${DO} ( ${MAKE} package 2>&1 ); \
) 2>&1 | tee -a ${BUILDLOG} ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
if [ -f ${PKGFILE} ]; then \
remove debug code
2000-09-02 16:05:36 +02:00
${RM} ${BUILDLOG} ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
else \
${MV} ${BUILDLOG} ${BROKENFILE} ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
( \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
${ECHO_MSG} "BULK> ${PKGNAME} was marked as broken:" ; \
${LS} -la ${BROKENFILE} ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${ECHO_MSG} ${MAKE} deinstall ; \
${DO} ${MAKE} deinstall ; \
when a package fails to package, keep a count of how many others packages fail as a result. Then report this number in the generated email and html summary. The goal is to help the pkgsrc crew focus our efforts on the broken packages which have the largest impact. Thanks to Christoph Badura for suggesting that I do this and Hubert Feyrer who suggested an easier to read output format.
2001-02-08 20:17:38 +01:00
nbrokenby=0;\
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
if [ "${USE_BULK_CACHE}" = "yes" ]; then \
${ECHO_MSG} "BULK> Marking all packages which depend upon ${PKGNAME} as broken:"; \
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
for pkgdir in `${GREP} "^${PKGPATH} " ${SUPPORTSFILE} | ${SED} -e 's;^.*:;;g'`; do \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
pkgname=`${GREP} "^$$pkgdir " ${INDEXFILE} | ${AWK} '{print $$2}'` ;\
${ECHO_MSG} "BULK> marking package that requires ${PKGNAME} as broken: $$pkgname ($$pkgdir)";\
pkgerr="-1"; \
pkgignore=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=IGNORE)`; \
if [ ! -z "$$pkgignore" -a ! -f ${PKGSRCDIR}/$$pkgdir/${BROKENFILE} ]; then \
${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) may not be packaged because:" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
${ECHO_MSG} "BULK> $$pkgignore" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
if [ -z "`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=BROKEN)`" ]; then \
pkgerr="0"; \
else \
pkgerr="1"; \
fi; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
fi; \
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) is broken because it depends upon ${PKGNAME} (${PKGPATH}) which is broken." \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
>> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
nbrokenby=`expr $$nbrokenby + 1`;\
-add support for a BULK_PREREQ make variable. BULK_PREREQ contains a list of packages which must stay installed during the build, but are not pulled in by the DEPENDS for each package. Currently, BULK_PREREQ will always include pkgtools/digest. At this time, the primary use will be to add BULK_PREREQ+=pkgtools/xpkgwedge in /etc/mk.conf to do an xpkgwedge'd bulk build. It is up to the user to make sure that the list of packages in BULK_PREREQ is a flattened list (ie all the DEPENDS are listed too). Again, at this time, xpkgwedge is really the only package that should be added to the list. -add an ADMINSIG build.conf variable. This is the signature at the end of the email report. Maybe now I'll quit forwarding reports signed as "-Hubert". -while here, eliminate grep|awk lines and `grep >/dev/null` replacing them with pure awk and grep -q.
2001-03-19 12:25:39 +01:00
if ! ${GREP} -q " $$pkgdir/${BROKENFILE}" ${PKGSRCDIR}/${BROKENFILE} ; then \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
${ECHO} " $$pkgerr $$pkgdir/${BROKENFILE} 0 " >> ${PKGSRCDIR}/${BROKENFILE} ;\
fi ;\
done ;\
fi ;\
when a package fails to package, keep a count of how many others packages fail as a result. Then report this number in the generated email and html summary. The goal is to help the pkgsrc crew focus our efforts on the broken packages which have the largest impact. Thanks to Christoph Badura for suggesting that I do this and Hubert Feyrer who suggested an easier to read output format.
2001-02-08 20:17:38 +01:00
nerrors=`${GREP} -c '^\*\*\* Error code' ${BROKENFILE} || true`; \
${ECHO_MSG} " $$nerrors ${PKGPATH}/${BROKENFILE} $$nbrokenby " >> ${PKGSRCDIR}/${BROKENFILE} \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
) 2>&1 | tee -a ${BROKENFILE}; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
fi ; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${ECHO_MSG} "BULK> Cleaning packages and its depends" ;\
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
if [ "${USE_BULK_CACHE}" = "yes" ]; then \
some code cleanup. - use the PKGPATH variable instead of getting it indirectly - don't use `foo` when we only want the return value, not the output. Also, in the non-cache case, we need to remove un-needed packages as noted in the most recent non-cached bulk build. Patches provided by Hubert and slightly modified by me.
2001-05-03 23:19:55 +02:00
for pkgdir in ${PKGPATH} `${GREP} "^${PKGPATH} " ${DEPENDSFILE} | ${SED} -e 's;^.*:;;g'`; do \
only use the depends cache files during a bulk build, otherwise revert to the old behaviour. This is done because unless we're in the middle of a bulk build, we don't really know that the cache files are up to date. These are fairly time consuming (relative to a single smallish package build) to generate and depend on all of the pkgsrc makefiles. During a bulk build, the overhead is far outweighed by the savings. In particular, 'make bulk-install' will now work correctly outside of a bulk build (useful for debugging broken packages). Thanks to Hubert who noted the 'make bulk-install' problem in a private email.
2001-02-14 00:02:23 +01:00
${DO} (cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} clean) ; \
done ;\
else \
${ECHO_MSG} ${MAKE} clean CLEANDEPENDS=YES;\
${DO} ${MAKE} clean CLEANDEPENDS=YES;\
fi ;\
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
fi
@if [ ! -f ${PKGFILE} ]; then \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${ECHO_MSG} "BULK>" Build for ${PKGNAME} was not successful, aborting. | tee -a ${BROKENFILE} ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
false; \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
else \
${RM} -f ${BUILDLOG} ;\
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
fi
# Install pkg - if possible from binary pkg (i.e. available & upto date)
# else revert to the old recompiling.
# Don't rely upon pkg_add to snarf in depending pkgs as they may have
# been modified and need rebuilding.
bulk-install:
@if [ `${MAKE} bulk-check-uptodate REF=${PKGFILE}` = 1 ]; then \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
if ! ${PKG_INFO} -qe ${PKGNAME} ; then \
s/depends/install-depends/
2000-08-01 04:56:43 +02:00
${DO} ${MAKE} install-depends ; \
be consistant about the use of "BULK> " as the prompt relating to the bulk build system instead of a mix of ">> " and "BULK> ". replace a few foo -> ${FOO}
2001-01-28 22:20:48 +01:00
${ECHO_MSG} "BULK> " ${PKG_ADD} ${PKGFILE} ; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
${DO} ${PKG_ADD} ${PKGFILE} ; \
fi ; \
else \
This commit incorporates several speed improvements which have been tested over several bulk builds on an alpha: - At the start of a bulk build, 4 files are created to allow fast lookup of various dependency tree things. These files are .index == maps package name (foo-2.3) to directory (bar/foo) .dependstree == contains the complete pkgsrc dependency tree in a tsort(1) compatible format. .depends == contains 1 line per package and lists all the build depends for the package. .supports == contains 1 line per package and lists all packages which depend on this package. - When a package fails to build, the list of all packages which depend upon it is read from .supports. Each of those packages is immediately marked as broken. This prevents us from trying to build those packages which can have significant overhead if lots of other depends are installed before the system notices the broken one. In addition, the post-build postprocessing will now indicate that a package is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which is broken because it depends on a failed package. This assists in determining where to focus our efforts in fixing broken packages. - In the old approach, all packages are removed after each one is built. The purpose was to a) conserve disk space, b) verify that all required dependencies are in fact listed, and c) prevent conflicts. The problem was that often times several packages in a row have similar depends. For example we might want to build several perl packages in a row. In the old approach, we would install perl, build the pkg, deinstall perl and continue with the next one. In the new approach, when it is decided that a pkg is out of date and should be rebuilt, the .depends file is used to obtain a list of pkgs we require. Then and pkgs which are no longer needed are removed. This helps to minimize the total number of pkg_add/pkg_delete's in a bulk build. - Since the order of the build is controlled by the depends tree, all depends for a given package will have been built by the time the pkg in question needs them. If any of the depends failed to build, then the pkg which needs the failed one will have been marked as broken. Given this, the complete depends list for a pkg is read from .depends and the depends installed via pkg_add rather than relying on recursive make calls to install the depends. - while here, fix a few minor bugs x - don't leave .make log files around when the build succeeds x - make sure we refer to the correct report file in the email x - use '.order' for the build order file instead of '.l' x - use 'grep -c' instead of 'grep | wc -l'
2001-02-01 09:47:29 +01:00
${ECHO_MSG} ${MAKE} bulk-package PRECLEAN=no; \
${DO} ${MAKE} bulk-package PRECLEAN=no; \
initial commit of bulk build targets (more to come)
2000-07-28 02:46:17 +02:00
fi
Reference in a new issue Copy permalink