ADDED: -p option. If you want to build a few packages, you
can specify them in command line:
distbb -p 'wip/dict-server wip/pkg_online wip/distbb'
Actually this option was added in version 0.16.0.
I just forgot to mention it in appropriate NEWS section.
FIXED: packages for which pkg_src_summary (pkgsrc tree scanning) failed
(bad PKGPATH, errors in Makefile etc.)
appear in META/packages_built_prev.txt.
Now such packages are collected in META/package_failed_scan.txt
If such failures happen, report.html includes an additional line
Packages failed (pkg_src_summary/scanning) <COUNT> <LOGS>
Slave hosts/chroots save building logs locally and send them to the
master host only after building is completed. This can speed-up
building.
"Top offenders" -> "Failures per maintainer"
Some developers treat the word "offender" offensive.
Packages failed at dbb_canbebuilt (that is packages failed because
of ONLY_FOR_PLATFORM, ONLY_FOR_PLATFORM etc.) and dbb_excluded
(packages are excluded from bulk build by person who runs a bulk
build) stages do not increase "failures per maintainer" counter.
It is recommended to always include dbb_canbebuilt to TARGETS,
otherwise counters in "Failures per maintainer" will be not fully
correct.
Speeds-ups for stage_gen_report
Version 0.16.0, by Aleksey Cheusov, Sat, 19 Jul 2008 14:13:26 +0300
fixed: after bulk build finished, not all empty CATEGORY/PACKAGE
directories were removed.
packages_built_prev.txt, packages_built.txt, packages_failed.txt,
packages_failed_deps.txt and packages_total.txt files now contain
both PKGPATH and PKGNAME
META/summary_cmp.txt is generated (pkg_summary vs. pkg_src_summary
comparison) and is included to report.html
META/pkg_src_summary.txt is generated and is included to report.html
Progress messages are generated in META/progress.txt.
This file is included to report.html
Cleaning the working directories is now optional.
distbb.conf:
CLEAN_TARGET=clean # clean them! (default in example distbb.conf)
CLEAN_TARGET=dbb_none # keep them!
Update your distbb.conf!
html link to itself is removed from report.html
CATEGORY/PKG/stage_*.html log files also conform to "HTML 4.01 Strict".
Tested via http://validator.w3.org
An internal "init" target and initial (before building start)
"clean" targets are not hardcoded in libexec/distbb/slave script
anymore. The special target "dbb_init" is implemented in
share/distbb/distbb.mk for initialization (removing PREFIX/PKGDBDIR
and unpacking bootstrap). Both "dbb_init" and "clean"
targets should be a part of TARGETS variables. Update your local
distbb.conf and set I_AM_READY to 2
META/packages_built.txt is sorted alphabetically and is included to
report.html as "list", That is, no "sorted" and unsorted" lists
anymore.
Minor fixes and additions in example distbb.conf
============================================================
Example of bulk build result:
http://www.mova.org/~cheusov/pub/pkgsrc-distbb/Linux/current-unprivileged/log/20080719.1147/META/report.html
META/packages_total.txt file is build that contains a list built
packages plus those that are not rebuilt (no need for this). This
file is uploaded by stage_upload_logs.
report.html now includes links to META/packages_total.txt,
META/packages_built_prev.txt, META/packages_built.txt,
META/packages_failed.txt and META/packages_failed_deps.txt.
added: option -B to 'distbb' executable. This is usefuil for
debugging. It sets an unchangable BUILD_ID.
LIST_ALL_PKGS_CMD command is run on the first slave host from
$SLAVES list in case both SLAVES and TRANSPORT are set. The problem
with master host is that master's and slave's /usr/pkgsrc may be not
the same.
FIXED: sometimes packages are built in a wrong order.
FIXED: there were two examples for PKG_UPDATE_SRC_SUMMARY_CMD
in the sample distbb.conf:
'pkg_update_src_summary' and 'pkg_src_summary'. But the latter
just didn't work because these two commands have imcomppatible usage.
Now this is fixed but changes are incompatible with old distbb.conf.
Update it according to the new sample for distbb.conf.
Also set I_AM_READY variable to 1.
Side effect: 'pkg_src_summary -A' can now be used for generating summary.
This will build pkg_src_summary only for requested packages (distbb -f)
and their dependancies, not for entire pkgsrc
tree (pkg_update_src_summary).
Both Hubert's bulk build and pbulk work this way. Now distbb too.
FIX: EXTRA_DEPS now can contain more than one package and is handled
correctly. Copy EXTRA_DEPS_CMD from the sample to your distbb.conf
The following files are sorted alphabetically:
CATEGORY/PACKAGE/failed_packages.txt
CATEGORY/PACKAGE/failed_because.txt
META/packages_built_prev.txt
META/packages_failed_deps.txt
In case of LOTS of failures this make searching in these files much easier.
Latest paexec, runawk and pkg_summary-utils are needed.
report.html conforms to HTML 4.01 Strict specification.
Thanks to validator.w3.org.
UTF-8 charset is also used (See NOTES and RESPONSIBLE variables)
"build_end" date/time are moved from stage_gen_report to stage_build.
Lots of comments and suggestions in the sample for distbb.conf
Other minor fixes, improvements and code clean-ups.
ADDED: mega-feature/new variable SPECIAL_PACKAGES
Example and notes from distbb.conf:
...
# some packages fail to build because they are a part of pkgsrc
# bootstrap. if you want to build them try to uncomment lines in
# SPECIAL_PACKAGES variables
#
# you may also want to use different building options
# to build some packages, e.g. to use gcc-3.4 for building
# audio/festival,
# under Linux Debian/etch or DragonFlyBSD where it is a part
# of "base" system, see lines in SPECIAL_PACKAGES.
# NOTE: There are no quotes around string containing spaces
# (this is not shell ;-) )
# NOTE: An empty string after EACH package section is mandatory
# NOTE: Do not forget that you can define your own Makefile targets
# in distbb.local.mk. this new target can run, for example,
# pkg_add -u
SPECIAL_PACKAGES="\
#PKGPATH=pkgtools/digest
#TARGETS=depends fetch checksum configure build package
#EXTRA_OPTS=BATCH=yes PKG_VERBOSE=1 USE_DESTDIR=full CHECK_SHLIBS=no
#
#PKGPATH=pkgtools/bootstrap-mk-files
#TARGETS=depends fetch checksum configure build package
#EXTRA_OPTS=BATCH=yes PKG_VERBOSE=1 USE_DESTDIR=full CHECK_SHLIBS=no
#
#PKGPATH=net/tnftp
#TARGETS=depends fetch checksum configure build package
#EXTRA_OPTS=BATCH=yes PKG_VERBOSE=1 USE_DESTDIR=full CHECK_SHLIBS=no
#
#PKGPATH=sysutils/checkperms
#TARGETS=depends fetch checksum configure build package
#EXTRA_OPTS=BATCH=yes PKG_VERBOSE=1 USE_DESTDIR=full CHECK_SHLIBS=no
#
#PKGPATH=devel/bmake
#TARGETS=depends fetch checksum configure build package
#EXTRA_OPTS=BATCH=yes PKG_VERBOSE=1 USE_DESTDIR=full CHECK_SHLIBS=no
#
#PKGPATH=pkgtools/pkg_install
#TARGETS=depends fetch checksum configure build package
#EXTRA_OPTS=BATCH=yes PKG_VERBOSE=1 USE_DESTDIR=full CHECK_SHLIBS=no
#
#PKGPATH=audio/festival # TARGETS is not changed!
#EXTRA_OPTS=CC=/usr/bin/gcc-3.4 CXX=/usr/bin/g++-3.4
#
"
distbb.conf: 'extract' and 'patch' targets are in TARGETS now.
distbb.conf: fix in comment: package-deinstall -> deinstall
Two new variables (for distbb.conf) are introduced:
- STDIN_FROM_DEV_NULL:
Some broken packages need an interaction with user even if BATCH
variable is set to YES. This may "hang" a bulk build. If you
don't want this, set the following variable to "1", otherwise
leave it unset.
- LOG_SIZE_LIMIT:
If STDIN_FROM_DEV_NULL=1, some packages may produce infinite
garbage to stdout and in turn may waste ALL your free disk
space. To prevent this, set the following limit. Leave it unset
for no limit.
fix:
If there is nothing to be built stage_build may fail under Linux,
Solaris, HP-UX, Interix and probably many others. This is because
Free/NetBSD xargs works differently - in case of no input it just
do NOT call program (xargs's argument) at all. Others do.
stage_*.txt files are not uploaded anymore.
Only html versions are uploaded.
README: distbb is NOT fork of pbulk in any way! ;-)
It was written from scratch.
Upgrade to version 0.11.0
Major changes:
Both .html and .txt report include a number failed dependancies for
each indirectly failed package. This number was limited by hardcoded
constant, now it is regulated by variable MAX_FAILED_DEPS_CNT Add it
to your distbb.conf.
stage_gen_report: fix for Debian dash - its ` and $( work
differently :-(
distbb uses TARGETS variables that contains a list of targets to run
in order to build, install and create a binary package. TARGETS
variable can easily be changed by user and this makes distbb very
flexible - user may change build order or even define his own
build/packaging procedure. As of this version user can define
his/her own make rules without changing mk.conf and without
recreating bootstrap tarball - by changing
/usr/pkg_distbb/etc/distbb.local.mk file, second configuration file
of distbb. distbb also defines two its own special targets which you
can use: dbb_canbebuilt and dbb_excluded. Read comments in
/usr/pkg_distbb/share/distbb/distbb.mk about them. Note that
etc/distbb/local.mk MUST exist, copy it from
share/examples/distbb/distbb.local.mk to etc/distbb.local.mk
and edit for preferences.
Major changes:
At startup, PREFIX and PKGDBDIR directories are deleted and
bootstrap is unpacked (on all $SLAVES when MASTER_MODE=yes).
NOTE: if you are upgrading from previous versions of distbb,
recheck your EXTRACT_BOOTSTRAP_CMD (tar -C / is needed).
distbb.conf: more examples for alternative/distributed
PKG_UPDATE_SRC_SUMMARY_CMD. If you'll use distributed one,
make sure PATH variable contains /usr/pkg_distbb/bin directory.
Major changes:
Instead of red counts the "Breaks" column now contains the link to
the actual list of indirectly failed packages.
stage_gen_report: hardcoded string "Responsible person" is not printed
distbb.conf: another example for PKG_UPDATE_SUMMARY_CMD
Additions and more details in README
BUILD_START renamed to BUILD_ID
Sometimes building packages stops because some of them require an
interaction with user. BATCH=yes doesn't help. In order to prevent
this, bmake is run with < /dev/null. As a result, such a bad
packages can not break/stop bulk builds anymore.
distbb version is included to the report.
An information about the person that runs a bulk build, is
optionally included to the report.
LIST_ALL_PKGS_CMD variable is introduced. It defaults to
'pkg_list_all_pkgs'. If set, the specified command is run to get
list of packages to be built.
You need to upgrade your distbb.conf
Examples:
LIST_ALL_PKGS_CMD='@@bindir@@/pkg_list_all_pkgs'
#LIST_ALL_PKGS_CMD='( @@bindir@@/pkg_list_all_pkgs
cat my_extra_wip_packages.txt )'
added: distbb --version|-V
"on <SLAVEHOST>" is appended to 'succeeded' and 'FAILED' log
messages. That is now it is seen on which host a particular package
succeeded or failed.
distbb.conf.in:
# The following is for pkg_summary-tools > 0.5.0
#PKG_UPDATE_SRC_SUMMARY_CMD='@@bindir@@/pkg_update_src_summary -f \
# "PKGNAME PKGPATH DEPENDS BUILD_DEPENDS"'
NEWS:
stage_init.in: completely reimplemented. An algorithm become easier
and cleaner. FIXED: incorrect calculating "build_previously" count,
and therefore "total_packages" too. The code also become cleaner.
FIX: build_init.in now doesn't depend on fields order in
pkg_src_summary (doesn't reply on that PKGPATH is before
[BUILD_]DEPENDS). This may cause an inifinite loop because of
oncorrect (cyclic) dependancies in META/tmp/trans_closure.tmp
distbb.conf.in:
minor fix: pkg_src_update_summary -> pkg_update_src_summary
Exporting LC_ALL=C and other important initializations have been
moved from distbb.in to common.in. Now every significant
initialization is made in PREFIX/libexec/distbb/stage_*, i.e. these
scripts become more all-sufficient.
Minor changes and fixes in README
