Support for unprivileged bulk builds without su/sudo etc.
Features:
- The following targets are always run in an unprivileged mode:
dbb_canbebuilt, dbb_excluded, dbb_vars, fetch, checksum, extract,
patch, configure and build.
- "package" targets are run in unprivileged mode
depending on variables PKG_DESTDIR_SUPPORT, USE_DESTDIR
and UNPRIV_USER (set in distbb.conf).
Requirements:
- slave hosts should be accessed using user "root".
- special variable UNPRIV_USER should be set to unprivileged user
in distbb.conf.
- target "clean" should be replaced with "dbb_clean" in TARGETS
(the default).
- target "package" should be changed to "dbb_package" in TARGETS
(the default).
stage_init: Minor fix for some broken shells
(http://gnats.netbsd.org/39466). Solution: touch command
is used instead 'echo >'
slave: do-bin-install target is not used anymore,
there is no need for it.
BOOTSTRAP_DEPENDS are handled automatically. There is not need to
list them in EXTRA_DEPS. Latest pkg_summary-utils>=0.39 is needed.
${MSCOMMONDIR} directory visible on master and slave hosts is needed
for running bulk builds. It defaults to ${PACKAGES}/.distbb.
FIX: some packages having alternatuive dependencies unexpectedly
failed at "depends" phase. Now new target "dbb_depends" is used for
installing dependencies. Make sure your locally modified TARGETS
contains "dbb_depends", not "depends".
New implementation for stage_init which is much smaller, easier and
cleaner. Some clean-ups and improvements in stage_build.
Latest pkg_summary-utils, pipestatus and runawk are required.
fix in stage_init.in: we should not rewrite build_src_summary_log_fn
file while searching for versions going backward.
fix in distbb: usage message should not interpret $ signs.
add: new build stage "upload" in BUILD_STAGES instead of two
"upload_logs" and "upload_pkgs".
Two new variables are introduced: CSS_FILE and ICON_FILE.
Now user can override the default distbb.css and distbb.ico.
.pkg-total style is not used for "Inspected packages" line.
all .html pages of the report have favicon taken from here
http://www.netbsd.org/~dillo/ made by Dieter Baron.
Later I'll try to create my own...
added: option -s. If applied, an exit status is 0 if building *ALL*
packages succeeded, otherwise exit status is 5.
distbb.conf is splitted into two parts: global defaults
(distbb.default.conf) and user's settings (distbb.conf). This is
major change in distbb! Instead of upgrading your existing
distbb.conf it is *strongly* recommended to recreate it from
examples/ template. Sorry for inconvenience.
added: 'distbb -r|--retry' option. if your bulk build failed for
some reason, you may continue it from that point it stopped at by
running 'distbb -r -B YYYYMM.HHMM'
stage_upload_pkgs now supports NO_BIN_ON_CDROM and NO_BIN_ON_FTP.
By default all packages are uploaded. If you want to change this,
set UPLOAD_PKGS_FILES variable in your distbb.conf
(See distbb.default.conf for a list of available variants).
stage_xxxx.html improvements. Packages failed due to their
dependencies now have their own info.html. So, you can see its
summary and decide whether to fix this package or not.
DEPENDS_TARGET configuration variable is removed, now it should be
a part of EXTRA_OPTS. See distbb.default.conf
mini report is introduced (META/report_mini.txt) which is sent by
stage_report (by default). In this mini report "Failures per
maintainer" and "Failed due to dependencies" sections are removed.
A purpose of this is to reduce a size of the report sent by email
(pkgsrc-bulk mailing list has limitations).
New config variable REPORT_MINI_SIZE_LIMIT. report_mini.txt sent by
email is limited to approximately REPORT_MINI_SIZE_LIMIT bytes.
PKGNAMEs are removed from report{,_mini}.txt. This also reduces
report's size.
New configuration variable BUILD_STAGES which stores a list of
building stages. If you want to, for example, send report before
uploading binaries, change this variable in your distbb.conf.
But be careful, list of available stages may change in the future.
dbb_init target does not output useless noise to stdout anymore. As
a result it is usually not listed in failed package's stage_*.html.
Usage of REPORT_CMD changed. Now redirection from stdin (report.txt)
should be inside it. Update your distbb.conf and set I_AM_READY to 4
ADDED: new config variables USER_MAIL and MAIL_PROG. They are for
stage_report.
ADDED: PKG_CMP_SUMMARY_CMD configuration variable, for setting
a program for comparing pkg_summary(5) for available binaries and
pkg_src_summary collected from pkgsrc source tree.
distbb.default.conf SPECIAL_PACKAGES:
'dbb_canbebuild' and 'replace' targets removed, 'dbb_vars' added
dbb_vars target fix: POSIX sed does not except \n
New stage_post_build containing user's actions is added.
Actions are defined in 'post_build' shell function in distbb.conf.
Update your existing distbb.conf! I_AM_READY=3 is required.
distbb.mk: new 'dbb_vars' target that outputs formatted variables
values. They may be useful for debugging.
I put dbb_vars to the default TARGETS.
Code updating "virtual best" bulk build directory
is now separated to an individual stage - stage_update_best.
It is run after 'stage_gen_report'.
distbb.conf: logging directory default changed from xxx/log to xxx/logs.
stage_gen_report: progress messages become more verbose
Major changes:
FIX: serious bug in "stage_build is rerunable" feature.
Generating a queue of tasks was not fully correct.
FIX: stage_gen_report may fail because of spaces inside MAINTAINER
field in package's summary (lang/gauche/Makefile revision 1.50).
Now everything after comma is removed, that is only first maintainer
in the list is printed.
fix: META/packages_{built,failed}.txt may have duplicates when
PREFIX/libexec/distbb/stage_build is run more than once
(rerun manually after unexpected failure)
New section in report.{html,txt} - "old bulk build vs. new bulk build".
It contains a list of packages that was built previosuly but failed
and more. See information about distbb_diff executable below.
To activate it, run 'distbb -B'. By default this feature is disabled.
New section in report.{html,txt} - "BEST bulk build vs. new bulk
build". The same as "old bulk build vs. new bulk build" but instead
of "old bulk build", "virtual best" bulk build is used. It is not a
real bulk build but a collection of packages that where built
sucessfully in the past. In this section you'll see ALL regressions,
not only regressions in comparison with ONE PREVIOUS bulk build.
report.txt is reorganized. Now it is aligned to 75-columns.
No "Failed Stage" column, ---- separator between sections,
"responsible person" is not included anymore
and other minor changes.
"failed due to deps" section is reorganized in both report.html and
report.txt. Both maintainers are output, i.e. maintainer of the
indirectly failed package and maintainer of failed dependancies.
New executable - 'distbb_diff' intended for comparing two
bulk builds.
usage: distbb_diff [OPTIONS] bbdir1 bbdir2
OPTIONS:
-h|--help display this help message
-V|--version distbb version
bbdir1 - path to bulk build directory, e.g. ~/bulk-logs/20080915.1010
bbdir2 - path to bulk build directory
Output format:
<CMP> <PKGPATH> <PKGBASE> <Version(s)>
where CMP is a combination of the following characters
+ - package failed in bbdir1 but succeeded in bbdir2
- - package was built sucessfully in bbdir1 but failed in bbdir2
R - package was removed or pkg_src_summary failed for it
d - package failed due to broken dependency
fix in stage_gen_report: detecting a "failed stage" is more correct
now (it takes into acount SPECIAL_PACKAGES field) and becomes clearer.
Log files PKGNAME/failed_packages.txt have been renamed
to PKGNAME/failed_deps.txt
META/*.raw files are not uploaded
report_html is run once, a whole stage_gen_report is now much faster
example distbb.conf:
CREATE_WRKDIR_SYMLINK=no added to EXTRA_OPTS,
obsolete PAEXEC_BUFSIZE removed,
-m option added to pkg_src_summary (PKG_UPDATE_SRC_SUMMARY)
stage_build becomes rerunable. If, for example, your SLAVE hosts
hanged up, turned off due to power lost or something similar
happened while BUILDING stage, you can continue running bulk build
by running PREFIX/libexec/distbb/stage_build <BUILD_ID>.
NOTE: In case MASTER host hanged up or turned off, rerunning
stage_build may be not fully correct because of possible data loss.
stage_build script is fully reworked. As it was planned, tricky and
dirty shlock/sh/awk code that processes packages graph is now
removed from distbb. All necessary functionality is generalized and
becomes a part of external utility "paexec" (see paexec -s option).
As a result pkgtools/shlock is not used anymore.
wip/netbsd-uuencode is not used anymore. PAEXEC sends unencoded data
to distbb.
Upgrade to version 0.11.0
Major changes:
Both .html and .txt report include a number failed dependancies for
each indirectly failed package. This number was limited by hardcoded
constant, now it is regulated by variable MAX_FAILED_DEPS_CNT Add it
to your distbb.conf.
stage_gen_report: fix for Debian dash - its ` and $( work
differently :-(
distbb uses TARGETS variables that contains a list of targets to run
in order to build, install and create a binary package. TARGETS
variable can easily be changed by user and this makes distbb very
flexible - user may change build order or even define his own
build/packaging procedure. As of this version user can define
his/her own make rules without changing mk.conf and without
recreating bootstrap tarball - by changing
/usr/pkg_distbb/etc/distbb.local.mk file, second configuration file
of distbb. distbb also defines two its own special targets which you
can use: dbb_canbebuilt and dbb_excluded. Read comments in
/usr/pkg_distbb/share/distbb/distbb.mk about them. Note that
etc/distbb/local.mk MUST exist, copy it from
share/examples/distbb/distbb.local.mk to etc/distbb.local.mk
and edit for preferences.
NEWS:
stage_init.in: completely reimplemented. An algorithm become easier
and cleaner. FIXED: incorrect calculating "build_previously" count,
and therefore "total_packages" too. The code also become cleaner.
FIX: build_init.in now doesn't depend on fields order in
pkg_src_summary (doesn't reply on that PKGPATH is before
[BUILD_]DEPENDS). This may cause an inifinite loop because of
oncorrect (cyclic) dependancies in META/tmp/trans_closure.tmp
distbb.conf.in:
minor fix: pkg_src_update_summary -> pkg_update_src_summary
Exporting LC_ALL=C and other important initializations have been
moved from distbb.in to common.in. Now every significant
initialization is made in PREFIX/libexec/distbb/stage_*, i.e. these
scripts become more all-sufficient.
Minor changes and fixes in README
