Some work/cleanup here would probably be desirable. I'm committing them
now to give others the ability to reproduce these package splits and help
with making these scripts better. The README should have some more
documentation in the future.
A better solution to the package set size problem would be to teach
sysinstall to ask for the CD on which a particular package is when it
needs to add it, but for now this will do.
Requested by: murray
Written by: steve
Discussed with: steve, kris (some time ago)
* Add a trap handler to try and clean up the build if it is interrupted
by a signal (one problem with the previous version is that package builds
whice are interrupted by ptimeout because they are stuck, leave their
working files lying around in the chroot).
* Switch to NFS v3 mounts instead of v2
* Autogenerate the version string to report in uname within the chroot,
based on the version string in the head of the CVS branch being built.
* Copy packages via cp from the NFS mount, not scp.
* Require an additional <tmpdir> argument so the client knows where its
temp directory is.
* Mount the portbuild directory readonly via NFS, and copy files that
way instead of via scp, which has too much overhead
* Don't assume the script will be called from the ports directory
* Use buildenv to set environment variables
* Set LOCALBASE and X11BASE to dummy variables to prevent the ports tree
from picking up packages installed on the host system, and patch up
the generated index at the end
operations in one central place, instead of doing them piecemeal all over
the place. This also includes the ability to customize settings per
port branch (e.g. XFREE86_VERSION)
Also, instead of hardcoding values of OSVERSION and OSREL which are going
to get forgotten again, pull them out of the source tree in ${branch}/src
so they track the head of the branch.
tangled up.
* Sort the environment variables so that all of the commented out ones
are grouped together
* Reverse sense of NOPLISTCHECK -> PLISTCHECK, since it's not an option
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
* Copy across UNAME_VERSION into the chroot area so that uname inside the
chroot reports the desired version
* Shorten timeout period from 12 hours to 4 hours to avoid delaying the builds
unnecessarily.
* Reverse sense of NOPLISTCHECK -> PLISTCHECK, since it's not an option
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
* Display correct pathnames in build progress messages
* Add a -nocdrom option to avoid the (time-consuming) clean-for-cdrom-list
target
* Remove x11/XFree86 from dummyports since we don't want to use an old
stale version of the package which never gets rebuilt
* Add comments noting that we should check for CVS conflicts in the cvs
update scripts and exit gracefully.
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
* Don't require the script to be called from inside the branch directory;
follow standard practise and pass the branch to build as an argument.
* Populate the chroot with the BSD.local.dist mtree file
* Add whitespace for readability, and sprinkle liberally with comments
* Comment on some inscrutable parts of the script which were presumably
put there to work around a (perceived) problem, but which I can't
understand.
of just proclaiming "error(s) occurred". This should immediately identify
which port is causing the build to break (although it's a bit of a verbose
solution).
* Update the list of ports which takes a long time to build.
* Don't use a single ports collection for every port build, indirect through
${pb}/${branch}/ports so we can have different ports collections for
each branch. Add -noportscvs option to prevent cvs update of ports
collection.
* Remove 3.x as a valid branch, add 4-exp for the 4.x experimental builds
used for testing bsd.port.mk patches
* Update usage information
decide machines which should be handed new jobs. Also, at some point someone
added a note about a possible bug in the reportload script which causes
this one to fall over.
down to user support flaws in the FreeBSD ports system. The flaw in question
is related to the fact that dependencies are often "chained", which allows to
simplify maintenance of ports with large number of implied dependencies (a la
Evolution, Nautilus, you-name-it). Dependency chaining it's not a problem by
itself, but the fact that when building or installing a port the system doesn't
check chain integrity - it's only checks that dependencies explicitly
specified in port's Makefile are satisfied, which opens wide window for
various hard-trackable problems when one or more links in the middle of the
chain missed.
The idea behind the tool is quite simple - it should be executed right after
main dependency checking procedure, two times for each build - check build-time
chain before building the port (pre-pre-extract) and check run-time chain
before installing the port (pre-pre-install). When executed, the tool checks
integrity of the specified chain (build-time, run-time or both) and reports all
errors, both fatal (dependency isn't installed) and non-fatal (dependency is
installed, but different version).
I've wrote this tool mostly to simplify maintenance of the GNOME ports, but
it doesn't contain anything GNOME-specific, so that it could be used in the
other parts of tree as well.
As an example I've added GNOME_VALIDATE_DEPS_CHAIN knob into bsd.gnome.mk (off
by default), which enables automatic chain validation for all ports that
USE_GNOMELIBS. This is a bit hackish, because I've used pre-extract and
pre-install targets - what we probably need is a generic way to plug various
custom tasks specified in bsd.xxx.mk (where xxx is kde, gnome, python, etc.)
into various parts of the build process (something like {pre,post}-pre-foo,
{pre,post}-post-foo springs into my mind).
The code is quite raw, so that I would appreciate any bug reports, patches,
suggestions, constructive critiquie and so on.
file. For example I'm usually keeping all working directories in /tmp using
WRKDIRPREFIX, while for the quick access to a port's files creating a symlink
to this directory in skeleton's dir (i.e. ports/foo/bar/src -->
/tmp/usr/ports/foo/bar/work/bar-0.0) and with this patch the tool correctly
works when I'm specifying `src/foo.c' as an argument.
Add mkppackage, which will create a "fake"-style package, but not
really. It doesn't try fake PREFIX and such to the build (it's not
really possible anyway yet), but it uses pkg_create's '-s' option to
pretend the package generated was installed in ${PREFIX}.
a work/ dir in the repository placed there by addport without permission.
Also restore the rcsids from the predecessors of this script by removing
their $'s.
If I had more time, I'd just revamp parts of this script.
Submitted by: greid
Apologies to: greid && cvs@
tool has something in common with update-patches shell script, but has more
features and is more intelligent. See README.patchtool or source code for
details.
categories which were failing to be mapped from directory to port name
because 'make index' can't know to run 'make describe' in directories
it doesn't know exist.
(2) (portbuild) Change no-output timeout from 1800 to 3600. Some ports
seem to actually take more than 30 minutes inside a command.
(3) (portbuild) Add -p flag to tar when extracting bindist.
(4) (makeindex) Remove the -j flag to make index, the argument was 1 anyway.
(1) The script now assumes make(1) knows how to handle long dependency
chains properly. quickports is a list of ports that take a
long time to build by thesmelves (not ports that have long
dependency chains). The script adds several extra dependency
levels to the generate Makefile for those ports to make their
dependency chains longer.
(2) Use -R flag to cvs. Comment out the cvsup part.
(3) Preserve error messages for ports that didn't build the first
time around but did build on the retry.
(4) Record the package's timestamps in the log directory by doing
an "ls -asFlrt" in the packages/All directory.
(5) Add a little sleep after two background jobs so outputs won't
be garbled.
guess WRKSRC or PATCHDIR. Don't munge "." characters in filenames.
This requires support in bsd.port.mk, a patch for which is in PR
24292.
Submitted by: Christian Weisgerber <naddy@mips.inka.de>
which holds final veto power over what @dirrm lines go into the plist.
This is a bit less evil than all the regexps previously used to manually
remove those directories.
mkpbuild just builds a port with WRKDIRPREFIX and PREFIX set to a
directory in /tmp, and with the owner and group variables set to the
user. A placeholder - in future, it'll have build error/log parsing.
mkpinstall's claim to fame is to do the same, except with "make install"
instead of "make build", and it then parses the mtree file, and compares
that and the current directory tree and the files, and auto-generates a
plist.
They're not pretty, but they finish the mkptools set basic
functionality: automatic skeleton generation with mkpskel, on-extraction
requirements guessing, and plist generation.
Future direction might be parsing build and install failures, checking
for '/usr/local' abuse, more requirements guessing, and stuff like that.
to the extra testing's make procs. Fix _stupid_ mistake where -n option
had no effect on hindering ``cvs add'' and CVSROOT/modules updating.
Cosmetics; remove extra space in generated commit log message and fix the
usage section's style in regards to options that take arguments. Add
missing -g option's mention.
Submitted by: assar (DISTDIR, CVSROOT/modules updating routine bugs)
than CVSROOT (now ADDPCVSROOT) to avoid conflicts for people who normally
set CVSROOT to something besides freefall (like myself). Second, allow
autofill to work for non-PR additions by checking for a -l argument of -1.
Third, remove the bogus $portname check in autofill that screws up if you
are adding something with a PKGNAMEPREFIX (i.e. p5-).
let portlint fill out your commit log message. It takes data from your
port's Makefile and pkg-comment to generate this, then displays the
resulting file and gives a chance to fix any problems.
Also, change -c option to move the file to the temp directory. Since -c
and -l option conflict, -l will take precedence. Move -c option down to
where directories are parsed.
Reviewed by: Jeremy Shaffner <jeremy@external.org>
new port. Slight hack used to obtain absolute path for the file. I'll
probably change that to make it copy the file to the tmpdir instead. For
now, this provides a stepping stone to use to begin automating commit log
messages - with, of course, a chance to edit the file again.
is going to commit a large number of ports and doesn't want to bloat
the repo any more than necessary. Later on I'll add support to
modulesupdate to do multiple modules at the same time. I should check that
script in CVS, but nobody's repocopied non-ports-specific tools to src yet
to my knowledge so I can't do that.
cvs ops. Many committers found it difficult to debug while using addport
with the -n argument. I'm inclined to agree with the sentiment.
Submitted by: sada, deischen, others
ftp site (specified in portbuild.conf). (It used to be implied by
-norestr.)
(2) Remove x11-toolkits/Motif-dummy from list of dummy ports, since
open-motif is now the default. Also gone is the MOTIF_OPEN
variable.
(3) Run the tar -czf of the ports tree and generation of CHECKSUM.MD5
in the background.
(4) Print the amount of time it took for the entire process, as well
as phase 1 and phase 2 of the compilations. (It is done by date
-r, so it will only work modulo 24 hours. Hopefully the build
will never go over 24 hours again.)
(5) Make symlink of the form [ae].${branch}.YYYYMMDD ->
[ae].${branch}.YYYYMMDDHH so we can have the simple "date" form as
well as the new "date+hour" directories for logs.
(6) Remove temporary make status files from /tmp that are over 60
minutes since the last modification. It was filling up the root
filesystem.
(7) Print out an "ls -lrt" of packages/All into logs/ls-lrt. This is
going to be used to evaluate make's job dispatch policies.
1) Re-add functionality removed in revision 1.11, but only as an option.
This time, it checks out its own copy of ports/Mk and points PORTSDIR
at the correct location. To use this, use the -a option. It will not
work if it can't find portlint in your path.
2) Add option -f to disable fetching the distfile.
3) Add option -m to disable checking out a copy of Mk for 1). For those
that have a current Mk in PORTSDIR and don't want to check out a fresh
copy for whatever reason.
4) Support CVSROOT environment variable.
5) Note environment variables recognized by addport.
6) Fix breakage trying to use -d with multiple directories (not tested).
7) Actually use $interactive variable.
8) Update usage().
9) Note original RCSID for my shell script; take over as MAINTAINER.
Tested by: gshapiro (this revision was used in adding sendmail)
mkptools are broken up into simple scripts:
mkpskel takes a distribution file as an argument and generates a
skeleton; it guesses the extract method, the package name, and so forth,
and generates populated Makefile and distinfo and empty pkg-descr,
pkg-comment, and pkg-plist.
mkpextr goes through the extract phase, and generates what it believes
are the necessary variables necessary to build the port. It guesses the
work source directory, what the Makefile is called, whether it has
configure, whether to use libtool, and other bits such as wildly
guessing kde, qt, gtk, gnome, and ssl requirements. Run it in a port
directory, and it generates Makefile.extr.
mkpmerge merges the results from mkpskel and mkpextr (and will later
merge the results from the other phases) into Makefile.
mkpclean cleans up any extra files that may be hanging about.
automatic checking on their ports to be added, I'm removing it and its
overriding -v option. Let the improper port additions continue, and screw
anyone who doesn't care about the repo.
Propelled by: obrien, msmith
mtree outputs.
Also change the title of the file from "List of extra files and
directories" to "List of files and directories that do not match their
mtree description", which is more accurate.
/var/portbuild is the new designated home of the portbuild setup, and
is expected to be a symlink to wherever you choose to put the stuff.
Also, change reportload to use /var/portbuild to store temporary files.
Seems there are some bugs in the null mount code that make the files
inaccessible if you are using an NFS root.
around now.
Make a symlink "Makefile" so I don't have to "make -f ../../Makefile"
when I'm restarting an aborted build.
Print out pnohang messages along with ptimeout messages.
Count the number of jobs at the beginning of the second phase too.
Check integrity of packages at end with gzip -t. Remove the ones that
don't pass the test.
Don't try to compare list of packages if there is no prior list to
compare with.
If the -norestr flag is given, copy the packages to the ftp site.
Make INDEX build failure a fatal error.
Copy INDEX to INDEX-${branch} for later perusal.
There is no -a flag in OpenSSH's scp.
Use sed to remove /usr/ports and stuff from restricted.sh and
cdrom.sh, so they can be run without further modification using
relative directories.
Take more options, namely -norestr (don't build RESTRICTED ports),
-noplistcheck (turn off "extra files check"), and -nodummy (use
open-motif instead of Motif-dummy and XFree86-4-* instead of
XFree86-3). -nodummy will become the default soon. Also a new flag
-nocvs will prevent cvs update on ${branch}/src only (while -nocvsup
will prevent cvsup and cvs update on usr/ports). -nocvs is to be used
when builds for more than one branch shares a same cvsup session.
Write out time cvsup is done so it can be used by log processing
scripts.
Handle dummy ports by using a more generic approach -- list the
directories they reside in (e.g., x11-toolkits/Motif-dummy) and
generate the package name from there, instead of hardcoding them
package names everywhere. Hopefully this will go away when we get rid
of the special case handling for Motif and XFree86.
Add something called "quickports" -- these are packages with very long
dependency lists that always seem to throttle the parallel build near
the end. By spelling them out explicitly, try to make make build
dependencies for them earlier while there are a lot of other stuff to
do. (Our make doesn't do anything with the target list, but maybe I
can fix that someday.)
Copy bindist.tar to clients before the second run of package build
too. It can be useful to "sneak" in fixes if something minorly wrong
is discovered in the bindist during the first run. Since there is a
md5 checking, this copying will not be done unless the files are
actually different.
When -norestr is set, delete restricted packages and distfiles using
restricted.sh immediately after completion of package build.
Generate CHECKSUM.MD5 after packages are built.
Copy distfiles over to the ftp site after everything is done. This is
run in the background on bak/distfiles so the next package build can
immediately start.
distfile to the server right after checksum. This will make it
possible to "harvest" distfiles even for ports that don't currently
build. (This was a reason for a lot of distfiles being lost in the
past -- someone commits a broken port, distfile doesn't get fetched to
ftp.FreeBSD.org since package doesn't build, and distfile is gone from
the master site before the port is fixed.) This also enables more
efficient sharing of distfiles since we can set MASTER_SITE_OVERRIDE
to the port build server and have a better chance of finding the
distfiles there.
Instead of using a simple make package | tee, redirect the output of
make package into a separate log file and concatenate it later. For
some reason, the gconf build always hangs after the build succeeds
(tee doesn't exit).
Handle the new arguments (-norestr for NO_RESTRICTED, -noplistcheck
for NOPLISTCHECK and -nodummy for NODUMMY) by converting them to
environment variables.
Add the branch=5 case. Run ldconfig on /usr/lib/compat too.
dopackages and abort build if there were any errors. It intentionally
leaves lock files behind in that case because it usually requires my
manual intervention to fix whatever's wrong (usually a botched INDEX
build due to someone forgetting to update dependency lines) and we
don't want subsequent failed tries to wipe out all the good packages
and distfiles we have accumulated.
effect since the job scheduler will only look at the first field, but
will give humans looking at the load reports some idea about the
machines (especially crashes).
location.
This is done to avoid write-on-write race conditions, when two package
builds try to write the same distfile back into the server. To avoid
this, the distfiles are first copied over to
distfiles/.pbtmp/${PKGNAME}, and then a .done file is created in that
directory. This script runs on the server periodically to copy the
files to the central location. Since this script copies one distfile
at a time, the race mentioned above doesn't exist. Also, it will only
copy files from directories where the .done file exists to avoid
read-before-write race conditions.
logs when there are logs newer than the .stamp file.
(2) Print time cvs update is done for reference.
(3) Print number of affected ports (the "aff." field). Basically the
number of times the port appears in INDEX minus one (itself), so
it won't take care of all chained build dependencies, but it will
give you some idea how critical a port is when looking at the long
list.
(4) Catch a few more error cases.
$DISTDIR to put distfiles in, if they can't touch $PORTSDIR/distfiles.
If this is freefall, default to $tmpdir, which is relatively secure since
most users on it are trusted. Reset FETCH_BEFORE_ARGS to "-btA" instead
of "-btsA", so the distfile is actually fetched. Add a method to allow
the name of the module to be changed if -i is used. Update to use the
modulesupdate that's in /usr/local/bin.
Addport should now work properly on freefall, without -v option.
Submitted by: roger (-s, $tmpdir by default on freefall)
Thanks to: peter (copying modulesupdate to /usr/local/bin)
