in several cases totally rewritten. I am also glad to say that
all of the advertised features now work as advertised. :)
The most exciting new feature is the completely new dependency
updater. I finally took the time to write the line-oriented version
of update_contents() that I wanted to write in the first place, so
no more sed, and no more warnings sent to $PAGER. This version relies
entirely on exact matches to DEPORIGIN to do its work, so it's 100%
reliable.
So what else is new? A lot of micro-optimizations have been done to
eliminate duplicated work, avoid using external commands when not
necessary, etc. There are also some not-so-micro optimizations, which
means that this version is noticably faster.
Totally new features:
1. --check-depends, which as described above calls the new functions
to parse existing +CONTENTS files for dependency data, uses that data
to update +REQUIRED_BY, and updates the +CONTENTS files. These same
routines are also used for updating ports.
2. A "fetch only" feature (-F) that will go get distfiles for new
versions of installed ports by default, and can also be used to get
the files for all of your installed ports, or certain branches by
using the -aft options.
3. An option to hide the build and install process by saving it to a
log (-H). These logs are deleted on successful install, and saved if
the update process aborts.
4. We now keep track of all the ports that get updated or installed in
a given invocation, and print the list at the very end of the process.
5. There is now a little "map" to show you where you are in the
process of the form "parent >> child >> grandchild >> etc."
6. Support for running as an unprivileged user (via sudo) has been
added. See the man page for details on how to configure this.
Improved features:
1. When showing the pkg-message/+DISPLAY files at the end of the
process, all files are grouped together and spit out to $PAGER (or
just cat if -u). If there is an error, there is a pause first
(unless -u).
2. There is now much more sanity checking on user input so that we can
bail out early if there is a problem.
3. The error messages have been completely re-done to be more
informative, and many of them include helpful suggestions on how to
fix the problems.
4. In the expunge (-e) feature, if there is dependency data for the
port you want to delete we now offer you the chance to delete that
instead of just aborting. (Use at your own risk!)
5. The "save old shared libs" feature (-w) is now a lot more
efficient, only runs ldconfig when it has to, and uses
/etc/rc.d/ldconfig.
6. The /var/db/pkg/*/+IGNOREME file now has more functions associated
with it. See the man page for details.
7. When using the -L feature, there is a warning printed if the port
has been set to FORBIDDEN, DEPRECATED, BROKEN, or IGNORE.
8. If you have a real /usr/X11R6 directory (not a symlink) this is
added to the PATH. Otherwise it's assumed that LOCALBASE == X11BASE.
9. Backup packages of existing ports are now saved to
/usr/ports/packages/portmaster-backup, and left there if -b is used.
This way you can create backup packages of a port without disturbing
existing packages with the same version.
There are numerous other things that are not user-visible which have
been cleaned, polished, made more efficient, etc.
===========+
1. If an +IGNOREME file is present for a package that has a new
version, let the user know.
2. Use printf to display BROKEN/FORBIDDEN/IGNORE messages so that
multi-line messages with embedded newlines will display properly.
Typo Fix
========
Fix two typos in the "it's interactive" message
... on the same line
... in two consecutive words. A new personal record.
I did run through the rest of the script with aspell and didn't
notice anything else, but who knows?
===============
1. In delete_all_distfiles(), $distdir already contains a trailing slash
2. Only print the "no new versions" message for -L, not -l
Big bug fix
===========
1. cd back to the port directory before trying to run 'make install'
pointy_hat_supply++
Approved by: portmgr (marcus)
to bottom read through prompted by a user reporting bug #1 below.
Fortunately I only found that one additional instance of forgetting
to virtualize a value we derive from bsd.port.mk, but I did find and
fix a few more things as well.
Bug Fixes
=========
1. Virtualize an occurrence of $distdir instead of using $pd/distfiles
2. Make all variables in read_distinfos() local
3. When specifying multiple ports on the command line, the test for
PARENT_PID to start read_distinfos() would always fail, so start it
in the initial PARENT_PID block instead. This also implies pulling the
DI_FILES creation up into that same block.
Non-Functional Changes
======================
1. Adjust whitespace in the code, and the script output in a few places
2. Use a shell expansion instead of sed to trim multiple results
3. Remove one redundant error check
4. Replace the name find_contents_distfiles() with find_dl_distfiles()
since the info is not in +CONTENTS anymore
5. Add/update comments to the existing code
Small changes to actual behavior
================================
1. When dealing with unrecorded dependencies pipe the results to
'less -F'
2. Minor optimization: In read_distinfos(), check to see if we're
upgrading an existing port, and if so do that port first so we're
sure the right distfiles will be in DI_FILES even if the function
doesn't complete before the port is done building/installling
3. Add a message to -L to indicate when there are no new versions
4. Move the checks for run-depends after installation of the parent
port so that if a run-depends port has a build-depends on the parent,
it will work. This is a supported, but apparently not recommended
configuration. To make this work do the install with -DNO_DEPENDS so
that we can take charge of the run-depends installs.
===================
1. When I changed how the MOVED file was treated in version 1.19
I accidentally typed a literal /usr/ports/. Change that to the
variable for PORTSDIR as it should be. [1]
2. When using the --show-work feature and the port has no dependencies,
exit immediately instead of recursing.
Submitted by: Jason C. Wells <jcw@highperformance.net> [1]
=========
1. For all values derived from bsd.port.mk, assert that they are not empty.
This helps prevent problems when running 'find $value ...'
2. Don't try to append to the master list of distfiles if we're using -D
Refinements for /var/db/ports/$name/distfiles
=============================================
1. Only create a distfiles file if the port has distfiles
2. Delete the distfiles list and try to delete PORT_DBDIR when we pkg_delete.
(The latter will fail if there is an options file present.)
3. Define the distfiles list even if we are using -D
============
1. Switch to saving distfile info in /var/db/ports/$name/distfiles
ala PR http://www.freebsd.org/cgi/query-pr.cgi?pr=106483
2. Keep track of distfiles we've already examined so we won't have to
do them again (or print the "keeping" message for -v more than once).
This has more benefit the more distfiles a port has.
3. Add all distfiles for the new port to DI_FILES after install.
This only makes a difference in a few cases, but it's easy to do.
Bug Fixes
=========
1. Make sure that $distdir always has a trailing slash in case it
is a symlink. [1]
2. In delete_stale(), test whether or not a file is in the _current_
distfile list to make sure that it is not auto_deleted by mistake. [1]
3. When building DI_FILES, only use lines starting with MD5 to avoid
adding every file > 1 time.
4. Only run read_distinfos() when we really need it.
[1] Inspired by ...
PR: ports/116298
Submitted by: NIIMI Satoshi <sa2c@sa2c.net>
===========
1. Before calling pkg_create for the installed version of the port,
cd into the PKGREPOSITORY directory. This not only simplifies the code,
it helps handle the problems of ports trees mounted read-only. [1]
2. Factor out initialization of the $pkgrep directory into a function.
Bug fixes
=========
1. Before calling 'find ${pkgrep}/ ... -delete' assert that $pkgrep is
not empty.
2. Make sure that $pkgrep is initialized everywhere it's needed.
Suggested by: Hans Lambermont <hans@lambermont.dyndns.org> [1]
the DESTDIR changes today.
2. Export pd, pdb, and distdir to avoid having to look them up each time
3. Streamline find_moved() by grepping for lines, and then recursing
4. Further streamline by using variable stripping instead of cut
=========
1. If we are doing --show-work, don't launch background processes
that are only useful for real builds.
2. Fix a rather obscure bug that can only happen:
IFF you are doing an install of a new port, and
IFF you have a dependency "early" in the list with an +IGNOREME file, and
IFF you have one or more dependencies "late" in the list that are not
installed. The bug is a false positive on the +IGNOREME for the uninstalled
ports. The solution is to reset the "port to test for +IGNOREME file"
variable each time through the loop. This will usually be a noop, but it
costs almost nothing in any case.
============
1. Keep a temporary flag that indicates a port has already been upgraded
during a -r or -f update, and add a -R (retry) option to let the user
skip those ports on subsequent attempts if the first update is interrupted.
2. Change the default behavior to use 'make config-conditional' to
take advantage of the recent enhancements to the OPTIONS functionality.
3. Add a new --force-config flag to use the old behavior of doing
'make config' for every port.
4. Don't delete backup packages until the original (parent) portmaster
process has finished successfully. That way if an update fails, the user
can back out the whole thing.
5. Add -w option to save old shared libraries from an installed port.
6. Add -x option to avoid building ports that match the glob pattern.
7. Split tracking of build and run depends to before and after the
port build respectively.
Enhancements
============
1. If the user interrupts the update, search for errant dialog
processes as well as errant portmaster child processes.
2. Avoid invoking pkg_version in check_for_updates() if the installed
version is already up to date.
3. Call pkg_info for seemingly stale dependencies (-s) so user has
a better idea what he's being asked to delete.
4. Use -DBATCH when calling 'make checksum' in the background so that
ports that have pre-everything dialogs no longer cause runaway dialog
processes.
5. Add BEFOREPORTMK=yes where it can be used to speed things up.
Thanks to Stephen Montgomery-Smith <stephen@math.missouri.edu>
6. Be smarter about not starting read_distinfos in the background if
there is no need to, and be more aggressive in killing it off when
we're done.
7. Make read_distinfos quieter to avoid printing confusing messages
out of synch with the foreground process.
Changes
=======
1. Use the PKGREPOSITORY directory for all packages, create it if it
doesn't already exist.
2. If we fail(), call one of {trap|safe}_exit to do a better job of
cleaning up after ourselves.
3. Make failures of various commands silent when we provide our own
error message.
4. Make killing background processes generally much quieter.
Bug Fixes
=========
1. In check_for_updates(), when FORCE is on, don't skip an up to date
port.
2. Change MAKE_ARGS to PM_MAKE_ARGS to avoid collisions with ports that
use the former already.
3. Fix grammar in a usage() line, add details for --show-work, and
other various improvements.
4. Move the clean target to after (optional) 'make package' to avoid
having to rebuild the port.
5. Only try to update the +CONTENTS file if it exists.
============
1. --show-work option which will recurse through all-depends-list
and show what dependencies are, and are not installed.
2. Dramatically reduce false positives for per-port distfile cleaning
by creating a list of valid distfiles for all ports. Use this new
method for the --clean-distfiles* options as well.
Bug Fix
=======
Honor DISTDIR if set for --clean-distfiles*
============
If a user were to specify /var/db/pkg/foo-1.2/ on the command line,
the old code already failed, but by telling people that it's ok to
do that now, I exposed one more place this needed to be fixed. Take
the opportunity to fix this in a more thorough way.
===========
--clean-distfiles[-all] options to check /usr/ports/distfiles for
stale stuff, and either offer to delete them or delete them all
without prompting.
Bug Fix
=======
Fix brain-o that broke the -o option.
Usability Improvements
======================
1. If a port skeleton is missing, but the pkg db has an +IGNOREME
file, skip the update check, but issue a warning for it.
2. If there is no +IGNOREME file, and no version information is
present, fail with a helpful error message.
3. Clean up any trailing / character on directory names input
at the command line.
4. Allow user to specify just category/portname on the command
line without -p.
5. Work around a missing +REQUIRED_BY file for a -r port.
Minor cleanup
=============
Make case consistent for all usage() items
============
1. Add -t mode to "thoroughly" recurse dependencies, using the
all-depends-list target instead of the combination of the
build-depends-list and run-depends-list targets. This is useful
when the dependencies of a dependency need to be updated, but
the dependency itself doesn't; and -a mode is impractical for
whatever reason.
2. If pkg_create fails, give the user the option of how to handle
it: fail, or ignore the error.
Improvements
============
1. If we are trying to delete distfiles, but a dist subdir has
gone missing, assume that the files are safely deleted (and
inform the user) rather than error'ing out.
2. Search for BROKEN state before FORBIDDEN state, since if
the former is set, the latter is implied, which made the error
message confusing. While I'm here, trim the error message by
removing some redundant information.
3. After successful install, before the dependencies are updated,
apply the same search pattern whether we're updating an existing
port, or installing from scratch using -p /usr/ports/foo/bar.
Otherwise, existing dependencies for a port that was forcibly
pkg_delete'd won't be updated, nor will the newly installed
port's +REQUIRED_BY file be up to date.
Fix
===
1. The file that contains the checksums is not always spelled
"distinfo," so use the MD5_FILE variable to find it
Cleanups
========
1. Further local'ize variables in functions, and factor in some
variables there were used only once. Add more comments that
describe usage of variables with global scope used in a function.
2. Twiddle white space a little more to help key messages stand
out better, and change wording on one message to (hopefully) make
it more clear.
3. Bump copyright
============
1. Attempt to avoid the problem of multiple background checksum
processes stepping on each other trying to download the same sets of
distfiles. The most pathological case for this is trying to portmaster
the xorg port on a clean box with no distfiles downloaded. (Brought to
my attention by George Hartzell <hartzell@alerce.com>.)
2. In cooperation with sem and skv, add three new environment
variables that can potentially be used by port authors, etc.
UPGRADE_TOOL contains the name of the tool being used, in this case
'portmaster'. UPGRADE_PORT is the full string for the port that we are
currently upgrading, and UPGRADE_PORT_VER is just the version string
part of that.
3. Collect a list of any pkg-message files that we "saw" while doing
the installs, and display them all at the end when everything is done
installing. (Old idea of mine, and others, most recently prompted by
kris.)
4. You can now do 'portmaster .' if you are in a port directory you
want to install.
5. If there is no DISTFILE information in the +CONTENTS file after
an installation, add it in the manner worked out on freebsd-ports@,
and documented in http://www.freebsd.org/cgi/query-pr.cgi?pr=106483.
If that information is present, use it to delete the last known set
of distfiles with certainty, but continue searching the "old" way
until the DISTFILE stuff has been around for a while.
Major Fixes
===========
1. Change the code that reads the MOVED file to recurse through the
whole file. This fixes the problem of getting an "answer" to a
find_moved_port() call only to find out that the "new" port has itself
been moved, or deleted. This is not a major problem, but when it
happens it's ugly. The fix slows down the function by about 1/2 second
on my (fairly fast) box, but that function isn't called often, and
it's better to be thorough. (Brought to my attention by erwin.)
2. Remove a change from the last version that caused way too many
false positives from the MOVED file when there was no installed
version of the port.
3. When running -e, warn the user if there are dependencies on the
port, and improve the error message both here and in the same code
for -s.
Minor Cleanups
==============
1. Assign TMPDIR early in the script, and export it to be safe. Change
occurrences of this variable to use the short form.
2. Factor some code that had been moved to a function back in line
since we're down to just one caller.
3. Only print the verbose message about keeping the current version
of a distfile if we're not in fact deleting all distfiles.
4. Make distfile listing more efficient by using the ALLFILES target
instead of the combination of DISTFILES and PATCHFILES.
=============
1. You can now specify multiple ports to upgrade on the command line,
such as 'portmaster foo-1.23 bar-4.56 baz-7.98'. [1] Pass -n argument
to child processes (as needed) to handle this case.
2. Handle an alternate port that can satisfy a build requirement by
reading the CONFLICTS for the "standard" port, and seeing if we have
one of those installed. [1]
3. Unless the user specifies the new -B option, always build a backup
package when updating an existing port. Unless the user has specified
the -b option, delete the package on successful installation of the
new port, or print a helpful message telling the user where they can
find the package if install fails.
4. Don't only test a port for IGNORE, also test FORBIDDEN and BROKEN
to avoid doing a lot of work on dependencies for a port that we aren't
going to install.
Big Improvements:
=================
1. Significantly enhance the -r option by tracking what ports should
be rebuilt as a result of it, and rebuilding them "in line" if they
are dependencies of other ports that should also be rebuilt. Thus,
make sure that these ports get rebuilt once, and only once. Also,
don't rely on just the installed port's +REQUIRED_BY file to get the
list of dependencies, since it may be out of date. Search the pkg tree
for that port's DEPORIGIN to make sure that we get them all.
2. Track answers to ports that have +IGNOREME files, so the user is
not asked twice.
3. Rather than use all-depends-list to handle dependencies, use a
combination of build-depends and run-depends. This seems to get
everything we actually need, without sucking in a lot of stuff we don't.
4. Stop abusing the config_only mode, and make a clear separation
between first run (if any) and build run that does not depend on it.
This makes -G mode work as intended, with no bad side effects. [1]
Small Improvements:
===================
1. Add PREFIX/sbin to the PATH [1]
2. Cache "no" answers when using -i ("yes" was already cached).
3. If pkg_version thinks that two versions are the same even though
they have different pkg names, don't warn the user.
4. Clean up package code a little.
5. Twiddle the "waiting on" message for fetch & checksum.
6. If a user has a stale +REQUIRED_BY file in a pkg directory, print a
helpful message that suggests how to fix it. [2]
7. Don't tell a user about a -r port to rebuild if we're not going to
rebuild it for whatever reason.
Bug Fixes:
==========
1. Clean up trailing white space.
2. If the pkg data is corrupt, a search for installed port by ORIGIN
could return more than one answer. So, use only the first answer.
3. In dependency_check(), if a port has moved, check the new location
to see if it needs updating.
4. If there are no valid ports to build based on the command line
args, don't try to build /usr/ports/ [3]
[1] Suggested, debugged, and generally helped greatly by mezz
[2] Wondered about by yar
[3] Reminded by Bill Blue
===========
If a port is marked IGNORE, don't do all the dependency/checksum work,
instead bomb out with a helpful message
Minor Fixes
===========
1. Add some whitespace to make the "dependency check complete" message
stand out better
2. Tell the user when we start pkg_delete in -s mode, just like in -e
Bug Fixes
=========
1. Export MAKE_ARGS so that the child checksum processes can see them
2. Move the "just in case" cd back to $portdir up one line so it's
before 'make clean', just in case
============
1. -e mode to expunge a port via pkg_delete, and optionally
remove its distfiles
2. -s and -e modes now respect -b (backup) and -d|D options
regarding always deleting (or not deleting) distfiles
General
=======
1. Re-factor more code out into functions, and move them earlier in the
script so that they are available to the new features.
2. Clean up the code in a few places to make it slightly more efficient,
and a lot more readable.
Bug Fixes
=========
1. Fix exit status of various code paths to be more in keeping
with Unix tradition [1]
2. Be a lot more thorough about killing off errant children when
the user hits ^C [1]
3. If the directory referred to by PKGREPOSITORY exists, use it
to store packages built for -b and -g instead of HOME [2]
My thanks to both Darren and James for their patience and testing
of several development versions to nail down bugs 2 and 3.
Brought to my attention by: Darren Pilgrim <darren.pilgrim@bitfreak.org> [1]
Brought to my attention by: James O'Gorman <james@netinertia.co.uk> [2]
============
1. -L mode to list all installed ports, and search for updates.
2. Trap SIGINT, and use the opportunity to reap all the child processes
we can find.
3. Warn the user during config mode if a port sets IS_INTERACTIVE
4. Support +IGNOREME files in /var/db/pkg/<portname>, ala portupgrade
5. Store the results of the user's choices for -i during config, so they
don't have to enter it again during build
6. When deleting stale distfiles, recurse through successively shorter
versions of file-name-* till we find something to delete, or run out
of dashes. This means a lot fewer false positives, although the
algorithm is still aggressive.
Performance Optimizations
=========================
1. Skip dependency check prior to build if the 'make config' recursion
does not find any ports that need updating.
2. Cache the dependency list for the parent port.
3. -s mode is now recursive, so if a removed port has dependencies,
they are checked too. Remember what ports the user does not want to
delete so they don't have to keep saying "no."
4. Much better support for failed downloads in the background, and the
'make checksum' process is only run once for each port.
5. Switch to using variables and case rather then grep to maintain state
for current dependencies, answers for -i, etc.
6. Re-order some tests to micro-optimize do_update, etc.
7. Handle regular files in /var/db/pkg more efficiently
8. Track the pid of the parent process, and use that to manage state
more efficiently. Also makes for more readable code.
9. Don't clean dependencies, since we don't recurse through them using
the ports tree, we'll let each portmaster process handle it.
10. Cache information about dependencies that are already up to date
to avoid having to do this check when not necessary. This dramatically
improves performance for building/updating large ports (especially
meta-ports like gnome or kde) with a lot of dependencies.
Bug Fixes
=========
1. Add -f to pkg_delete in -s case
2. Reverse the "is required by" statements to make more sense for -r
3. Add PATCHFILES to the list of distfiles to delete when looking for
stale distfiles
4. Minor cleanups, and better code consistency
5. usage() already exits, so clean up h) getopt
6. Add a catchall getopt so user will get usage() for an invalid option
7. Update some messages to make them more meaningful and simplify code
8. Fix specifying second option to -o in <category/portname> format
9. Properly handle the case where the second argument to -o is a port
that is already installed
10. Don't mention the 'make config' step if -G
Security Enhancement
====================
1. Be more careful to only delete temp files if they haven't already been
deleted or moved
