resources when fetchers wouldn't be able to save what they download. On
my build farm with pkgsrc (and pkgsrc/distfiles) mounted over NFS,
however, the check gives false positives for NetBSD 9.2 and -current.
Downgrade it to a warning so that these fetches can succeed, while
leaving a breadcrumb in case someone encounters a true positive.
It would of course be interesting to sort out why, in my environment, a
wide variety of other OSes get 1 for "${TEST} ! -w $fetchdir" while
NetBSD gets 0. In the meantime, joerg@ suggested this workaround and
gdt@ agrees it's reasonable to try.
The previous shell script version's runtime was quadratic against the
number of distfiles to verify. Historically this has not been an issue,
with usually only a handful of files per package. However, with the
introduction of Go modules the number of distfiles used by a single
package can be very high.
For example, in an upcoming update of www/grafana to version 7.1.5, the
number of GO_MODULE_FILES is 821. Running 'bmake checksum' takes:
real 18m20.743s
user 17m27.975s
sys 0m49.239s
With the awk code, this is reduced to a far more sensible:
real 0m4.330s
user 0m3.241s
sys 0m0.875s
The script has been written to emulate the previous version precisely,
preserving the same output and error messages and supporting all of its
behaviour, with the one exception that previous exit values of 128 have
been changed to 3, in order to avoid any potential signed 8-bit issues.
The one change in the pkgsrc infrastructure is that the mk/fetch/fetch
script no longer sets a working default value for ${CHECKSUM}. This is
not a problem in a pkgsrc environment as all of the required variables
are set correctly, but if there happen to be any users who are using
this script in a standalone environment, they will need to set it
accordingly. This was probably required in many situations previously
anyway, as none of the script's environment variables were set, and
trying to support this would be fragile at best.
This is based on the decision The NetBSD Foundation made in 2008 to
do so, which was already applied to src.
This change has been applied to code which is likely not in other
repositories.
ok board@, reviewed by riastradh@
as an acceptable input.
Fixes Pkgsrc for users that set PKG_RESUME_TRANSFERS=yes in their mk.conf if
installing a package that has an interactive fetch stage, after
mk/fetch/fetch.mk revision 1.66.
Tested for all combinations of:
+ PKG_RESUME_TRANSFERS=yes/no; and
+ interactive fetch=yes/no; and
+ distfile=downloaded/not downloaded
... with no signs of misbehaviour.
The specific case that was broken, and this change fixes, is:
+ PKG_RESUME_TRANSFERS=yes; and
+ interactive fetch=yes; and
+ distfile=downloaded
This change was designed with the following considerations:
1. Given it's a freeze, keep it simple.
2. The change in behaviour allows 'fetch' to succeed where it would previously
fail fatally, so it's unlikely to affect any (intentional) existing
behaviour in Pkgsrc.
3. The behaviour of 'fetch' with zero sites is essentially the same as a
fetch where all the possible download sites fail, I.e. Pkgsrc already
expects to handle such behaviour.
ok gdt@
currently exist. This triggers a bug in the Bourne shell of at least
DragonFly and Irix, when set -e is also active. Remove set -e to avoid
this. Discussed with jlam@, shown by DragonFly bulk builds, complains on
DragonFly's user list and PR 34036.
A test case for this shell bug is, which should return 0:
set -e
testf () {
test -f /nonexistent || return 1
return 0
}
if testf; then
exit 1
fi
exit 0
* All the smarts is now encapsulated in the "fetch" script. The fetch
script understands how to use the distinfo file (if specified) to
look up the size and checksums of the file to fetch and will use
that information to verify checksums of the fetched files or resume
transfers of interrupted fetches.
* Move the default settings for FETCH_RESUME_ARGS and FETCH_OUTPUT_ARGS
for "ftp" from mk/defaults/mk.conf into mk/fetch/fetch.mk. We rewrite
it to avoid needing conditional statements.
* Avoid spawning a new make(1) process just to mirror a distfile.
* Split out fetch-list targets into a separate file fetch-list.mk.
These targets should probably be moved into a standalone script.
* Fix distclean target to properly remove partial downloads.
fetch.mk. This script currently completely replaces the functionality
in _FETCH_FILE. I will eventually add the ability to resume a file
transfer to this script.
