The previous shell script version's runtime was quadratic against the
number of distfiles to verify. Historically this has not been an issue,
with usually only a handful of files per package. However, with the
introduction of Go modules the number of distfiles used by a single
package can be very high.
For example, in an upcoming update of www/grafana to version 7.1.5, the
number of GO_MODULE_FILES is 821. Running 'bmake checksum' takes:
real 18m20.743s
user 17m27.975s
sys 0m49.239s
With the awk code, this is reduced to a far more sensible:
real 0m4.330s
user 0m3.241s
sys 0m0.875s
The script has been written to emulate the previous version precisely,
preserving the same output and error messages and supporting all of its
behaviour, with the one exception that previous exit values of 128 have
been changed to 3, in order to avoid any potential signed 8-bit issues.
The one change in the pkgsrc infrastructure is that the mk/fetch/fetch
script no longer sets a working default value for ${CHECKSUM}. This is
not a problem in a pkgsrc environment as all of the required variables
are set correctly, but if there happen to be any users who are using
this script in a standalone environment, they will need to set it
accordingly. This was probably required in many situations previously
anyway, as none of the script's environment variables were set, and
trying to support this would be fragile at best.
This is based on the decision The NetBSD Foundation made in 2008 to
do so, which was already applied to src.
This change has been applied to code which is likely not in other
repositories.
ok board@, reviewed by riastradh@
and checksum scripts.
Fixes the problem where DISTINFO_FILE=../../foo/bar/distinfo doesn't
work without having an extraneous and ugly ${.CURDIR} stuffed into it
by hand.
the SHA512 digest to the mix of digests we keep for each distfile.
All part of providing stronger digests for pkgsrc, as discussed on
tech-pkg recently, with unanimous agreement. There will be further
changes in this area in the near future, as we transit away from
reliance on SHA1 and RMD160.
New distinfo files will gain a SHA512 digest entry. Existing
verification of distinfo files will just use the SHA1 and RMD160
digests which exist right now.
Mac OS X Mountain Lion's "sed" will otherwise reject some patch files
(e.g. "pkgsrc/devel/libcfg+/patches/patch-ab") because of broken
UTF-8 encoding.
It would probably be better not to use the bundled "sed" under
Mac OS X Mountain Lion at all. But it seems that this is not
supported by "pkgsrc" at the moment.
This change modifies the algorithm used to keep track of the files that
have not yet been checksummed to use a simple loop instead of shell pattern
matching.
For packages with few distinfo entries, either way yields the same result
as the list of files to check is very short. But for those packages with
hundreds of distinfo entries (vim, I'm looking at you), the difference is
huge. In my old macppc machine, the checksum of vim used to take around
40 minutes and now it takes ~35 seconds. The difference is also clearly
visible in my faster amd64 machine (although I haven't bothered to time it).
Make _ALL_FILES and in turn _CKSUMFILES available earlier.
Use it to only depend on digest if _CKSUMFILES is non-empty.
NO_CHECKSUM will now only skip the checksum and checksum-phase
targets. FAILOVER_FETCH will independently check the sums on
distfiles and depend on digest, fixing the remaining issues
originally raised in PR 34914.
target by pkgsrc. This new target computes checksums only up until
the extract phase is complete.
The "checksum" target is now a target that can always be run by the
user at any time, regardless of the presence of the work directory.
These changes were proposed in PR pkg/36603 by Robert Elz.
provided in the distinfo file for those distfiles, rather than checking
specifically for the ones listed in _DIGEST_ALGORITHMS. If a distinfo
file actually provides checksums for other algorithms, e.g. TIGER,
then go ahead and verify them too.
run now at any time up until the "extract" phase completes, and the
"checksum" target no longer keeps a state (cookie) file.
From the checksum.mk file comments:
We do not actually wish to create a cookie file for the completion
of the checksum phase because we want to be able to detect if
any of the files have changed right up until the distfiles are
extracted. Therefore, we use the presence of the cookie file
from the "extract" phase to determine whether we need to continue
to verify checksums.
This addresses PR pkg/36262 and PR pkg/36603.
a distfile or a pkgsrc patch. It's simple, cleaner, less magic, etc.
(duh!).
While here, I notice that _CHECKSUM_CMD was already being defined in
mk/checksum/checksum.mk (as it should be), so update the definition
from mk/patch/patch.mk and remove it from patch.mk.
first stripping them of NetBSD RCS ID tags. Use the checksum script
in the patch module to verify patch checksums instead of hand-coding
a miniature version of the checksum script in the do-pkgsrc-patches
target.
distfiles' checksums once and only once, between fetching and extracting
them.
The DO_CHECKSUMS_REPEATEDLY variable can be set to "yes" by those who
need the checksums checked more than once.
the old bsd.pkg.mk behavior (noted by adrianp in private email). Keep
"makedistinfo" as an alias for "distinfo".
While here, sprinkle some .PHONY declarations for correctness.
This fixes problems where a package sets PKG_*_REASON, which causes
bsd.pkg.mk to define its own "checksum" replacement, which causes a
"duplicate script" make error to occur.
* Avoid shell differences between /bin/sh and Korn shell by using:
while read line; do list; done < FILE
instead of
cat FILE | while read line; do list; done
stages, and that installs dependencies listed in BOOTSTRAP_DEPENDS.
The bootstrap-depends step works just like the normal depends step
and honors the value of DEPENDS_TARGET. It's now possible to add
dependencies solely to facilitate fetching the distfiles, e.g.
BOOTSTRAP_DEPENDS+= curl-[0-9]*:../../www/curl
* Teach the tools framework about ":bootstrap" as a tools modifier
which indicates the tool should be added as a dependency via
BOOTSTRAP_DEPENDS.
* Add "digest" to the tools framework.
* Use USE_TOOLS+=digest:bootstrap to force pkgsrc to install digest
before anything else. Get rid of unused "uptodate-digest" target
and related digest version-checking code.
* Finish the refactoring work: split checksum-related code out of
bsd.pkg.mk and into pkgsrc/mk/checksum and replace the "checksum"
target command list with a script that does all the real work.
* Make DIGEST_ALGORITHMS and PATCH_DIGEST_ALGORITHM into private
variables by prepending them with an underscore. Also, rename
_PATCH_DIGEST_ALGORITHM to _PATCH_DIGEST_ALGORITHMS and adjust the
makepatchsum target to allow that variable to contain a list of
algorithms, all of which are used when creating the patch checksums
for ${DISTINFO_FILE}.
