- Introduce FETCH_USING, which specifies the mechanism to use; possible
values are ftp, fetch, curl, wget, manual and custom.
- Depend on the correct tool/program if not using manual or custom.
- For manual, just use /usr/bin/false to bail out if a distfile is
missing.
- For custom, FETCH_CMD and related variables are used as before.
- Default value is ftp.
It was uncompleted and now service suspended, noticed by Zafer Aydogan.
While here, update MSTER_SITE_MOZILLA taken from mirrors list:
http://www.mozilla.org/mirrors.html
version of tnftp, otherwise it will use net/tnftp. Require ftp as
bootstrap dependency when the package has files to fetch.
The dependency is currently unconditional and FETCH_CMD is set to
TOOLS_PATH.ftp by default.
Make _ALL_FILES and in turn _CKSUMFILES available earlier.
Use it to only depend on digest if _CKSUMFILES is non-empty.
NO_CHECKSUM will now only skip the checksum and checksum-phase
targets. FAILOVER_FETCH will independently check the sums on
distfiles and depend on digest, fixing the remaining issues
originally raised in PR 34914.
phase pkg_install-depends before bootstrap-depends that just tries to
install a new pkg_install if the current version is too old. Still
keep it as bootstrap dependency for the bulk build code.
For NetBSD, PKG_TOOLS_BIN has to be computed in shell code due to a make
bug.
OK: jlam@
There are three types Mozilla mirrors.
(http://www.mozilla.org/mirroring.html)
* mozilla-current
contains only the current version of Firefox and Thunderbird
* mozilla-release
contains Firefox, Thunderbird, and Sunbird releases
* mozilla-all
complete archive
Define following variables for mozilla master sites:
MASTER_SITE_MOZILLA_ALL = mozilla-all
MASTER_SITE_MOZILLA = mozilla-release
and change some packages to use appropriate variable.
Update contents of MASTER_SITE_MOZILLA with master and primary mirrors
taken from http://www.mozilla.org/mirrors.html and add some sample definitions.
PATCHFILES nor DISTFILES, make fetch would complain about an unbuildable
dependency. Fix this by explicitly checking that _ALLFILES is not empty.
Reported by Sean Boudreau.
in missing distfiles retrieving www.nic.funet.fi.file_not_found.html instead
of failing and carrying on to the next site.
Use FTP instead which does the right thing.
that its ${DISTFILES} is also a common ${PATCHFILES}, resulting in
the file being listed twice in ${_ALLFILES}. This one-line change
removes duplicate filenames from ${_ALLFILES}, mirroring an identical
transformation to ${ALLFILES} that used to do the trick. (The fetch
code, obviously, has since changed a bit.)
What the noise looked like:
$ cd pkgsrc/mail/qmail
$ make clean PKG_OPTIONS="netqmail"
make: "/home/build/trees/pkgsrc-current/mk/fetch/fetch.mk" line 133: warning: duplicate script for target "/home/build/trees/pkgsrc-current/distfiles/netqmail-1.05.tar.gz" ignored
make: "/home/build/trees/pkgsrc-current/mk/fetch/fetch.mk" line 133: warning: using previous script for "/home/build/trees/pkgsrc-current/distfiles/netqmail-1.05.tar.gz" defined here
===> Cleaning for qmail-1.03nb12
$
Tested in a bulk build by joerg@.
in the format MASTER_SORT_REGEX expects when MASTER_SORT_RANDOM is off,
in order for MASTER_SORT to actually work when MASTER_SORT_RANDOM=YES
(default nowadays)
XXX testing stuff before committing would surely be a good idea
lead to overloads of very first distribution site. Moreover, if first
site in the list is not available (often seen for sourceforge mirrors)
you have to wait for timeout each time. To distribute load on master
distribution sites and to make second problem not so annoying randomly
intermix list of MASTER_SITES with MASTER_SORT_RANDOM feature. Any of
MASTER_SORT and MASTER_SORT_REGEX can be applied later.
The feature is turned ON by default and is disabled for PKG_DEVELOPERs
or if MASTER_SORT_RANDOM=no.
fetch files with an empty name, for example "geda/".
The problem was that the expression ${LIST:S,^,${DIST_SUBDIR}/,} results
in ${DIST_SUBDIR}/ when ${LIST} is empty. This is surprising but matches
the documentation in the manual page, so this cannot be called a bug.
The proper fix is to use ${LIST:@f@${DIST_SUBDIR}/${f}@} instead.
Noticed by Don Woodstock on #netbsd-code.
environment ${PKGSRC_MAKE_ENV} is also passed along. Create a
convenience variable RECURSIVE_MAKE that does exactly this and that
can be used in place of MAKE when invoking make recursively.
Use RECURSIVE_MAKE everywhere in pkgsrc/mk that we invoke make
recursively.
If ${FILESDIR}/getsite.sh exists, then use it to determine the fetch
URL for each of the distfiles for the package. Otherwise, use
SITE_<file> and MASTER_SITES, in order, to determine the URL for each
distfile.
If the script path differs from ${FILESDIR}/getsite.sh, then set
DYNAMIC_SITE_SCRIPT to the full path to that script.
Remove the need to set DYNAMIC_MASTER_SITES explicitly in the package
Makefile for:
graphics/ns-cult3d
wm/sawfish-themes
www/apache-tomcat55
www/jakarta-tomcat4
www/jakarta-tomcat5
currently exist. This triggers a bug in the Bourne shell of at least
DragonFly and Irix, when set -e is also active. Remove set -e to avoid
this. Discussed with jlam@, shown by DragonFly bulk builds, complains on
DragonFly's user list and PR 34036.
A test case for this shell bug is, which should return 0:
set -e
testf () {
test -f /nonexistent || return 1
return 0
}
if testf; then
exit 1
fi
exit 0
in do-fetch-file (formerly they were inserted by the old _FETCH_FILE
inserting some quotes that are no longer added by the fetch command list
macro). This makes fetching with MASTER_SORT* set work again.
* All the smarts is now encapsulated in the "fetch" script. The fetch
script understands how to use the distinfo file (if specified) to
look up the size and checksums of the file to fetch and will use
that information to verify checksums of the fetched files or resume
transfers of interrupted fetches.
* Move the default settings for FETCH_RESUME_ARGS and FETCH_OUTPUT_ARGS
for "ftp" from mk/defaults/mk.conf into mk/fetch/fetch.mk. We rewrite
it to avoid needing conditional statements.
* Avoid spawning a new make(1) process just to mirror a distfile.
* Split out fetch-list targets into a separate file fetch-list.mk.
These targets should probably be moved into a standalone script.
* Fix distclean target to properly remove partial downloads.
fetch.mk. This script currently completely replaces the functionality
in _FETCH_FILE. I will eventually add the ability to resume a file
transfer to this script.
"fetch" target, but it does affect the "checksum" target's shell
script, which errors out on the second occurrence of a file. The
shell script should perhaps also be fixed, but it seems sensible
regardless for ${ALLFILES} not to contain duplicate filenames. As
a side effect, the file list is sorted.
Regression found by building mail/qmail with the "qmail-netqmail"
option, which adds to PATCHFILES a file that's already in ${DISTFILES}.
Arguably this is gross, but it worked before, and now works again.
Tested on my usual pkg_comp(8) build of 200+ packages, with an
initially empty ${DISTDIR} and ${PACKAGES}. Thanks seb@ for the
more idiomatic make(1) construction.
and those that are defined by the infrastructure (_BUILD_DEFS). This
allows the build-defs-message target to be moved to the end of
bsd.pkg.mk. Now it prints the correct result even in unprivileged
builds, which had been wrong due to the order in which the files have
been included. For example, ${UNPRIVILEGED_USER} was displayed as (not
defined) although its value was defined, which could be checked with
"bmake show-var".
Tested with one package that _does_ define BUILD_DEFS and with one that
doesn't. The behavior stays the same.
stages, and that installs dependencies listed in BOOTSTRAP_DEPENDS.
The bootstrap-depends step works just like the normal depends step
and honors the value of DEPENDS_TARGET. It's now possible to add
dependencies solely to facilitate fetching the distfiles, e.g.
BOOTSTRAP_DEPENDS+= curl-[0-9]*:../../www/curl
* Teach the tools framework about ":bootstrap" as a tools modifier
which indicates the tool should be added as a dependency via
BOOTSTRAP_DEPENDS.
* Add "digest" to the tools framework.
* Use USE_TOOLS+=digest:bootstrap to force pkgsrc to install digest
before anything else. Get rid of unused "uptodate-digest" target
and related digest version-checking code.
* Finish the refactoring work: split checksum-related code out of
bsd.pkg.mk and into pkgsrc/mk/checksum and replace the "checksum"
target command list with a script that does all the real work.
* Make DIGEST_ALGORITHMS and PATCH_DIGEST_ALGORITHM into private
variables by prepending them with an underscore. Also, rename
_PATCH_DIGEST_ALGORITHM to _PATCH_DIGEST_ALGORITHMS and adjust the
makepatchsum target to allow that variable to contain a list of
algorithms, all of which are used when creating the patch checksums
for ${DISTINFO_FILE}.
of the logic from fetch/fetch.mk into flavor/pkg/check.mk, so that
check-vulnerable can be used as a source target.
Make check-vulnerable a source target for every phase of the build
workflow, which ensures that it is always run if the user starts a
new phase from the command line.
Fix the cookie-generation targets so that they don't append, only
overwrite to the cookie file. This works around potential problems
due to recursive makes.
Move the cookie checks so that they surround the corresponding phase
target. The presence of the cookie should now inform the make process
to avoid doing any processing of phases that occur before the phase
corresponding to the cookie.
bsd.pkg.mk. They didn't actually need to be defined in bsd.prefs.mk,
just somewhere before the "main" bsd.<phase>.mk files were included.
This moves some conditional (?=) definitions back into bsd.pkg.mk so
they won't conflict with any conditional definitions in package
Makefiles.
This should fix the "checksum" problems in lang/php-gd as noted here:
http://mail-index.netbsd.org/pkgsrc-users/2006/06/05/0012.html
where EXTRACT_SUFX had the wrong value due to the order in while *.mk
files were included.
subdirectories of pkgsrc/mk. Move the following files around for
locality:
pkgsrc/mk/scripts/extract -> pkgsrc/mk/extract/extract
pkgsrc/mk/bsd.sites.mk -> pkgsrc/mk/fetch/sites.mk
Also get rid of the recursive make for the "fetch" and "extract"
targets. This basically merges the "fetch" and "extract" phases into
the "patch" phase.
There is still much more work to do to simplify the fetch code, but
this is a good start.
