assumptions being made by the USE_PKGLOCALEDIR code and the wrapper
framework since it added backtick expressions to the SUBST_FILES.*
variables, which were being mangled by the :Q modifier. This is
evident when running "make regress" in regress/buildlink-unwrap.
Mea culpa.
As data for a given binary package is loaded, increment a count
stored with PKGPATH as the index. Then use a psuedo-multidimensional
array with PKGPATH and the # as the index. This allows for much
faster lookups and scales linearly with the number of packages.
The old way scaled as the product of the number of pkgsrc entries
and the total number of binary packages. Not a pretty value.
This makes the actual README.html generation part run about 3x faster.
Approach suggested by soda.
${WRKSRC}. Just directly create the msgfmt wrapper in the proper
target. Also, note that the msgfmt handling should eventually migrate
to the tools framework so that build dependencies and binary paths are
correct.
some of the older packages on ftp.netbsd.org. For example,
pkg_info -B ftp://ftp.netbsd.org/pub/NetBSD/packages/1.5.2/vax/All/cascade-1.4.tgz
will give
OPSYS= NetBSD
instead of
OPSYS=NetBSD
due to a directory permissions error, then error out.
- when the master cache has just been newly created, don't also report
that the master cache is up to date.
Both were suggested by Christian Hattemer.
Plan:
(1) Change USE_PERL5=build into USE_TOOLS+=perl.
(2) Change all other USE_PERL5 into including perl5/buildlink3.mk.
Possibly, for packages that don't actually build anything with perl,
but merely require it for the perl interpreter, we can instead do:
USE_TOOLS+= perl
TOOLS_DEPMETHOD.perl= DEPENDS
but this is more verbose than simply including the perl5/buildlink3.mk
file.
Move the PERL5_REQD computation into a lang/perl5/version.mk file,
and only do the USE_PERL5 logic in bsd.pkg.use.mk if we're not using
the new tools framework. This consolidates all of the perl-handling
into two places -- lang/perl5 and mk/tools/perl.mk.
- completely redo the code which decides on the machine architecture,
operating system, and operating system version for the binary packages.
The old way just used to directory names to take a guess. The new
way creates a cache file containing meta-data for all the binary packages
in each "All" directory. This cache file is consulted when generating
the lists of available binary packages. The meta-data is obtained with
pkg_info so it should always be correct even if you do something silly
like mix OS_VERSION or MACHINE_ARCH packages up in the same directory.
Among the benefits are: works when PACKAGES is not $PKGSRC/packages,
works with a more or less arbitrary subdirectory structure, works
when there are subdirectories for multiple operating systems.
This portion of the fix should address PR25390.
The cache files are only updated when the contents of an "All" directory
changes or if the cache file format changes. There is some room for
improving the updating of the cache files, but its not too bad the way
it is.
- fix up some of the awk code so that generadme.awk works with Solaris
nawk as well as NetBSD's nawk and gawk (for pre-2.0 systems).
- remove some "if ! foo" shell constructs to increase portability.
- be more consistent with what variables get passed to mkreadme from
make and which ones are determined automatically. Mostly this meant
moving stuff into mkreadme to make it easier to run it standalone.
directory or the local patches directory exist. Also, emit the message
informing the user that pkgsrc patches are being applied that was lost
in the previous commit.
apply-distribution-patches and apply-pkgsrc-patches. Rewrite the
latter code to be more "whitespace-friendly" and to not need to invoke
${AWK} as many times. Drop support for PKGSRC_SHOW_PATCH_ERRORMSG
and simply always show the error message.
GNU tar archives. Create a new EXTRACT_USING value "nbtar" that causes
tar/ustar archives to be extracted using pax-as-tar, which understands
most GNU tar extensions. Default to EXTRACT_USING=nbtar, as quite a
few packages are distributed in GNU tar archives.
right in each variable, starting with the most generic one. Later
definitions override earlier ones. This makes the ``order'' testcase in
regress/pkg-options succeed.
which can take multiple values -- "pax" or "gtar". The default value
of EXTRACT_USING is "pax", which more closely matches reality since
before, we were using bootstrap "tar" for ${GTAR} and it was actually
pax-as-tar. Also, stop pretending pax-as-tar from the bootstrap kit
or on NetBSD is GNU tar. Lastly, in bsd.pkg.extract.mk, note whether
we need "pax" or "gtar" depending on what we need to extract the
distfiles.
"bison" and "bison-yacc". bison is just a symlink to the real bison,
but "bison-yacc" provides a "yacc" in ${TOOLS_DIR} that does "bison
-y". This allows the tools framework to provide everything that a
package might look for when asking for "bison" and avoids relying on
the bison being available in the PATH.
Ditto for mailx. Lastly, /usr/bsd/zcat isn't gzcat (doesn't understand
gzipped files), so don't use it as a gzcat replacement. Changes from
Georg Schwarz in private email.
the final man pages to be compressed. If MANZ is not defined, then
we want the final man pages to be uncompressed.
We need to figure out if during installation, we need either gunzip
or gzip to decompress or compress the installed man pages. If a
package sets MANCOMPRESSED to "yes" or "no", then it's an indication
to the install code that the package itself installed the man pages
either compressed or uncompressed. If a package sets
MANCOMPRESSED_IF_MANZ, then the package uses BSD-style makefiles, so
we need to determine if the BSD-style makefile causes the man pages
to be compressed or not. In this case, we need to check in PKGMAKECONF
whether MANZ is set or not.
XXX Perhaps we should also check for MKMANZ if a package uses BSD-style
XXX makefiles?
duplicates in the output, output warnings if necessary. The $$dlist at
the end is unquoted intentionally to omit spaces in the output.
Approved by jlam.
as needing both (no impact since they're both satisfied by the tar
binary installed by the bootstrap kit). There's some funniness in
the extraction code where we check for pax or GNU tar that needs to
be resolved. Remove the TAR=${GTAR} hack since it's no longer needed
after these changes.
XXX Later, a sweep needs to be made to see where we actually need GTAR
XXX and where we only need TAR, probably triggered by whether we call it
XXX with the "z" option or not. Packages that need GTAR should explicitly
XXX add USE_TOOLS+=gtar to the package Makefile.
separate file pkgsrc/mk/bsd.pkg.patch.mk. Also, include this file
ahead of the inclusion of bsd.tools.mk so that tools required to
patch are created by the tools framework.
other files without making assumptions about the location of the top of
the pkgsrc tree. This fixes problems with running "make bulk-cache"
in /usr/pkgsrc (PR pkg/30288).
at the time that they're defined: ${AWK}, ${GREP}, ${SED}, ${TRUE}.
Move the inclusion of compiler.mk below the inclusion of bsd.tools.mk
so that these variables are properly defined by the time they are
used. This should fix problems where pkgsrc gcc was not being properly
detected when using the new tools framework.
This change has the side effect of changing the relative order of the
compiler directories and the tool directory in the PATH, but this has
no impact on existing packages, and actually makes more sense anyway.
check for whether ${PATCHDIR} exists or not, or whether PATCHFILES is
defined or not. This avoids the use of != in a variable definition
just to find out if we need patch or not.
is undefined. This avoids many unnecessary make(1) warnings. As this file
is only intended to be used from packages, don't look for obsolete.mk in
other directories than ../../mk/defaults/.
modify bsd.pkg.extract.mk to use them. As a side-effect, we stop
hardcoding ${LOCALBASE} as the install prefix for these tools so this
becomes more pkgviews-friendly.
bsd.pkg.mk under the new tools framework. Instead, just note that if
we set PKGSRC_LOCKTYPE, that pkgsrc will require shlock. Remove
"shlock" from the large PKGSRC_USE_TOOLS list since we don't need it
all of the time. This fixes PKGSRC_LOCKTYPE != none when using the
new tools framework.
in mk/tools/perl.mk since many packages expect to be able to use the
variables defined in vars.mk, but those variables can only be defined
if PERL5 is correctly defined, and that is only true after it is set
here.
loop are only used afterwards, split out the EVAL_PREFIX code into a
separate file find-prefix.mk that can be included at any point to
generate a prefix-computation for the variables specified by FIND_PREFIX
at the inclusion point.
them to PLIST_SUBST. They may simply be defined later on in the
Makefile processing. Instead check whether USE_PERL5 was specified
to determine whether to pass along PERL5_{SITEARCH,SITELIB,ARCHLIB}
to PLIST_SUBST.
VAR= a
PTR= VAR
${PTR}?= b
Given these definitions, ${VAR} == "b" (unintuitively). We can work
around this by doing:
VAR= a
PTR= VAR
.for _v_ in ${PTR}
${_v_}?= b
.endfor
In this case, ${VAR} == "a" (as expected).
Use the second form of assignment in this changeset so that the expected
behavior happens for the "TOOL" name variables. This fixes "recursively
defined" errors for some of the "TOOL" variables when the pkgsrc tool
replaces the system-supplied one.
These files would normally go away after packages have been taught to
replace including autoconf.mk with "USE_TOOLS+= autoconf", and similarly
for automake.mk.
within make. This allows us to use EVAL_PREFIX to locate the installation
prefix for autoconf and automake, which is more pkgviews-friendly. In any
case, this implementation is a lot simpler anyway.
${TOOLS_DIR} so that we avoid inspecting various variables during the
top-level make invocation -- rather, we defer inspection until the
target is actually made. This allows TOOLS_REAL_CMDLINE.*,
TOOLS_REAL_CMD.*, and TOOLS_REAL_ARGS.* to be defined after bsd.tools.mk
is included and still affect the creation of the tools under ${TOOLS_DIR}.
value of TOOLS_REAL_CMDLINE.makeinfo. In this case, we split MAKEINFO
into the command and the arguments, and allow the default command line
to be constructed.
separate file pkgsrc/mk/bsd.pkg.extract.mk. Also, include this file
ahead of the inclusion of bsd.tools.mk so that tools required to
extract are created by the tools framework.
This makes more sense since there is never going to be a pkgsrc
replacement for ldconfig. We now always create an ldconfig tool in
${TOOLS_DIR} that either calls the system-supplied one if it exists,
or is a no-op.
that a package needs. Tools that pkgsrc needs are listed in
PKGSRC_USE_TOOLS, and tools that a package needs on top of that are
listed in USE_TOOLS.
Define "TOOL" variables, e.g. SED, AWK, MKDIR, etc. for each of the
tools that pkgsrc needs, and "TOOLS_TOOL" variables, e.g. TOOLS_SED,
TOOLS_AWK, TOOLS_MKDIR, etc. for each of the tools that a package
needs. These variables contain the full command line to the real
command and arguments needed to invoke the tool.
created under ${TOOLS_DIR} that invoke <tool>, e.g.
TOOLS_ALIASES.gawk= awk
The example above causes the "gawk" tool to be added to ${TOOLS_DIR} as
both "gawk" and "awk".
apps. Used at the moment for X11_TYPE=[xorg|XFree86] to fix
the build/installation of some packages, see
http://mail-index.netbsd.org/tech-pkg/2005/05/05/0024.html
for more information.
To switch between 'build' or 'full' dependency method use:
X11CLIENTS_DEPMETHOD=[full|build]
caches variable definitions that were computed by make. These variables
are specified by listing them in MAKE_VARS, e.g.,
.if !defined(FOO)
FOO!= very_time_consuming_command
.endif
MAKE_VARS+= FOO
bsd.pkg.mk will include only the one generated during the most recent
phase. A particular phase's makevars.mk file consists of variable
definitions that are a superset of all of the ones produced in previous
phases of the build.
The caching is useful because bsd.pkg.mk invokes make recursively,
which in the example above has the potential to run the very time-consuming
command each time unless we cause FOO to be defined for the sub-make
processes. We don't cache via MAKE_FLAGS because MAKE_FLAGS isn't
consistently applied to every invocation of make, and also because
MAKE_FLAGS can overflow the maximum length of a make variable very
quickly if we add many values to it.
One important and desirable property of variables cached via MAKE_VARS
is that they only apply to the current package, and not to any
dependencies whose builds may have been triggered by the current
package.
The makevars.mk files are generated by new targets fetch-vars,
extract-vars, patch-vars, etc., and these targets are built during
the corresponding real-* target to ensure that they are being invoked
with PKG_PHASE set to the proper value.
Also, remove the variables cache file that bsd.wrapper.mk was generating
since the new makevars.mk files provide the same functionality at a
higher level. Change all WRAPPER_VARS definitions that were used by
the old wrapper-phase cache file into MAKE_VARS definitions.
framework figure out by itself whether a wrapper or a symlink should be
created based on the real command and any arguments that may need to be
invoked.
