Before, relative paths had been stored as-is. This affected those
packages that defined PATCHDIR or FILESDIR as relative directory instead
of prefixing it with ${.CURDIR}.
Since there are already several other paths that are interpreted relative
to the package directory (CONFLICTS, DEPENDS), allow PATCHDIR and
FILESDIR to be specified as relative paths, too. This makes the package
Makefiles a bit shorter.
The rewritten check for unknown configure options already checks for
options that are not defined anywhere. After that, the configure scripts
don't need to check for these options anymore since all remaining options
are known to the package, in at least one of its configure scripts.
The previous implementation could not reliably detect outdated configure
options. This was apparent in devel/gettext-tools, where the option
--with-included-libcroco had become unknown between May 2019 and May
2020, but the check was not run.
The behavior is the same in the pkgsrc default configuration. Only if
GNU_CONFIGURE_STRICT=yes, the new check is activated and will make
packages fail that previously succeeded to build. Since that variable is
not widely known, there won't be much sudden breakage, if any.
This makes the SUBST blocks stricter than before, to detect outdated or
unnecessary definitions.
Filename patterns that are not affected by any of the SUBST_SED
expressions make the build fail. It is still ok if only some of the
files from a pattern are affected and some aren't.
The latest bulk build shows that most of the build failures are fixed.
The packages that fail in that build are mostly due to other failures,
like missing C headers, wrong PLIST files, unresolved references at link
time. There may be a few packages that still fail because of this, but
these are near the leaves of the dependency tree.
https://mail-index.netbsd.org/pkgsrc-bulk/2020/05/14/msg018919.html
In a basic regular expression, a dollar-sign only means end-of-string if
it appears at the end of the pattern, or (at the choice of the
implementation) at the end of a \(...\) subexpression.
This affects the package converters/help2man that uses a regular
expression containing a dollar in a non-final position. This regular
expression had not been detected as an identity substitution even though
it is one.
Since GHC 7.10 or 7.8, the Haskell packages are installed in directories
whose name contains the package hash. This made it harder to predict the
exact pathname. Havin the exact pathnames in the PLIST file is the
ideal, it also helps to record the general structure of the installed
files to see whether some file unexpectedly appear or disappear.
To enable this for Haskell packages, the various base directories are
replaced with placeholders during print-PLIST. These placeholders are
translated back to their respective paths when the +PLIST is generated
from the PLIST in the package directory.
Except for 2 packages, all Haskell packages in main pkgsrc had their
package PLIST file removed. To help in adding them back, the pkgsrc
developer can set HS_UPDATE_PLIST=yes in mk.conf, which will generate the
PLIST directly into ${PKGDIR}/PLIST upon installation.
Most packages in pkgsrc-wip still have their old PLIST, and these are
migrated automatically as well.
This case can only happen in the following special case:
TOOLS_CREATE+= asdf
TOOLS_PATH.asdf= # empty
If there is a lonely TOOLS_CREATE without a corresponding TOOLS_PATH, it
defaults to ${FALSE} and thus doesn't trigger this code.
Packages that don't declare USE_TOOLS+=perl and whose configure script
invokes perl produce a warning.
Usually warnings are ignored, but they can also be configured as errors,
for example during a strict bulk build. In this situation it is
necessary to override the default behavior of the perl tool to fail
silently. Up to now, defining both TOOLS_BROKEN+=perl and
TOOLS_FAIL+=perl produced a duplicate make target.
To handle this situation, let TOOLS_BROKEN+=perl take precedence over
TOOLS_FAIL+=perl. This is much easier than finding out in each case how
to disable the perl check in the configure script, which is most often
done by adding any of the following to CONFIGURE_ENV: PERL=#none,
ac_cv_prog_PERL=#none, ac_cv_path_PERL=#none.
This information is useful for getting the variable name that corresponds
to a tool. In most cases this is just the uppercase name of the tool,
but there are exceptions like ${SETENV} for env, ${HOSTNAME_CMD} for
hostname.
A commonly occuring scenario is that a package patches the configure
script, but that the corresponding configure.in contains shell code that
is not portable. In cases like these, configure.in is typically not used
during the build, therefore there is no need to check it for portability.
This also applies to all other combinations where a file is patched and
the corresponding file.in contains unportable shell code.
Some packages using mk/djbware.mk do not have error.h but need to fix the
errno declaration in other files. Up to now, there was no good way of
achieving exactly this.
Building packages like these in a SUBST_NOOP_OK=no build makes these
packages fail. To fix these packages, the configuration of the files
needed to be a bit more flexible.
The escaping inside the backticks had been wrong. Because of this,
parentheses and semicolons were interpreted as shell syntax.
Switching to $(...) command substitution removes the need for quoting
some of the characters and makes the whole command simpler to understand.
Doing the escaping for the backticks command properly would have involved
lots of special cases.
The $(...) command substitution was used sparingly in pkgsrc up to now
because some older or broken shells do not support it. Since these
shells do not end up as the shell that runs the commands from Makefiles,
that's not a problem.
To work properly, the $(...) should have been $$(...).
In pkgsrc the command substitution is usually done via `backticks`, for
compatibility with /bin/sh from Solaris. To fix the shell parse errors,
the special characters are properly escaped inside the command
substitution.
Since SUBST_FILTER_CMD is a shell command, it may contain arbitrary
characters. The condition in mk/subst.mk that tested whether
SUBST_FILTER_CMD was the default filter command was evaluated at run
time. In such an evaluation, the variables (lhs and rhs) are fully
expanded before parsing the condition. This means that these variables
must not contain quotes or unquoted condition operators.
Exactly this situation came up in one of the regression tests. The
quoted "0-9" was copied verbatimly into the condition, including the
quotes. The resulting condition was:
"tr -d "0-9"" == "LC_ALL=C /usr/bin/sed "
This produced a syntax error because of the left-hand side. Adding a :Q
modifier would have helped for the left-hand side, but this would have
been necessary for the right-hand side as well. Since an empty SUBST_SED
is defined not to "contain only identity substitutions", the first
condition can simply be removed.
The whole condition in the shell program had not worked anyway since it
expanded to either "[ true ]" or to "[ false ]", and both of these
commands exited successfully.
There are several cases where patterns like s|man|${PKGMANDIR}| appear in
SUBST_SED. Up to now, these had been categorized as no-ops and required
extra code to make the package build when SUBST_NOOP_OK was set to "no".
This was against the original intention of SUBST_NOOP_OK, which was to
find outdated substitution patterns that do not occur in SUBST_FILES
anymore, most often because the packages have been updated since.
The identity substitutions do appear in the files, they just don't change
them. Typical cases are for PKGMANDIR, DEVOSSAUDIO, PREFIX, and these
variables may well be different in another pkgsrc setup. These patterns
are therefore excluded from the SUBST_NOOP_OK check.
Before, adding "Binary file matches" (including the quotes) to
CHECK_WRKREF_EXTRA_DIRS led to a syntax error. Adding this string is so
obvious that it should have been added a long time ago already.
The indentation of the inner loop has been fixed.
The chmod is only run if the file has actually changed. In the other
case, the file would have been removed right after the chmod, which made
the chmod unnecessary.
For compatibility with ancient operating systems whose /bin/sh still does
not understand negated conditions (SunOS), these conditions have been
avoided and were written using && and || instead.
The inner loop has been flattened a bit, to compensate for the
indentation of the outer loop.
This avoids creating a temporary directory.
The "set -f" option is not used anywhere else in pkgsrc, even though it
has been available since 1985 in the 8th Edition of Research Unix. Even
AIX and IRIX have that option, so it seems a safe bet.
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html
says in section 9.3.2 BRE Ordinary Characters that only very few
characters may be preceded with a backslash.
As a side effect, this change allows parentheses in the variable names
listed in SUBST_VARS (even if that will never happen in practice).
The reason that the regression test had not replaced VAR.[] before was
simply that this variable had not been listed in SUBST_VARS.
if quoted. Seen on NetBSD 7.
#!/bin/sh
in="/path/to/dir with space/file"
: "${file=${in##*/}}"
: "${dir=${in%/*}}"
echo "dir:$dir"
echo "file:$file"
[ "$dir" = "$file" ] && echo "dir and file are same"
Leads to errors when adding packages such as:
./+FILES: cannot create /var/db/pkg.refcount/files/etc/rc.d/xenguest//var/db/pkg/xe-guest-utilities-7.0.0: directory nonexistent
This variable allows to make SUBST stricter than before. This will break
several packages that have redundant filename patterns. Most of these are
typos or outdated and should be updated or removed.
While here, add more verbose documentation on PLIST_SUBST since the way
from the package's PLIST file to the +PLIST file can easily be confused
with the other way round, which is handled by print-PLIST.
The .for loops are expanded by bmake as soon as they are parsed. Since
there are many variable groups, and since each of these groups has
several variables, this takes quite a bit of time.
The time for running "bmake clean" in pkgtools/pkglint went down from 1.0
second to 0.7 seconds. This may not seem much, but when multiplied with
23088 times 9, this little change may speed up a full bulk build by 62000
seconds, which is about 18 hours.
The side-effect is that the show-all-* targets can only be run from the
command line, not as a dependency of other targets. This restriction will
probably go unnoticed.
Used to form _HASKELL_PKG_DESCR_DIR. Defaults to DISTNAME, but for
some packages (hledger, incoming), the same DISTNAME is shared by
multiple packages. So let the package define it explicitly if need
be.
Before, searching for topic=socket did not find the documentation.
The detection of useful help topics is still not perfect since it now
finds sections that consist of a single word, such as the word
"undo-replace" in mk/install/replace.mk, but that will be fixed later,
after adding a few unit tests.
The code maps gcc-4.8.x to "4", and then won't find gcc4. This is a
bug, but it may be that it's just as well to never match 4.X of any
kind, and use 7 anyway. Explain this issue with a \todo to fix the
bug or document the consequences as intended.
(This is a comment-only change.)
Adjust regexp that removes .Y.Z from gcc-X.Y.Z.
Test for gcc being contained in PKGSRC_COMPILER, vs ==, so that a
value of "ccache gcc" is handled properly.
(ok for mk during freeze jperkin@)
Currently virtualsize is only defined for NetBSD, breaking builds that use it
on any other platform. Adding defaults for all other platforms should be done
at some point, this at least unbreaks package builds for now and provides a
safer default for any future additions.
There are several places in pkgsrc where the files to be patched are
listed individually instead of just saying util/*/*.sh. This is
unnecessarily detailed. Encourage package authors to use filename
patterns more often.
An example is REPLACE_PYTHON in lang/clang, which currently fails because
some of the listed files don't exist anymore.
These appear to have been cargo culted around for a while, don't even have
anything to do with DESTDIR mode, and are completely useless. Simplify the
logic a little while here. Noticed by rillig@.
When fixing the SUBST definitions in a package, it can hapen that the
substitution aborts in the middle. In such a case the cookie should not
be written and the substitution should be retried. Otherwise the build
may continue with half the substitutions done.
The severity now depends only on the setting of SUBST_NOOP_OK. Right now
this means that some former warnings will be reported as info only, but
that will change after switching the default of SUBST_NOOP_OK after
2020Q1. Then they will all be reported as warnings, followed by the final
error saying that the pattern has no effect.
This change makes it easier to detect inconsistencies and outdated
definitions, for example by setting the global SUBST_NOOP_OK=no and
redefining WARNING_MSG to actuall fail.
The diff program is only used to produce informative output in the build
logs, nevertheless its output might be translated if there are lines that
do not end with a newline.
This hack is mostly needed for older software like sysutils/daemontools
that was created when errno was still a global variable. Newer packages
like devel/bglibs don't need that hack anymore. Therefore make it
configurable, to avoid build failures when subst.mk does not find any
error.h to patch.
The variables are now sorted chronologically instead of alphabetically,
to express the order in which the actions happen. The sentences are a bit
shorter than before.
The visual layout has been changed to make the whole variable name stick
out more. Before, the part that was easily visible was the same "EXTRACT_"
for all of them.
Before, variables containing dollar characters displayed so wrong that it
was hard to explain.
To fix the problem, I typed almost random characters into the code until
the output was exactly as expected. I still do not understand:
* why the list variables need 8 dollars to survive the @x@ loop,
* why the code only works if the dollars come from an external variable
instead of being written inline,
* why the backslash in the :C modifier needs to be doubled.
Anyway, the output of "bmake show-all-extract" now contains the shell
variable $${extract_file}, just as it should. The dollars are now doubled
in the output and thereby match the source code from the Makefile
exactly.
This prevents unintended shell word expansion.
It also makes that argument visible in "bmake show-all-extract". Before,
it was left out entirely from the output. Now it is printed as an empty
string literal, which at least gives a hint that "there is something".
The default value of SUBST_MESSAGE is based on SUBST_FILES, and that
variable may use the :sh modifier to list files from WRKSRC, which may
not exist at load time.
In the case of pkglocaledir, the SUBST_FILES are generated by a shell
command. That command assumes that the WRKDIR already exists. Therefore
SUBST_FILES must be evaluated as late as possible.
See mk/configure/replace-localedir.mk; an example package that fails is
devel/gettext-tools.
In a bulk build with very strict settings (WARNING_MSG fails, as well as
no-op substitutions), it became clear that nearly all of the cases where
SUBST didn't replace anything were bugs in the package definition.
Most of them were just outdated, which is no surprise given that some
packages are already over 20 years old.
For backwards compatibility, SUBST_NOOP_OK defaults to "yes" right now.
After correcting the affected packages, the default will change to "no".
This file allowed one to in theory specify a wxGTK version, but only
two programs used it. Both now directly include a preferred version.
Generally, packages should use wxGTK30, unless there is a good reason
to use 28.
A major motivation earlier was not being able to install 28 and 30 at
once, but this has been solved.
As briefly discussed on pkgsrc-users.
In an attempt to resolve problems on SunOS, move EXTRACT_USING to the
early variables file, in the hopes that it will be defined before code
that checks it is executed.
When all files are skipped, the tool dependency is not added as well.
This allows packages to skip the check without defining the user-settable
variable CHECK_PERMS.
Since the portability checks are concerned about bashisms, it doesn't make
sense to apply them to programs that will later be run in bash anyway.
Suggested by gdt via private mail.
These files typically come with a corresponding Makefile.in file, and the
same portability issue is in the .in file as well. For building the
package it is only necessary to patch the .in file. Patching the .am file
as well would mean additional work for the pkgsrc package maintainer.
Nevertheless, such fixes should be reported upstream. The upstream
maintainer is more interested in a patch for Makefile.am, since the .in
file is only generated.
Suggested by wiz via private mail.
Before, file names that start or end with spaces were trimmed by read(1),
as specified by POSIX 2004 and POSIX 2018.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html
This edge case bug had been present since the very beginning of this
code, in November 2006.
Filenames containing backslashes or newlines are still not supported, and
probably never will be intentionally supported.
