BUILDLINK_CONTENTS_FILTER, as discussed on tech-pkg on may 26 and 27.
Fix an issue with www/libwebsockets/buildink3.mk where some cmake files
are missing from the .buildlink dir, causing config failure in consumer
packages.
This is part 1 to support additional platforms with the RC scripts provided in
pkgsrc, in privileged as well as in unprivileged mode, including on NetBSD
(with part 2 in pkgtools/rc.subr).
This variable is meant to point to the configuration directory of the base
system (as opposed to pkgsrc's own prefix) when it should be used by pkgsrc in
special cases (e.g. installing RC scripts), or to point to the existing
PKG_SYSCONFBASE directory otherwise (e.g. for any unprivileged bootstrap).
This teaches pkgsrc where the RC scripts should be installed, and more
importantly, where the local copy of rc.subr can be expected. Part 3 will
progressively update each and every RC script to substitute this path as
expected.
No functional changes are intended in privileged mode without a bootstrap. The
only variable affected by this change directly is RCD_SCRIPTS_DIR, which
currently remains with the same default of /etc/rc.d, and can be overridden as
before.
When bootstrapping, SYSCONFBASE also remains with the existing default when no
prefix is set or is "/usr/pkg" or "/usr"; it is set to $prefix/etc otherwise. It
can be specified specifically with --sysconfbase if necessary.
Existing installations or bootstraps are not affected, as this change needs
setting SYSCONFBASE in the corresponding $sysconfdir/mk.conf to have an impact.
Tested in privileged and unprivileged modes on NetBSD/amd64, and unprivileged
mode on Darwin/amd64; submitted for review on tech-pkg@.
- Avoid shouting in version names. Users may still set MYSQL_VERSION_DEFAULT
to "MARIADB104", but it is preferred to switch to "mariadb104".
- Set the correct variables in BUILD_DEFS_EFFECTS and _SYS_VARS.
- Instead of hardcoding library names with per-OPSYS logic and testing for
their existence to see if the package is installed, do it the correct way
using pkg_info(1).
- Make it easier to add new MySQL versions.
- Avoid unnecesary variables. Use bmake(1) slices to select the first item
in a list rather than a temporary variable.
- Improve documentation.
Based on a patch I've had sitting in the joyent/pkgsrc tree for far too many
years. No functional change other than the switch to lowercase package names
by default. Tested in a bulk build with additional Percona packages.
At least on FreeBSD 13.0, awk '/^[\t -~]/' does not match alphabetical
characters with some utf-8 locales (e.g. neither en_US.UTF-8 nor
fr_FR.UTF-8 works but C.UTF-8 does).
This enables use of MPI compiler wrappers present in the host
system via MPI_TYPE=native. Also, it checks for conflicts with
a preintalled different MPI choice from pkgsrc and.
Now it should be more obvious when a package needs it as a dependency,
as it will fail loudly if it isn't declared as a tool.
While here, some duplicate dependencies on itstool were removed from the
MATE packages
most of these simply extend matching from "aarch64" to "aarch64eb"
in various forms of code. most remaining uses in pkgsrc of
"MACHINE_ARCH == aarch64" are because of missing aarch64eb support,
such as most of the binary-bootstrap requiring languages like rust,
go, and java.
no pkg-bump because this shouldn't change packages on systems that
could already build all of these.
It was a long-standing issue that Haskell packages which didn't contain a
library could not be correctly handled.
There are fewer substitutions in PLIST_SUBST now. As a result existing
PLIST files will all be considered as outdated and should be updated
either by setting HS_UPDATE_PLIST=yes or by manually running print-PLIST.
They will be ignored until that.
Capitalize Free and Open Source more consistently.
Use "approved" rather than "accepted" when discussing Debian, to
avoid confusion between their rules and processes and ours.
(This is not a substantive change.)
Turned out this was necessary when using `cabal-install v2-install` outside of pkgsrc, because the command creates a per-user package environment and in turn makes Cabal hidden.
Move the navigation to the top and reword it slightly. This makes it
more immediately visible. I'm not sure if this justifies a "Skip to content"
button yet.
Give the binary package table headers.
- Add a maintainer field.
- Use lists and sections to separate information.
- Make license information a hyperlink.
- Move available build options and vulnerabilities listings to the bottom,
as they tend to use up a lot of space and create reader fatigue.
- Correct stylesheet paths.
- Only bother with top-level dependencies and don't attempt to recurse
the tree, this creates very messy output for modern packages.
- Normalize output for both build-time and runtime dependencies.
- Remove slow workarounds for Solaris 9 awk limitations.
For example, the recently release macOS 11.2 does not ship a 11.2 SDK, but
continues to use the compatible 11.1 SDK. This now works correctly without
having to enable OSX_TOLERATE_SDK_SKEW.
These should only be added for releases within the same major version where we
can guarantee compatibility.
This variable appears to be incorrect as "_PKGSRC_USE_RELRO" is set in
bsd.prefs.mk. This was causing RELRO_SUPPORTED=no to not function as intended
in lang/go/go-vars.mk
ok'd by jperkin
Previously this was only done on Big Sur to work around the issue where XCode
does not support running these programs via a symlink, breaking .tools/bin.
However, with the update to autoconf 2.70, the native GNU m4 from 2006 on all
Darwin systems is too old and breaks the build on Catalina and older, causing
massive dependency failures.
Avoiding them both completely at this time is the simplest way forward.
gcc4.8,4.9,5 have bugs preventing them from being useful within pkgsrc
for the primary use case that finds them handy:
glibc + FORTIFY + those GCC versions = build failures.
Additionally, requiring fewer versions of GCC is an improvement for
the vast majority of use-cases considered.
We might want to bump this further than gcc6 later on, but this is a
big improvement for CentOS builds.
This broke various things.
Also remove the comment in devel/autoconf/Makefile that says to update it, so
that the next person does not fall into the same pitfall again.
2.70 deprecates a lot of stuff, so expect more warnings, but generally things
seem to work fine, so updating to 2.70 shouldn't break much.
Also update mk/gnu-config/*, as per the comment in devel/autoconf/Makefile.
mk/gnu-config/missing is not actually part of autoconf, but of automake, which
I did not update - however, the file was quite out of date, so I took the
liberty to update that one with the latest automake.
The change log is too long to include in this commit, given how many years
there were between 2.69 and 2.70. Check the file ChangeLog after `make
extract`.
As part of the pkgdb migration (NetBSD only), PGKTOOLS_REQD was set to
a recent value (20200828). However, that results in a cyclic
dependency of pkg_install on cwrappers on pkg_install. Once people
set PKG_DBDIR in pkg_install.conf and mk.conf to match their setup,
there is no need -- because of the migration -- to force newer tools.
Testing on TNF's pkgbuild machine indicates this revert works well,
and I received two positive comments and none against.
breakage for users who have other package managers that use /var/db/pkg
(Reported by Frederic Cambus on FreeBSD, OpenBSD)
Adjust warning: specifying PKGDB_DIR in mk.conf should be sufficient
for building packages for the compatibility pkgdb location.
This is still an error message, because:
- While we can handle the existing references of PKGDB_DIR, new ones
might be created.
- It is a minor inconvenience to people building packages from source.
This makes it easier to use the mk fragment with fonts that need a build
step, like new liberation-ttf.
While here switch some fonts using post-install unnecessarily to do-install
(Committed at the same time as it wasn't tested separately)
No PLIST changes to the packages/build breakage, so no changes expected
to the packages.
Now dependencies can be listed either by package name, or path to the
package (eg with "make PACKAGE_NAME_TYPE=path build-depends-list").
Users of PACKAGE_NAME_TYPE=html could use a combination of
PACKAGE_NAME_TYPE=name and PACKAGE_NAME_TYPE=path instead now.
No objection from tech-pkg@
lang/gcc8 has patches for NetBSD/aarch64 and lang/gcc10 has support mostly
upstreamed. Nobody seems interested in fixing gcc9, but the pkgsrc
logic defaults to it when the system compiler is GCC 9 which leads to
broken fortran packages. Let's just skip forward to gcc10.
PostgreSQL 13 contains many new features and enhancements, including:
* Space savings and performance gains from de-duplication of B-tree index entries
* Improved performance for queries that use aggregates or partitioned tables
* Better query planning when using extended statistics
* Parallelized vacuuming of indexes
* Incremental sorting
Install the new interchangeable BLAS system created by Thomas Orgis,
currently supporting Netlib BLAS/LAPACK, OpenBLAS, cblas, lapacke, and
Apple's Accelerate.framework. This system allows the user to select any
BLAS implementation without modifying packages or using package options, by
setting PKGSRC_BLAS_TYPES in mk.conf. See mk/blas.buildlink3.mk for details.
This commit should not alter behavior of existing packages as the system
defaults to Netlib BLAS/LAPACK, which until now has been the only supported
implementation.
Details:
Add new mk/blas.buildlink3.mk for inclusion in dependent packages
Install compatible Netlib math/blas and math/lapack packages
Update math/blas and math/lapack MAINTAINER approved by adam@
OpenBLAS, cblas, and lapacke will follow in separate commits
Update direct dependents to use mk/blas.buildlink3.mk
Perform recursive revbump
pkgsrc changes:
---------------
* Remove the pkgconfig file generation since the version of libusb1 cannot
be obtained by parsing LIBUSB_API_VERSION from libusb.h (e.g. FreeBSD
provides 0x01000102 for 1.0.13 and Arch provides 0x01000107 for 1.0.23).
* At least FreeBSD, Debian and Arch provides a libusb-1.0.pc file for
their native implementation. Link this file to ${BUILDLINK_DIR}.
* Add logic in mk/buildlink3 to find pkgconfig files in common pkgconfig
directories (for at least FreeBSD, Debian and Arch).
These no longer support being executed via a symlink, failing with errors
such as:
xcode-select: Failed to locate 'gmake', and no install could be requested
This breaks the entire .tools/bin directory, so we just have to avoid them
and use tools from pkgsrc instead.
It's likely a lot more will need to be added to this list, but this is
enough to get devel/cmake building at least.
This is required for find-libs.mk to continue detecting the presence of
libraries supported by the system. It's definitely not ideal, and only
still works because Apple happens to ship .tdb files for each library, and
these are found via the current "lib${_lib_}.*" glob.
Patch taken from sjmulder@, I only limited it to Big Sur for now in case
there are issues using the SDK directory on older releases.
The new release of macOS removes system libraries from the file system, only
providing access to them via a linker cache and dlopen(). This obviously
breaks many assumptions about how libraries work on Unix systems, and so we
unfortunately need to cripple various checks when running on those systems.
Introduce DARWIN_NO_SYSTEM_LIBS which, when defined, will trigger alternate
behaviour in the infrastructure. Currently this is in two places:
* In CHECK_SHLIBS, skip any path beginning with /usr/lib.
* In registered package metadata, any path beginning with /usr/lib is
removed from REQUIRES.
The former fixes all package builds, while the second will be necessary for
package managers such as pkgin, as they will no longer be able to verify that
those files are available on the target system.
This is obviously a gross hack, and removes our ability to ensure that the
target system is suitable for the packages we are attempting to install, but
Apple have left us with no alternative, and users will unfortunately be left
to find out at runtime instead.
It's likely this will need to be extended to /System/Library paths too, but
this is required first to actually get packages building before we can start
running bulk builds.
This means that from now on, there is no global setting to switch off
this redundancy check. Individual SUBST classes can still set their own
SUBST_NOOP_OK.<id> in order to ignore no-op filename patterns.
The current bulk builds do not show any build failures that are caused
by this, which means that really almost all packages have been migrated.
-Werror=implicit-function-declaration, which will be great someday when
we're ready for it. Until then, on macOS, detect this situation and tell
the cc wrapper to prepend -Wno-error=implicit-function-declaration.
Taking mail/qmail as our example, this fixes the build on Catalina
with "Apple clang version 12.0.0 (clang-1200.0.32.2)". Adding
-Werror=implicit-function-declaration to CPPFLAGS or CFLAGS (in
mk.conf or on the command line) re-fails the build, as expected, with
a pile of "error: implicit declaration of function". Also as expected,
a full -Werror fails earlier on one of the many other problems with
qmail's code.
For clang on non-macOS platforms, no change.
clang-and-wrapper-ok: joerg@
during-the-freeze-ok: gdt@
The show-all code is mostly line noise, therefore it is all the more
important to provide at least a few hints to a potential reader, by
using descriptive variable names for the iteration variables:
g => grp
c => cat
v => var
w => width
x => word
I had been confused by the printf commands since some of them used '\n'
and some used '\\\n', which seemed as if there were some quoting issue
that would make it necessary to double the backslashes.
This assumption was wrong though. The printf commands for the
single-valued variables use the normal '\n', while the lines for the
multi-valued variables end with a real backslash in the output, to
mimick the continuation lines in makefiles.
As a hint that the '\\\n' means backslash + newline, add single quotes
between the two characters.
The previous code relied on the exact implementation of Var_Parse in
bmake, and that it does not issue any error messages in case of $$ in
variable modifiers.
In variable modifiers, a $ is escaped using \$, not $$, as documented in
the manual page.
At the time when I wrote the previous version with the _SHOW_ALL.d4 and
_SHOW_ALL.d8 hacks, I did not know about the backslash escaping rule,
and I just added dollar signs until everything seemed to work. I
couldn't explain why it worked though, which is not surprising since the
code was using an undocumented implementation flaw of bmake.
The previous shell script version's runtime was quadratic against the
number of distfiles to verify. Historically this has not been an issue,
with usually only a handful of files per package. However, with the
introduction of Go modules the number of distfiles used by a single
package can be very high.
For example, in an upcoming update of www/grafana to version 7.1.5, the
number of GO_MODULE_FILES is 821. Running 'bmake checksum' takes:
real 18m20.743s
user 17m27.975s
sys 0m49.239s
With the awk code, this is reduced to a far more sensible:
real 0m4.330s
user 0m3.241s
sys 0m0.875s
The script has been written to emulate the previous version precisely,
preserving the same output and error messages and supporting all of its
behaviour, with the one exception that previous exit values of 128 have
been changed to 3, in order to avoid any potential signed 8-bit issues.
The one change in the pkgsrc infrastructure is that the mk/fetch/fetch
script no longer sets a working default value for ${CHECKSUM}. This is
not a problem in a pkgsrc environment as all of the required variables
are set correctly, but if there happen to be any users who are using
this script in a standalone environment, they will need to set it
accordingly. This was probably required in many situations previously
anyway, as none of the script's environment variables were set, and
trying to support this would be fragile at best.
At some point CRAN added the https protocol to its repositories, but
this was never reflected in MASTER_SITE_R_CRAN. Add analogues for
all the http sites with responsive https servers.
There won't (or at least should never!) be any files under share/ or man/ that
require conversion for CTF or debug support, so set sensible defaults for both
CTF_FILES_SKIP and STRIP_FILES_SKIP. Further additions are welcome.
While here rearrange the ordering of the debug skips to match CTF and deliver
a small performance improvement by avoiding unnecessary file tests.
Combined, these reduce the runtime for "make install-ctf install-strip-debug"
in lang/rust down from wall/user/sys 10m33s/2m34s/9m30s to 1m13s/0m46s/1m4s.
There was only a single valid value for HASKELL_COMPILER, therefore the
variable was useless. It only made the implementation more complicated
than necessary.
The buildlink3 variable names are quite long. So long that using the
default column width of 24 characters, most of the variable values are
not aligned. In this case, it makes sense to shift them all to the right
a bit.
Now that haskell.mk distinguishes between plain and outdated PLIST files,
this is possible again. When haskell.mk knew only missing and outdated,
this was still ambiguous and therefore skipped.
The PLIST_SUBST and PLIST_PRINT_AWK definitions for Haskell library
packages are only useful if the package-description file exists. If
that file is absent though, these are skipped.
The test whether the file exists is made as late as possible since that
file does not yet exist at the point where the package Makefile is
parsed.
This also affects the show-all-haskell target, which only shows these
values after the install phase. This is not perfect but good enough for
practical cases.
Before, running "HS_UPDATE_PLIST=yes bmake update" in wm/xmonad did not
apply the proper substitutions to the generated PLIST file since the
PLIST file was created empty during the GENERATE_PLIST command, and that
empty PLIST file changed the status to "plain" instead of "missing".
Because of that, the HS_INTF and related placeholders were not defined.
The 2 conditions for the status "missing" had to be written in separate
.if clauses because of a bug in bmake that was introduced in 2015 and
will be fixed with the next bmake update. For further details, see
src/usr.bin/make/unit-tests/cond-short.mk.
It had been switched off to not affect packages in the stable branch
2020Q2. Now starts the last round where it is possible to disable this
check. After 2020Q3, all SUBST blocks must either find their patterns or
be explicitly marked as potential no-ops. This will help to find
outdated SUBST blocks.
With our current version of pdksh, a "false && something" construct under
"set -e" conditions will continue as it does with other shells, but if the
construct is within a for loop then it exits, causing failures in the
substitution code. An explicit "|| true" is necessary to avoid this.
Approved during the freeze by wiz.
Instead:
1. Package makefiles including their own options.mk
2. Packages say "SUBST_CLASSES+=djberrno" to get the hack, if needed
3. Packages adjust SUBST_FILES.djberrno, if needed
Should fix bulk build failures due to multiple inclusions of options.mk
and/or incorrect definitions of DJB_ERRNO_HACK.
Approved during the freeze by wiz@.
The package textproc/hs-cgrep does not install a Haskell library. This
was unexpected to mk/haskell.mk, which generated an obviously wrong PLIST
file for that package, and for 3 other packages.
Noticed by wiz.
This is intended to reduce the log output on ftp.NetBSD.org when
fetching all distfiles.
Also, we call the target in basically every step of package creation
(extract, patch, configure, build, install, package) - perhaps we should trim
this down some more.
There are still some packages that fail the strict SUBST check. These
packages should nevertheless be built using the default pkgsrc
configuration, at least in the stable 2020Q2 branch. After 2020Q2 has
been switched, the strict SUBST checks will be activated again in the
default configuration.
To avoid bmake warnings because of duplicate class names, the :O:u
modifier had been added in r1.66 on 2020-03-21. This had the side effect
that the subst classes are now applied in alphabetical order instead of
declaration order.
For this to actually matter, there must be a file that is affected by two
different subst classes and in which the substitutions depend on each
other or prevent each other. Chances for that are pretty low.
The order is intentionally documented as being unspecified, to allow for
future modifications, just in case that a bmake variable modifier is
invented that filters for duplicates without requiring the duplicates to
be adjacent to each other. In that situation, it would be nicer to
switch back to declaration order instead of alphabetical.
EARLY_PRINT_PLIST_AWK is like PRINT_PLIST_AWK but operates before the
file/directory lists are sorted.
Discussed on tech-pkg@ mainly to address `print-PLIST' order problems for
Python 3 packages:
<https://mail-index.NetBSD.org/tech-pkg/2020/05/27/msg023249.html>
Just in case any of these tools defines some command line arguments. The
correct path had already been used before since both env and sh are added
to USE_TOOLS in bsd.pkg.mk.
There is no evidence that any shell needs the empty string literal in
'for i in "" $var' to loop over a simple variable. This would only be
necessary if the shell code were generated by a preprocessor such as
bmake or autoconf, and the list of items might end up empty. In such a
case, the empty string literal prevents to generate 'for i in ; do',
which would produce a syntax error.
This assignment has popped up various times in the pkglint output, even
though it is defined in the infrastructure and not in a specific package.
Since there is no need to have this duplicate assignment, it is removed.
The general rule is that a SUBST_SED that contains _any_ identity
substitution may leave files unmodified, no matter if there are other
substitutions as well.
When RUN includes the -u shell option (see the pkgsrc guide, section
bulk.var.shvar), there is a stray "requires: parameter not set" error.
This error doesn't make the build fail immediately since it is on the
left-hand side of a pipe, where the exit status is ignored. It would
fail if the shell for bmake were set to bash and its option "pipefail"
were enabled.
For USE_LANGUAGES there is already a pkglint warning, but for GCC_REQD it
is missing. It's better to have this check directly in the
infrastructure since it is more reliable.
This check is disabled by default, to not cause any new breakage.
It should be enabled in bulk builds.
Right now, users who install the pkg-vulnerabilities database find that
the vast majority of packages fail to build, penalizing them too severely.
Package auditing can still be done via "pkg_admin audit".
Alternatively, the previous behaviour can be restored with
ALLOW_VULNERABLE_PACKAGES=no in mk.conf.
Additionally, bmake-ify the check.mk logic. It was easier to do this,
as the package relied on a single long ${RUN} command.
Proposed on tech-pkg, with no objections to the idea of changing the
default, just the method of doing so.
Before, relative paths had been stored as-is. This affected those
packages that defined PATCHDIR or FILESDIR as relative directory instead
of prefixing it with ${.CURDIR}.
Since there are already several other paths that are interpreted relative
to the package directory (CONFLICTS, DEPENDS), allow PATCHDIR and
FILESDIR to be specified as relative paths, too. This makes the package
Makefiles a bit shorter.
The rewritten check for unknown configure options already checks for
options that are not defined anywhere. After that, the configure scripts
don't need to check for these options anymore since all remaining options
are known to the package, in at least one of its configure scripts.
The previous implementation could not reliably detect outdated configure
options. This was apparent in devel/gettext-tools, where the option
--with-included-libcroco had become unknown between May 2019 and May
2020, but the check was not run.
The behavior is the same in the pkgsrc default configuration. Only if
GNU_CONFIGURE_STRICT=yes, the new check is activated and will make
packages fail that previously succeeded to build. Since that variable is
not widely known, there won't be much sudden breakage, if any.
This makes the SUBST blocks stricter than before, to detect outdated or
unnecessary definitions.
Filename patterns that are not affected by any of the SUBST_SED
expressions make the build fail. It is still ok if only some of the
files from a pattern are affected and some aren't.
The latest bulk build shows that most of the build failures are fixed.
The packages that fail in that build are mostly due to other failures,
like missing C headers, wrong PLIST files, unresolved references at link
time. There may be a few packages that still fail because of this, but
these are near the leaves of the dependency tree.
https://mail-index.netbsd.org/pkgsrc-bulk/2020/05/14/msg018919.html
In a basic regular expression, a dollar-sign only means end-of-string if
it appears at the end of the pattern, or (at the choice of the
implementation) at the end of a \(...\) subexpression.
This affects the package converters/help2man that uses a regular
expression containing a dollar in a non-final position. This regular
expression had not been detected as an identity substitution even though
it is one.
Since GHC 7.10 or 7.8, the Haskell packages are installed in directories
whose name contains the package hash. This made it harder to predict the
exact pathname. Havin the exact pathnames in the PLIST file is the
ideal, it also helps to record the general structure of the installed
files to see whether some file unexpectedly appear or disappear.
To enable this for Haskell packages, the various base directories are
replaced with placeholders during print-PLIST. These placeholders are
translated back to their respective paths when the +PLIST is generated
from the PLIST in the package directory.
Except for 2 packages, all Haskell packages in main pkgsrc had their
package PLIST file removed. To help in adding them back, the pkgsrc
developer can set HS_UPDATE_PLIST=yes in mk.conf, which will generate the
PLIST directly into ${PKGDIR}/PLIST upon installation.
Most packages in pkgsrc-wip still have their old PLIST, and these are
migrated automatically as well.
This case can only happen in the following special case:
TOOLS_CREATE+= asdf
TOOLS_PATH.asdf= # empty
If there is a lonely TOOLS_CREATE without a corresponding TOOLS_PATH, it
defaults to ${FALSE} and thus doesn't trigger this code.
Packages that don't declare USE_TOOLS+=perl and whose configure script
invokes perl produce a warning.
Usually warnings are ignored, but they can also be configured as errors,
for example during a strict bulk build. In this situation it is
necessary to override the default behavior of the perl tool to fail
silently. Up to now, defining both TOOLS_BROKEN+=perl and
TOOLS_FAIL+=perl produced a duplicate make target.
To handle this situation, let TOOLS_BROKEN+=perl take precedence over
TOOLS_FAIL+=perl. This is much easier than finding out in each case how
to disable the perl check in the configure script, which is most often
done by adding any of the following to CONFIGURE_ENV: PERL=#none,
ac_cv_prog_PERL=#none, ac_cv_path_PERL=#none.
