This is based on the decision The NetBSD Foundation made in 2008 to
do so, which was already applied to src.
This change has been applied to code which is likely not in other
repositories.
ok board@, reviewed by riastradh@
Skip any lines of otool output that do not start with a tab, ensuring we
only check valid files and ignore e.g. shell scripts. Older versions of
otool would simply print no output on such files, whereas newer releases
print "is not an object file".
The "split" program is not managed by the tools framework. It just has a
wrapper that is placed into the same directory. This is confusing since
intuitively, "split" is a tool like many others.
The "duplicate script for target" warning from Make is therefore correct,
albeit obscure.
Currently, neither the pkgsrc infrastructure nor pkglint check for
allowed tool names.
The TOOLS_CREATE variable is only ever appended to, without checking for
duplicates. In some rare cases, this produces warnings about
doubly-defined make targets. An example is adding USE_TOOLS+=strip to
pkgtools/pkglint:
".../mk/tools/create.mk" line 149: warning:
duplicate script for target ".../work/.tools/bin/strip" ignored
The above line number 149 is zero-based, which in reality means the
duplicate definition is in line 150.
CMAKE_PKGSRC_BUILD_FLAGS
If set to yes, disable compiler optimization flags associated
with the CMAKE_BUILD_TYPE setting (for pkgsrc these come in from
the user via variables like CFLAGS). The default is yes, but you can
set it to no for pkgsrc packages that do not use a compiler to avoid
cmake "Manually-specified variables were not used by the project"
warnings associated with this variable.
The voodoo that checks this now keeps track of which packages caused
what, and in addition to the list FORCED_PKGSRC of packages that have
been forced to non-builtin, it also produces a list FORCED_PKGSRC_REASONS
of the form pkg:causing-pkg{,causing-pkg...}.
(which means that USE_BUILTIN.pkg has been set to no because
USE_BUILTIN.causing-pkg is set to no.)
This could probably just be one list but I'm not sure if anything is
relying on the format of the current FORCED_PKGSRC.
The variable names are typically mentioned in one of these styles:
# Package-settable variables:
#
# VARNAME
# Description
# Package-settable variables:
#
# VARNAME
# Description
Lines that are indented with two tabs contain text. And if one of these
lines starts with a variable name, it is just a coincidence. A practical
example of this happening is in mk/misc/developer.mk 1.24, where PKGNAME
starts a line of description.
This switch is meant to be used by packages requiring an implementation of the
former libusb (as in devel/libusb). The original implementation can be
chosen by setting LIBUSB_TYPE to "native".
The alternative implementation libusb-compat (as in devel/libusb-compat) wraps
libusb1 (in devel/libusb1). This implementation can be chosen by setting
LIBUSB_TYPE to "compat". On NetBSD, it has the advantage of not requiring root
privileges to locate and use USB devices without a kernel driver.
This first part only imports the switch framework itself.
The pkgdb variable generated for pbulk.conf is ${PREFIX}/var/db/pkg but
bootstrap generates by default PKG_DBDIR=${PREFIX}/pkgdb.
This fix add a common PKGDBDIR variable used for both pbulk.conf and
bootstrapkit.
Using variables such as PKGSRC_USE_SSP in package Makefiles to disable SSP does
not work due to the parsing order in bsd.prefs.mk. Even if it did, it's not a
good idea to mix user and package settable variables, and would cause issues in
complex packages where bsd.prefs.mk is included early (e.g. Makefile.common).
Packages can now set {MKPIE,MKREPRO,FORTIFY,RELRO,SSP,STACK_CHECK}_SUPPORTED=no
to correctly disable security features if necessary.
Ninka can be installed from wip/ninka and analyzes each file individually,
thereby providing a much more detailed analysis than the ad-hoc method that
only looks at some COPYING files.
If Ninka is not installed, the naive fallback continues to be used.
Before, the first file that looked like a license file was considered.
The others were completely ignored. This led to a wrong license for
cross/arm-none-eabi-gcc. To prevent these cases in the future, the license
is only guessed if there is exactly one file with a typical license name.
This approach is still naive, but at least a little more precise. Replacing
the guess-license with a determine-licenses is much more complicated
though, since each source code file may have its own license declared, and
handling all these special cases leads to very complex license expressions
(like "gnu-gpl-v3 for all files, except for special.c, which is apache-2.0
or mit). This is very hard to do correctly.
It compares the license file from the package with the available licenses
in licenses/ and shows the diff to the best match.
This will hopefully make it easier for package authors to include the
LICENSE variable in the package Makefile. This variable being missing is
one of the most frequent error messages from pkglint (4187 out of 20044).
Third-party (i.e. non-pkgsrc) C toolchains (I am using chromebrew)
install to /usr/local, as that is where you can have write access.
With this, a bootstrap on ChromeOS finishes successfully.
When not using cwrappers, so far PKGSRC_MKPIE was only automatically
applied when linking using gcc(1) (when enabled). This is now also the
case for packages using ld(1) to link executables.
This is only relevant for PKGSRC_MKPIE. It partly reflects a fix that
was committed to the cwrappers for MKPIE, where the "-pie" flag was
automatically added in spite of the linker not actually creating an
executable.
This solves an issue with the command sink component of the MKPIE
wrapper for GCC, where the contents of the _MKPIE_CFLAGS.gcc and
_MKPIE_LDFLAGS.gcc variables was guessed. It is now communicated to
cmd-sink-mkpie-gcc through the environment instead.
The cmd-sink-mkpie-gcc component for PKGSRC_MKPIE support on GCC was
lagging behind the generic one. This makes sure it cannot happen again,
by invoking the generic sink right away.
It currently tackles two problems:
- gcc(1) hard-coding full paths in debugging information (with one
caveat at the moment)
- ar(1) hard-coding user IDs in archive headers
This allows packages built from the same tree and options to produce
identical results bit by bit. This option should be combined with ASLR
and PKGSRC_MKPIE to avoid predictable address offsets for attackers
attempting to exploit security vulnerabilities.
This is still disabled by default, and only supports NetBSD so far.
As discussed on tech-pkg@
Match cwrappers' expectations and place an argument per line in the
configuration. Tokenize the arguments when writing the configuration
instead of inside cwrappers.
This should fix PKGSRC_MKPIE.
This makes sure a simple "cc -o hello hello.c" will still build a valid
executable. It does not let us detect when CFLAGS or LDFLAGS are
ignored anymore, but it is legitimate for packages to expect it to work
without any additional parameter.
ld(1) does not expect "-fPIC" but it seems to be ignored by our wrappers
in this case, so no disruption is expected there.
This adds a detection for Chrome OS and Chromium OS based on /etc/lsb-release,
which sets LOWER_VENDOR, like for other Linux distros. It also sets OS_VARIANT
to the value of LOWER_VENDOR, so we can have conditionals for ChromeOS. It is
missing some things that are silently assumed to be part of Linux base
systems, such as POSIX attr support, NIS and more.
ok jperkin@
The currently implementation of SSP checks simply look for a DT_NEEDED
dependency on libssp, and doesn't yet have a way to check for it being
enabled when it is done via libc.
We can't rely on _GCC_VERSION as an accurate indicator of the GCC version,
as when using a pkgsrc GCC it will be set to 0 until the pkgsrc GCC is
available and we can detect its version. Instead, move the logic later to
after CC_VERSION is set and use that instead.
Enhance the reduce-depends.awk script to reduce a larger set of
dependencies into a single dependency. The patterns representing
intervals of version numbers (can be open-ended) are of the form:
pkg>lower
pkg>=lower
pkg<upper
pkg<=upper
pkg>lower<upper
pkg>lower<=upper
pkg>=lower<upper
pkg>=lower<=upper
These patterns are now condensed into a single dependency of the
same form. For example, given the following patterns:
pkg>=1.0
pkg>2.0
pkg<3.0
pkg<=4.0
pkg>=2.5<3.5
the reduced pattern becomes:
pkg>=2.5<3.0
Add the test script used to help with refactoring and adding the
new feature to the script.
This is a mostly complete rewrite of the script; change the
license to the standard 2-clause BSD license used by TNF.
While it would be useful, the current implementation has not been
updated even to use 'cvs add' instead of 'cvs import' (policy change
in 2012). Use the import-package.sh script currently residing in wip/
instead.
This is based on a patch submitted on 16/04/2017 on tech-pkg@ and
adapted by joerg@ for pkgtools/cwrappers. It only consists in the
missing part to actually generate PIE executables with cwrappers if
configured to do so (currently disabled by default). The aim is really
to produce safer binaries where ASLR is in use.
This part in pkgsrc is only supported on NetBSD (x86) with GCC at the
moment.
Tested on NetBSD/amd64, with and without cwrappers, with and without
PKGSRC_MKPIE (all four combinations).
Only pass "-C ${CONFLICTS}" to pkg_create(1) if ${CONFLICTS} is
non-empty; otherwise, a fatal error occurs. This allows CONFLICTS
to be defined, but empty.
Move the definitions of LDCONFIG_ADD_CMD and LDCONFIG_REMOVE_CMD
into suse_linux/emulator.mk. The commands are specific to the
Linux installed, including the path to ldconfig(8), so localize
them to the package that provides it.
Move the definition of RUN_LDCONFIG out of the emulator framework
and into the packages that use them as RUN_LDCONFIG is meant to be
a package-settable variable.
Fix the path to the Linux ldconfig(8) that was used on FreeBSD to
brand the ELF format that has been incorrect for over 10 years.
In the suse_base packages, explicitly set LDCONFIG_REMOVE_CMD to
${TRUE} since the ldconfig(8) command is provided by that package
and can't be invoked at the postremove stage because it has already
been deleted from the system by then.
Bump the PKGREVISION of all the suse_base packages because the
install scripts have changed.
Bump the PKGREVISIONs of the darwin_lib and osf1_lib packages
because the install scripts no longer try to run ldconfig(8),
which is not provided in either OpenDarwin or OSF/1.
The pkginstall framework had an UNPACK action invoked early
in the PRE-INSTALL stage to unpack scriplets that were invoked
in later stages to perform extra work for installation, removal,
or both.
Add an UNPACK action that is a no-op for any scripts that don't
support the action.
The postinstall check was improperly skipping the last N items
listed in CONF_FILES (and related) variables when verifying that
there is a multiple of N items listed. Fix the bounds check.
The shell code used to check for the existence of the source files
listed in the various CONF_FILES (and related) variables has been
refactored out into a separate macro variable. In addition, the
macro has been modified to only emit a warning if the source file
is not in ${PREFIX} and can't be found on the host system.
This supports using /dev/null as a source file, which is a common
idiom for creating an empty target file, even if ${DESTDIR} is
non-empty.
Replace @VAR@ with ${VAR} unconditionally in scripts, even if they
values are empty. This will allow for fall-through to using the
pkgtasks hard-coded defaults, usually found by searching the PATH
contents, and it avoids having ugly "@VAR@" in the generated
scripts.
changing the permission of ${PACKAGES} at the end of the script prevents that
from working.
Undo the change in r1.4 to create the All directory during setup & instead do
not change permissions.
Tested on OS X Tiger & FreeBSD.
Heads up by joerg
[1] http://mail-index.netbsd.org/pkgsrc-changes/2017/08/01/msg160826.html
The return values of the postinstall and postremove meta-tasks are
ignored unless the user-settable variable ${PKG_FATAL_ERRORS} is
"yes". ${PKG_FATAL_ERRORS} defaults to "no" to preserve the
existing behavior of install scripts by mk/pkginstall.
If ${DESTDIR} was non-empty and the source file was an absolute
path, the postinstall check target defined in mk/pkgtasks/files.mk
was improperly failing. Fix the bug by only prepending ${PREFIX}
if the source path is relative, and always prepending ${DESTDIR}.
Add definitions for the gtk-update-icon-cache(1) command to the
script header template, and a makefile fragment that creates the
data file for the "icon_themes" package task if ${ICON_THEMES} is
"yes" in the package Makefile.
Also bump pkgtasks dependency to version 1.12 for the "icon_themes"
package task and compatibility with pkgsrc/mk/pkginstall.
These should be explicitly set in the scripts, otherwise, the
pkgtasks defaults are used by locating programs in the $PATH
or are hardcoded defaults used for testing pkgtasks.
For target files listed in CONF_FILES (and other config file
variables), assert that the containing directories are named in
MAKE_DIRS or one of the other directory variables.
Change the assertion into an install-time check that the paths
listed in INSTALLATION_DIRS are relative paths or are in ${PREFIX}.
This delays the check so that some common variables used when
listing directories in INSTALLATION_DIRS are fully-expanded.
This appears to be causing issues with pkgsrc GCC attempting to insert
builtins it does not have, because we don't build libssp. unbreaks icu build.
there may be more problems from this issue in the future, but netbsd-7 is
better tested now.
Add an assertion that all paths listed in INSTALLATION_DIRS must
either begin with ${PREFIX}/ or are relative paths. Clarify in the
documentation that INSTALLATION_DIRS can list either type of path.
Some packages are using INSTALLATION_DIRS to create directory trees
outside of ${PREFIX}, e.g., under ${PKG_SYSCONFBASE} or ${VARBASE}.
However, these directories are only created as part of the
"install" target, which causes them to only be created during the
staged installation into ${DESTDIR} and not at all if their binary
packages are installed. These packages should be fixed to use
MAKE_DIRS or OWN_DIRS to create these directory trees at the proper
time.
creates or updates the checksum. Allows users to set FAILOVER_FETCH in
mk.conf unconditionally and be able to update distinfo files without
first having to remove them.
This is only actually in use where known supported.
Prepared for a while, also the default in the base system (for NetBSD and
numerous others), introduced on mailing-lists and in my talk "Hardening
pkgsrc", and finally accepted unanimously during pkgsrcCon 2017.
Used by Joyent already (according to jperkin@) and tested in EdgeBSD for a
couple years now (amd64 and i386).
This is only performed if PKG_DEVELOPER and RELRO are in use.
After a suggestion during my talk at BSDCan 2017; thanks!
Also, submitted on tech-pkg@ for review mid-June.
As a next step, it seems this can be extended to libraries, just like the
check for SHLIBS does (from which this is inspired).
Reference the notion of making compilers visible to the build
environment. Mention setting --std flags. Note that the text is
currently aspirtational relative to gcc and C++.
(Comment change only.)
ccache needs to call real compiler and it detects it with looking in $PATH.
In case of usage of /usr/local/bin/clang as the desired pkgsrc $CC option,
it will be shadowed by /usr/pkg/bin/clang if someone will build it in as a
package. Workaround the problem with setting CCACHE_PATH through
PKGSRC_MAKE_ENV. Little bit overzelaously set it to
${CCPATH:H}:${CXXPATH:H}:${CPPATH:H}, just in case that they might be in
different paths. To ensure that they are not shadowed in narrow cases,
there is an option to rename clang to xclang or similar and use it this
way in $PKG_CC.
While there, introduce new user settable option: CCACHE_LOGFILE.
If set, ccache logs to file specified in ${CCACHE_LOGFILE:Q}.
+# CCACHE_LOGFILE
+# If set to a file path, ccache will write information on what it is
+# doing to the specified file. This is useful for tracking down
+# problems.
This fixes another set of problems building large projects like Clang, LLVM
with ccache.
Sponsored by <The NetBSD Foundation>
Introduce Icon Theme cache handling framework
Icon Theme cache files are used by GTK+ and maintained with the
gtk-update-icon-cache tool. Each Icon Theme package duplicates
its own maintainance scripts: only the specified icon theme directory
differs. With this framework, if packages have ICON_THEMES=yes,
associated icon themes will be detected and their cache files will
be maintained automatically.
Change cache handling behaviour as follows:
* Icon theme caches will be updated if either gtk2+ or gtk3+
gtk-update-icon-cache tool is available.
* With installation of gtk2+ package, not only hicolor icon theme but
also any other icon theme cache files will be updated.
* Prevent removal of icon caches at deinstall, gtk3+ may be installed and
using them.
* Ditto with gtk3+, gtk2+ may not be installed now, so caches must be
maintained by gtk3+.
there are further issues with circular dependencies which need to be resolved
before the change to using find from coreutils can be made.
Revert previous change for now.
The ccache.mk file was checking for languages "c" and "c++".
New framework for C++ dialects (or revisions) was setting implicitly c++,
translating e.g. c++11 to c++.
compiler.mk set this c++ after including ccache.mk, so c++ was undefined
and ccache was ignored.
This helps to build large projects like LLVM+Clang+LLDB with ccache.
Sponsored by <The NetBSD Foundation>
@TYPE1INST@ should be replaced with ${TOOLS_PATH.type1inst}.
Before this change, the sed(1) script being generated was not
replaced @TYPE1INST@ at all, and instead generating a no-op
substitution pattern for TTMKFDIR.
LDCONFIG_ADD_CMD and LDCONFIG_REMOVE_CMD are set by emulator/*.mk
in order to update the library search paths in the emulated system
directories. Substitute for those values, but default them to the
empty string so that the default action taken by the "shlibs" task
can be used.
${CROSSBASE} and ${VARBASE} both point to important directories
for installed packages, so always substitute those values into
deinstall/install scripts.
The header template loads pkgtasks' load.subr unconditionally, so
if any script templates are used, then they imply a dependency on
pkgtasks. Trigger the dependency in the "pkgtasks" framework by
setting USE_PKGTASKS to "yes".
Add missing feature from "pkginstall" framework where the config
directory ${PKG_SYSCONFDIR} was automatically created if one of
the following conditions were true:
* PKG_SYSCONFSUBDIR was defined and non-empty.
* Any configuration files were copied into ${PKG_SYSCONFDIR}.
The platform/${OPSYS}.mk files now use platform-specific code to
set ${_OPSYS_SHLIB_TYPE} which is the default value for
${SHLIB_TYPE}. All previous uses of this script to detect the
native shared library type or native executable format have been
removed from pkgsrc.
