This is an important step to prepare the ports tree for VARIANTS(aka flavours)
and subpackage by making the dependency code easier to deal with.
Change:
- Externalize in a proper shell script the code that was an inlined shell script
- Add better validation on the syntaxe used
- test after the dependency has been installed that it actually really fulfill
the pattern searched (improving QA)
- Unify lib-depends with other dependency checks
- Make ${PORTSDIR} not mandatory anymore in _DEPENDS lines:
aka pattern:${PORTSDIR}/category/port can now be written pattern:category/port
/!\ Please to not use this syntax yet! poudriere have received a fix to be
able to handle this new syntax (but no new release of poudriere has it yet)
portmaster/portupgrade hasn't been checked. if one cares about those last 2 it
would be really nice to provide patches to them!
- Remove _DEPENDS_ALWAYS it has half broken for a while and did not really make
sense.
- Keep STRICT_DEPENDS for now it might not be necessary anymore given all the
new checks added, but until someone confirms it is worth keeping it.
Note that all the env passed are prefixed by 'dp_' to avoid polluting children
make
Differential Revision: https://reviews.freebsd.org/D2897
Reviewed by: antoine
Exp-run by: antoine
- Add --localstatedir=/var to _LATE_CONFIGURE_ARGS (like --mandir) but not
when CONFIGURE_ARGS already sets it. (GNU configure scripts set it to
PREFIX/var when PREFIX != /usr.)
- Add --localstatedir="${PREFIX}/var" to CONFIGURE_ARGS in some ports so
they aren't affected by this change (for now at least). This commit is
meant to ensure that new ports don't make the same mistake.
- games/acm: the configure script in this port is very old; instead of
patching it more, just replace GNU_CONFIGURE with HAS_CONFIGURE.
- irc/charybdis: it already used /var but adding --localstatedir=/var
changed the behaviour of the configure script; adjust the port to this.
PR: 199506
Exp-run by: antoine
Approved by: portmgr (antoine)
Skip ascii files. This significantly speeds up stage-qa on ports with a
considerable amount of text files being installed.
Differential Revision: https://reviews.freebsd.org/D2121
Approved by: bdrewery
Xorg is now looking in ${LOCALBASE}/share/fonts by default
Xorg now accepts symlinks in etc/X11/fontpath.d (as decribed in Xserver(1))
Large cleanup on lots of font ports
All fonts are now properly dynamically generating fonts.dir and fonts.scale
instead of sometime overwriting existing ones)
All fonts are generating fontconfig's cache
Improve consistency in fonts ports
Change the way the dependencies are registred so that now it follows the same
mechanism as when the *_DEPENDS making the dependency registration system ready
to allow multiple packages from the same origin such as flavours and subpackages
Also allowing to install twice the same port one built with python33 and the
other built with python27 for example.
PR: 19702
With help of: antoine
Exp-run: portmgr (antoine)
the makeplist code does not use parse_plist(). Rename the param to
'parse_comments' to be more clear on what it is for. This is kept for
upcoming refactoring.
With hat: portmgr
this prevents stage violations during build of ports using gconftool-2
- White list etc/gconf/gconf.xml.defaults from stage orphans as gconftool-2
is unpredictable
With hat: portmgr
It makes upgrading from one Perl major version to another way easier. For
binary package users, it means pkg upgrade will detect the libperl.so.x.yy
change, and reinstall the affected packages. For users using ports, it will
save rebuild time as it's easier to detect what ports really need to be
rebuilt.
PR: 195821
Differential Revision: https://reviews.freebsd.org/D1241
Submitted by: mat
Reviewed by: antoine, bdrewery
Exp-run by: antoine
With hat: perl
Sponsored by: Absolight
directories outside prefix
- Whitelisting orphan directories not starting with / is no longer necessary
- Adjust missing directories check to use a list of all directories instead
of the dirs from make-plist
Differential Revision: https://reviews.freebsd.org/D1101
Reviewed by: bapt
With hat: portmgr
pkg-1.3.8's auto directory handling makes this a non-issue.
We should add a new check to warn of possibly needing @dir for empty dirs
though.
With hat: portmgr
- Plist parser do not accept modes like o+w g+s u+r
- @owner/@group with no argument reset the default ownership to root/wheel
- Plug regression tests into the release mechanism as mandatory
- Allow url in repository configuration to be overriden by another configuration
file
- If one of -P, -I or -R is not explicitly given on the command line do not emit
error messages while searching for a mechanism to find data about the latest
available ports.
- External keywords can now take arguments
- Directories under PREFIX are automatically handled and removed if needed
- MTREE are not packaged anymore neither extracted on final installation
- OS major checking (to determine if pkg is running on an upgraded base system)
uses the same mechanism as ABI detection instead of relying on UNAME_r which
can have false positivie on jails
- Add @dir keyword for plist which is used to package explicitly directories
(directories with non root/wheel or outside prefix) and empty directories
(@dirrm and @dirrmtry are now considered as deprecated)
- root/wheel are now considered as the default user/group when creating a
package except @owner/@group are used to change that behaviour
- Import newer version of libucl which improves error messages in case of
problem parsing manifest or configuration files
- Fix pkg register so it checks for conflicts before actually copying files on
to the filesystem
- pkg now support a new no_provide_lib annotation to preventing automatic
populating of provided libs (useful for packages bundling libraries like wine)
- Improve documentations
- Ability to select usage of IPv4/IPv6 via config, repository config or command
line
- Lots of bug fixes
Side effects:
- Convert to install-mtree target into a no-op target Add empty directories to
plist of ports that needs them to run:
- share/aclocal to devel/automake and devel/automake14
- share/applications to devel/desktop-file-utils
- share/xml share/sgml to textproc/xmlcatmgr
- Prevent check-plist to issue warnings about some @dir*
- Adjist check_leftovers.sh to not take in account anything related to mtree
- Bump the default required version of pkg to 1.3.8
the port Makefile, into a stage-qa error. All ports that would trigger
this error have been converted. Many thanks to all people involved in
this, especially Dmitry Marakasov (amdmi3) who handled most ports.
At this moment over 2200 ports have USES=libtool and over 20000 library
dependencies between packages have been removed.
This also marks the point where :keepla is no longer special. It is now
only needed if a port uses *.la files at run-time.
QA-run by: antoine
Approved by: portmgr (antoine)
Merge back bsd.pkgng.mk into bsd.port.mk
Add a note about @stopdaemon not being supported anymore
With hat: portmgr
Differential Revision: https://reviews.freebsd.org/D693
Those are harmless (records from which file they were generated from) and
removing them would require ugly hacks according to hrs@
PR: ports/192933
Differential Revision: https://reviews.freebsd.org/D684
Reviewed by: bapt, hrs
With hat: portmgr
run dependencies if the port being checked has no RUN_DEPENDS. This could
cause false-positive "Possibly owned by dependency" warnings.
With hat: portmgr
does not incorrectly ignore plist items in %%DOCSDIR%% and %%EXAMPLESDIR%%.
It was incorrectly ignoring these items when %%PORTDOCS%% or %%PORTEXAMPLES%%
was missing. This manifested as a failure in 'make package' later when pkg(8)
detected the missing files.
With hat: portmgr
- .DS_Store files
- some VCS related files
Note: it's still recommended to not extract/install those files if they are
not needed
PR: ports/189859
Submitted by: adamw (first version)
Reviewed by: bapt
With hat: portmgr
and @cwd has changed away from PREFIX:
plist:
@cwd /
@dirrmtry var/spool/dma
====> Checking for pkg-plist issues (check-plist)
===> Parsing plist
DIR '/usr/local/etc/dma' CWD: /usr/local
DIR '/usr/local/var/spool/dma' CWD:
DIR 'share/licenses/dma-v0.9,1' CWD: /usr/local
DIR '/usr/local/share/licenses' CWD: /usr/local
===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: @unexec rmdir "/var/spool/dma" >/dev/null 2>&1 || :
===> Checking for directories owned by MTREEs
===> Checking for directories handled by dependencies
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: @dirrmtry var/spool/dma
===> Error: Plist issues found.
With hat: portmgr
Reported by: erwin
Separate file(1) and stat(1) output by a newline and read each line
individually to avoid losing spaces.
- Use <<-EOF with heredocs to avoid bad indent
- Add some more comments
- Quote filenames in output so it is less unclear
Spotted by: mandree
Reviewed by: antoine
With hat: portmgr
when doing plist checks. This could leak out if an unsupported format
was used such as 'rmdir -p' or 'rmdir dir1 dir2 dir3'
With hat: portmgr
Reported by: antoine
Pointyhat to: bdrewery
change the cwd for commented lines (such as %%OPTION%%file). Reset it
back to the normal on the first non-commented line.
- Sort files after stripping out @comment
With hat: portmgr
- Split dependency-owned check its own and make into a non-fatal warning.
This is not black-and-white. Python/Rubygem ports cleanup the lib dir
for every installed package, and for python/ruby-gems too. This is fine,
the extensions should not clean them up then; the error is valid here.
However, there's other frameworks which don't cleanup in the master package
and yield unexpected results. An example is that php does not cleanup
etc/php itself. Only extensions do. So if an extension depends on another
extension it will complain that the dependent extension owns etc/php,
so we remove from bsd.php.mk, now it's a leftover. The proper place is in the
php port itself, but it's unknown how many other cases there are like this.
So for now make this non-fatal and into a warning.
This is all mostly moot once pkg auto handles directories anyhow.
With hat: portmgr
Discussed with: antoine
- Bug fixes:
makeplist/check-plist:
- Fix showing directories owned by RUN_DEPENDS/LIB_DEPENDS [1]
- Use proper "rmdir PATH" syntax, not "rmdir >/dev/null... PATH" which
pkg will not recognize.
- Never consider base /etc/mtree/BSD.usr.dist or LOCLABASE
Templates/BSD.local.dist as needing @dirrm handling.
check-plist:
- Fix showing PORTDOCS/PORTEXAMPLES files when the OPTIONS are not set.
makeplist will still suggest them.
- Fix showing files installed through unselected OPTIONS as orphans,
by considering "@comment file" to be ignored. [2]
- Fix @sample, @fc, @fcfontsdir, @fontsdir support
- Fix return status when orphans are found to be non-zero
- Add note when PREFIX=!LOCALBASE
- Be more clear when orphans/no orphans are found.
- Add a whitelist mechanism for globally approved ignores.
- Add *.bak/*.orig to orphan whitelist for now.
- Fix false-positive with dirs installed to /, such as with
archivers/dpkg creating /var/db/dpkg
- Fix false-positive with @dirrm ending in /, such as with
ports-mgmt/poudriere with a @dirrmtry share/zsh/
- There are likely still some false-positives. I fixed as many as I could
find. Please let me know of others.
* One in particular that is not easily fixable is installing a file into
a directory owned by another port where that other port is not a run-time
dependency. So the leaf port may create all of the parent dirs and never
clean them up. Cleaning them up is not proper unless no other package is
depending on them. This will be addressed by pkg(8) once pkg_install is
EOL, or sooner.
- Consider @dirrm of directories owned by run-time dependencies, or
/etc/mtree/* or Templates/BSD.local.dist (at LOCALBASE) as fatal errors.
These should not be removed in the plist. @comment lines are not
considered for this; they will not ignore an error.
===> Checking for directories owned by dependencies or MTREEs
Error: Owned by dependency: @dirrmtry share/locale/af/LC_MESSAGES
Error: Owned by dependency: @dirrmtry share/locale/af
Error: Owned by dependency: @dirrmtry %%PERL5_MAN3%%
- Detect files in plist that do not exist in the stagedir. Pkgng already did
this, but now we have it unified with this check. @comment lines are
not considered for this; they will not ignore an error.
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: foo
Error: Missing: @dirrmtry bar
- Change orphaned output due to several new errors introduced:
===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: foo
- Send errors to stderr
check_leftovers.sh:
- Prefer longer values for PLIST_SUB.
bsd.options.mk:
- Fix NOPORTDOCS/WITHOUT_NLS/NOPORTEXAMPLES not unsetting their respective
OPTIONS.
- Mark NOPORTDOCS/NOPORTEXAMPLES deprecated and hook them into the
OPTIONS_WARNING to tell users the new format.
- check-stagedir.sh refactoring:
- Switch to using new PLIST_SUB_SED
- Use ! instead of , in sed(1) regex to allow files/dirs with ','
- Rework PORTEXAMPLES/PORTDOCS handling so it acts on PLIST_SUB_SED'd
value and not absolutes. Also simplify the regex for these a bit to
allow reuse.
- No longer need DOCSDIRS/EXAMPLESDIR in env
- Wrap long lines
- Unset some vars in env when they are done being used to free space for
larger sed vars
- Cleanup redundant sed regexes
- Add a Scripts/plist_sub_sed_sort.sh to prefer longer values when substituting
over shorter values.
- To make check-plist ignore a file *as an orphan* do one of the following:
1. Install it
2. post-install: ${RM} ${STAGEDIR}file
3. Put the file behind an OPTION with a PLIST_SUB: %%OPTION%%file
4. Add to plist as a @comment [2]:
@comment file
@comment @dirrmtry dir
Reviewed by: mat (much earlier version)
Discussed with: mat, antoine, bapt, swills (various bits)
With hat: portmgr
PR: ports/185561 [1]
Reported By: Alexander Yerenkow <yerenkow@gmail.com> [1]
Tested with: pkg and pkg_install
Reported by: many (false-positives)
[2] This is a partial solution, we may still need a plist.ignore too. It
doesn't make much sense to add files in main pkg-plist we don't care about,
but maybe it does since you'll see and reconsider them being ignored
someday. @comment is used as all the OPTION PLIST_SUB deactivations use
@comment instead of something like @ignore.
- Avoid a redundant 'test -d'
- Properly show %%PORTEXAMPLES%% %%PORTDOCS%% where needed
- Avoid some forking for always-ignored files
With hat: portmgr
This will contain all approved/whitelisted leftovers in a single
reference. It will also allow us to update the whitelist without
needing a new release of poudriere/tinderbox and forcing users
to update those tools.
Usage: env PORTSDIR=... check_leftovers.sh category/port
stdin:
- missing-file
+ new-file
M modified-file reason...
stdout:
same -/+/M format, but with files substituted, without approved
whitelisted files, and hides any directories already in plist.
- Add /var/qmail/queue to the list of whitelisted directories just as
/var/mail is. This queue directory contains mails in flight and may receive
or have mail removed during an upgrade.
Obtained from: Mostly poudriere
Reviewed by: antoine
With hat: portmgr
This means that files referrencing stagedir or linked to stagedir will
now be fatal errors as intended.
- Stop polluting global namespace with IFS changes
- Speedup find(1) -exec usage by execing far less
- Ignore known false-positive/harmless stagedir files in paths() [2]
Reported by: eadler [1]
Discussed with: antoine [2]
With hat: portmgr
positives as some .o files may embed the source path [1]
- Only check shebang of files with executable bit set
Requested by: bapt [1]
With hat: portmgr
by setting IFS to just LF
- in check-stagedir.sh, properly detect unset environmental variables,
and list them all
- in check-stagedir.sh, also list the full prefix as known, not just
its parents
PR: 184039
Approved by: portmgr (bapt)
Bugfixes:
* @cmd in pkg-plist is now properly handled.
- It was previously treated the same as though there was a directory
following it, missing the prefix. (ordering matters in case...esac)
- Due to the cwd=${PREFIX} inside the while read line loop, state
tracking was broken and every new line assumed that cwd were the
prefix.
* stage-qa no longer complains about unstripped binaries if debugging is
active (WITH_DEBUG set && WITHOUT_DEBUG unset).
* The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its
build step.
Additions:
* The plist parser now understands @unexec rmdir ... || ... lines,
including those with redirections, so that there are no false
positives for directories stripped with @unexec rmdir (usually
happens on stuff installed outside $PREFIX, as in /var).
* The system's root and var mtrees are now also expanded to avoid
false @dirrm positives if a port installs directories under /var
and has to create parents in the stagedir that are present in a fully
installed system (i. e. in the real $PREFIX).
* Given that pkg_create is deemed beyond repair with respect to deleting
files outside prefix, generate @unexec rmdir statements for such
directories, rather than @dirrmtry, to sidestep the problem.
Speedups:
* the orphan check now generates sorted lists of staged files,
and plisted/mtree files, and compares them with comm(1).
This saves us the overhead of running one grep process per file
and up to two per directory, and defers the actual list
processing to a shell utility. Complexity has not changed,
but overhead per item has.
* the orphan check now uses one file for directories and one file for
files mentioned in pkg-plist, so we need not decorate them with "dir "
and parse them out any longer.
* qa.sh's shebang scanner only looks at the first line of a file,
sed is told to exit from the 2nd line.
Other Changes:
* Split the makeplist/check-orphans logic out of bsd.stage.mk,
it is too unwieldy to maintain in make-escaped shell syntax,
and permits shell tracing with "SH=sh -x" (including quotes!)
* Unify the functions "makeplist" and "check-orphans" in one
script. The only difference is that makeplist assumes an empty
pkg-plist, whereas check-orphans parses it.
* overhaul the mtree extractor, avoiding awk.
Reviewed by: bapt
Approved by: portmgr (bapt)
failure on every port that does not use shared-mime-info or need it.
Must return 0 at end of functions or it considers the last command ran as
the return code.
Reported by: eadler
With hat: portmgr
A couple of Q/A tests are done if the DEVELOPER macros is set in make.conf
Right now the tests are:
- Check if the symlinks are properly created
- Check if the binaries are stripped (just warn)
- Check if the STAGEDIR or the WORKDIR are referenced in the final files
- Check if the ports provide script with bad shebangs.
