freebsd-ports/Mk/bsd.stage.mk

111 lines
3.1 KiB
Makefile
Raw Normal View History

Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
#
# $FreeBSD$
#
STAGEDIR?= ${WRKDIR}/stage
DESTDIRNAME?= DESTDIR
.if defined(_DESTDIR_VIA_ENV)
MAKE_ENV+= ${DESTDIRNAME}=${STAGEDIR}
.else
Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
MAKE_ARGS+= ${DESTDIRNAME}=${STAGEDIR}
.endif
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
QA_ENV+= STAGEDIR=${STAGEDIR} \
PREFIX=${PREFIX} \
LOCALBASE=${LOCALBASE} \
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
"STRIP=${STRIP}"
.if !empty(USES:Mdesktop-file-utils)
QA_ENV+= USESDESKTOPFILEUTILS=yes
.endif
.if !empty(USES:Mshared-mime-info)
QA_ENV+= USESSHAREDMIMEINFO=yes
.endif
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
CO_ENV+= STAGEDIR=${STAGEDIR} \
PREFIX=${PREFIX} \
LOCALBASE=${LOCALBASE} \
WRKDIR=${WRKDIR} \
WRKSRC=${WRKSRC} \
MTREE_FILE=${MTREE_FILE} \
GNOME_MTREE_FILE=${GNOME_MTREE_FILE} \
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
TMPPLIST=${TMPPLIST} \
- Rename check-orphans to check-plist. Keep the old for backwards-compat. - Bug fixes: makeplist/check-plist: - Fix showing directories owned by RUN_DEPENDS/LIB_DEPENDS [1] - Use proper "rmdir PATH" syntax, not "rmdir >/dev/null... PATH" which pkg will not recognize. - Never consider base /etc/mtree/BSD.usr.dist or LOCLABASE Templates/BSD.local.dist as needing @dirrm handling. check-plist: - Fix showing PORTDOCS/PORTEXAMPLES files when the OPTIONS are not set. makeplist will still suggest them. - Fix showing files installed through unselected OPTIONS as orphans, by considering "@comment file" to be ignored. [2] - Fix @sample, @fc, @fcfontsdir, @fontsdir support - Fix return status when orphans are found to be non-zero - Add note when PREFIX=!LOCALBASE - Be more clear when orphans/no orphans are found. - Add a whitelist mechanism for globally approved ignores. - Add *.bak/*.orig to orphan whitelist for now. - Fix false-positive with dirs installed to /, such as with archivers/dpkg creating /var/db/dpkg - Fix false-positive with @dirrm ending in /, such as with ports-mgmt/poudriere with a @dirrmtry share/zsh/ - There are likely still some false-positives. I fixed as many as I could find. Please let me know of others. * One in particular that is not easily fixable is installing a file into a directory owned by another port where that other port is not a run-time dependency. So the leaf port may create all of the parent dirs and never clean them up. Cleaning them up is not proper unless no other package is depending on them. This will be addressed by pkg(8) once pkg_install is EOL, or sooner. - Consider @dirrm of directories owned by run-time dependencies, or /etc/mtree/* or Templates/BSD.local.dist (at LOCALBASE) as fatal errors. These should not be removed in the plist. @comment lines are not considered for this; they will not ignore an error. ===> Checking for directories owned by dependencies or MTREEs Error: Owned by dependency: @dirrmtry share/locale/af/LC_MESSAGES Error: Owned by dependency: @dirrmtry share/locale/af Error: Owned by dependency: @dirrmtry %%PERL5_MAN3%% - Detect files in plist that do not exist in the stagedir. Pkgng already did this, but now we have it unified with this check. @comment lines are not considered for this; they will not ignore an error. ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: foo Error: Missing: @dirrmtry bar - Change orphaned output due to several new errors introduced: ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: foo - Send errors to stderr check_leftovers.sh: - Prefer longer values for PLIST_SUB. bsd.options.mk: - Fix NOPORTDOCS/WITHOUT_NLS/NOPORTEXAMPLES not unsetting their respective OPTIONS. - Mark NOPORTDOCS/NOPORTEXAMPLES deprecated and hook them into the OPTIONS_WARNING to tell users the new format. - check-stagedir.sh refactoring: - Switch to using new PLIST_SUB_SED - Use ! instead of , in sed(1) regex to allow files/dirs with ',' - Rework PORTEXAMPLES/PORTDOCS handling so it acts on PLIST_SUB_SED'd value and not absolutes. Also simplify the regex for these a bit to allow reuse. - No longer need DOCSDIRS/EXAMPLESDIR in env - Wrap long lines - Unset some vars in env when they are done being used to free space for larger sed vars - Cleanup redundant sed regexes - Add a Scripts/plist_sub_sed_sort.sh to prefer longer values when substituting over shorter values. - To make check-plist ignore a file *as an orphan* do one of the following: 1. Install it 2. post-install: ${RM} ${STAGEDIR}file 3. Put the file behind an OPTION with a PLIST_SUB: %%OPTION%%file 4. Add to plist as a @comment [2]: @comment file @comment @dirrmtry dir Reviewed by: mat (much earlier version) Discussed with: mat, antoine, bapt, swills (various bits) With hat: portmgr PR: ports/185561 [1] Reported By: Alexander Yerenkow <yerenkow@gmail.com> [1] Tested with: pkg and pkg_install Reported by: many (false-positives) [2] This is a partial solution, we may still need a plist.ignore too. It doesn't make much sense to add files in main pkg-plist we don't care about, but maybe it does since you'll see and reconsider them being ignored someday. @comment is used as all the OPTION PLIST_SUB deactivations use @comment instead of something like @ignore.
2014-04-19 08:44:56 +02:00
SCRIPTSDIR=${SCRIPTSDIR} \
WITH_PKGNG=${WITH_PKGNG} \
PLIST_SUB_SED="${PLIST_SUB_SED}" \
PORT_OPTIONS="${PORT_OPTIONS}" \
PORTSDIR="${PORTSDIR}"
.if defined(WITH_PKGNG)
CO_ENV+= PACKAGE_DEPENDS="${_LIB_RUN_DEPENDS:C,[^:]*:([^:]*):?.*,\1,:C,${PORTSDIR}/,,}" \
PKG_QUERY="${PKG_QUERY}"
.else
CO_ENV+= PACKAGE_DEPENDS=${ACTUAL-PACKAGE-DEPENDS:Q} \
PKG_QUERY="${PKG_INFO}"
.endif
.if defined(NO_PREFIX_RMDIR)
CO_ENV+= NO_PREFIX_RMDIR=1
.else
CO_ENV+= NO_PREFIX_RMDIR=0
.endif
Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
.if !target(stage-dir)
stage-dir:
@${MKDIR} ${STAGEDIR}${PREFIX}
.if !defined(NO_MTREE)
@${MTREE_CMD} ${MTREE_ARGS} ${STAGEDIR}${PREFIX} > /dev/null
.endif
.endif
# Compress all manpage not already compressed which are not hardlinks
# Find all manpages which are not compressed and are hadlinks, and only get the list of inodes concerned, for each of them compress the first one found and recreate the hardlinks for the others
# Fixes all dead symlinks left by the previous round
.if !target(compress-man)
compress-man:
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
@${ECHO_MSG} "====> Compressing man pages (compress-man)"
@mdirs= ; \
for dir in ${MANDIRS:S/^/${STAGEDIR}/} ; do \
Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
[ -d $$dir ] && mdirs="$$mdirs $$dir" ;\
done ; \
for dir in $$mdirs; do \
${FIND} $$dir -type f \! -name "*.gz" -links 1 -exec ${GZIP_CMD} {} \; ; \
${FIND} $$dir -type f \! -name "*.gz" \! -links 1 -exec ${STAT} -f '%i' {} \; | \
Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
${SORT} -u | while read inode ; do \
unset ref ; \
for f in $$(${FIND} $$dir -type f -inum $${inode} -print); do \
Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
if [ -z $$ref ]; then \
ref=$${f}.gz ; \
${GZIP_CMD} $${f} ; \
continue ; \
fi ; \
${RM} -f $${f} ; \
(cd $${f%/*}; ${LN} -f $${ref##*/} $${f##*/}.gz) ; \
done ; \
done ; \
${FIND} $$dir -type l \! -name "*.gz" | while read link ; do \
dest=$$(readlink $$link) ; \
rm -f $$link ; \
(cd $${link%/*} ; ${LN} -sf $${dest##*/}.gz $${link##*/}.gz) ;\
done; \
done
.endif
.if !target(makeplist)
makeplist: stage
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
@${SETENV} ${CO_ENV} ${SH} ${SCRIPTSDIR}/check-stagedir.sh makeplist
Add support for staging area in the ports tree The ports tree is now staged by default. With pkgng the sequence hasn't changed, the main difference is that creating package is now independent from installing it. With pkg_install, the package is now created first and make install, do install the package. New macros: - STAGEDIR: PATH to the directory where the port will be staged. - NO_STAGE: Keep the old behaviour of the ports tree (aka no staging area) Unsupported macro if NO_STAGE is not set: - MAN* with staging man page compression and handling of hardlinks and symlinks is automatically done in the stage. the manpages becomes then a "normal" plist files and should be tracked in pkg-plist. - MANCOMPRESSED the compress-man target is able to only compress when it needed. New target: - stage: this installs everything into the stage directory - makeplist: this will create a pkg-plist and print it to stdout. This is a sample plist and it should always be _reviewed_ not directly used. NOTE: with staging only what is in the plist will be installed, nothing more, meaning a port staged cannot have leftovers exect directories left. It is really important to double check the pkg-plist to make sure all the files the maintainer want to package are in! make makeplist can help in that area. The goal is to handle as quickly as possible a full removal of the NO_STAGE support. More features can be added once the full ports tree is converted to using the staging area Conversion notes: - Remove NO_STAGE - in {pre,do,post}-install * Before any usage of ${PREFIX}, ${ETCDIR}, ${PORTSDIR}, ${PORTEXAMPLES} prepend ${STAGEDIR} * Remove any thing that is done by @exec or +INSTALL script it will be done automatically when syncing packages * Remove any code to show PKG-MESSAGE it will be done automatically when syncing packages * Directory creation should remain in post-install (in particular because pkgng doesn't work like pkg_install in that area and pkgng ignores the @exec mkdir but directly pack the directory even if empty) - PORTDOCS/PORTEXAMPELS To support PORTDOCS, PORTEXAMPLES most of the time there is no more need for hacks. just add the right %%PORTDOCS%% or %%PORTEXAMPLES%% in your plist then the given files from the stagedir will or will not sync according to NOPORTDOCS and NOPORTEXAMPLES. With hat: portmgr Reviewed by: bdrewery
2013-09-23 07:56:35 +02:00
.endif
- Rename check-orphans to check-plist. Keep the old for backwards-compat. - Bug fixes: makeplist/check-plist: - Fix showing directories owned by RUN_DEPENDS/LIB_DEPENDS [1] - Use proper "rmdir PATH" syntax, not "rmdir >/dev/null... PATH" which pkg will not recognize. - Never consider base /etc/mtree/BSD.usr.dist or LOCLABASE Templates/BSD.local.dist as needing @dirrm handling. check-plist: - Fix showing PORTDOCS/PORTEXAMPLES files when the OPTIONS are not set. makeplist will still suggest them. - Fix showing files installed through unselected OPTIONS as orphans, by considering "@comment file" to be ignored. [2] - Fix @sample, @fc, @fcfontsdir, @fontsdir support - Fix return status when orphans are found to be non-zero - Add note when PREFIX=!LOCALBASE - Be more clear when orphans/no orphans are found. - Add a whitelist mechanism for globally approved ignores. - Add *.bak/*.orig to orphan whitelist for now. - Fix false-positive with dirs installed to /, such as with archivers/dpkg creating /var/db/dpkg - Fix false-positive with @dirrm ending in /, such as with ports-mgmt/poudriere with a @dirrmtry share/zsh/ - There are likely still some false-positives. I fixed as many as I could find. Please let me know of others. * One in particular that is not easily fixable is installing a file into a directory owned by another port where that other port is not a run-time dependency. So the leaf port may create all of the parent dirs and never clean them up. Cleaning them up is not proper unless no other package is depending on them. This will be addressed by pkg(8) once pkg_install is EOL, or sooner. - Consider @dirrm of directories owned by run-time dependencies, or /etc/mtree/* or Templates/BSD.local.dist (at LOCALBASE) as fatal errors. These should not be removed in the plist. @comment lines are not considered for this; they will not ignore an error. ===> Checking for directories owned by dependencies or MTREEs Error: Owned by dependency: @dirrmtry share/locale/af/LC_MESSAGES Error: Owned by dependency: @dirrmtry share/locale/af Error: Owned by dependency: @dirrmtry %%PERL5_MAN3%% - Detect files in plist that do not exist in the stagedir. Pkgng already did this, but now we have it unified with this check. @comment lines are not considered for this; they will not ignore an error. ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: foo Error: Missing: @dirrmtry bar - Change orphaned output due to several new errors introduced: ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: foo - Send errors to stderr check_leftovers.sh: - Prefer longer values for PLIST_SUB. bsd.options.mk: - Fix NOPORTDOCS/WITHOUT_NLS/NOPORTEXAMPLES not unsetting their respective OPTIONS. - Mark NOPORTDOCS/NOPORTEXAMPLES deprecated and hook them into the OPTIONS_WARNING to tell users the new format. - check-stagedir.sh refactoring: - Switch to using new PLIST_SUB_SED - Use ! instead of , in sed(1) regex to allow files/dirs with ',' - Rework PORTEXAMPLES/PORTDOCS handling so it acts on PLIST_SUB_SED'd value and not absolutes. Also simplify the regex for these a bit to allow reuse. - No longer need DOCSDIRS/EXAMPLESDIR in env - Wrap long lines - Unset some vars in env when they are done being used to free space for larger sed vars - Cleanup redundant sed regexes - Add a Scripts/plist_sub_sed_sort.sh to prefer longer values when substituting over shorter values. - To make check-plist ignore a file *as an orphan* do one of the following: 1. Install it 2. post-install: ${RM} ${STAGEDIR}file 3. Put the file behind an OPTION with a PLIST_SUB: %%OPTION%%file 4. Add to plist as a @comment [2]: @comment file @comment @dirrmtry dir Reviewed by: mat (much earlier version) Discussed with: mat, antoine, bapt, swills (various bits) With hat: portmgr PR: ports/185561 [1] Reported By: Alexander Yerenkow <yerenkow@gmail.com> [1] Tested with: pkg and pkg_install Reported by: many (false-positives) [2] This is a partial solution, we may still need a plist.ignore too. It doesn't make much sense to add files in main pkg-plist we don't care about, but maybe it does since you'll see and reconsider them being ignored someday. @comment is used as all the OPTION PLIST_SUB deactivations use @comment instead of something like @ignore.
2014-04-19 08:44:56 +02:00
.if !target(check-plist)
check-plist: stage
@${ECHO_MSG} "====> Checking for pkg-plist issues (check-plist)"
@${SETENV} ${CO_ENV} ${SH} ${SCRIPTSDIR}/check-stagedir.sh checkplist
@${ECHO_MSG} "===> No pkg-plist issues found (check-plist)"
.endif
.if !target(check-orphans)
- Rename check-orphans to check-plist. Keep the old for backwards-compat. - Bug fixes: makeplist/check-plist: - Fix showing directories owned by RUN_DEPENDS/LIB_DEPENDS [1] - Use proper "rmdir PATH" syntax, not "rmdir >/dev/null... PATH" which pkg will not recognize. - Never consider base /etc/mtree/BSD.usr.dist or LOCLABASE Templates/BSD.local.dist as needing @dirrm handling. check-plist: - Fix showing PORTDOCS/PORTEXAMPLES files when the OPTIONS are not set. makeplist will still suggest them. - Fix showing files installed through unselected OPTIONS as orphans, by considering "@comment file" to be ignored. [2] - Fix @sample, @fc, @fcfontsdir, @fontsdir support - Fix return status when orphans are found to be non-zero - Add note when PREFIX=!LOCALBASE - Be more clear when orphans/no orphans are found. - Add a whitelist mechanism for globally approved ignores. - Add *.bak/*.orig to orphan whitelist for now. - Fix false-positive with dirs installed to /, such as with archivers/dpkg creating /var/db/dpkg - Fix false-positive with @dirrm ending in /, such as with ports-mgmt/poudriere with a @dirrmtry share/zsh/ - There are likely still some false-positives. I fixed as many as I could find. Please let me know of others. * One in particular that is not easily fixable is installing a file into a directory owned by another port where that other port is not a run-time dependency. So the leaf port may create all of the parent dirs and never clean them up. Cleaning them up is not proper unless no other package is depending on them. This will be addressed by pkg(8) once pkg_install is EOL, or sooner. - Consider @dirrm of directories owned by run-time dependencies, or /etc/mtree/* or Templates/BSD.local.dist (at LOCALBASE) as fatal errors. These should not be removed in the plist. @comment lines are not considered for this; they will not ignore an error. ===> Checking for directories owned by dependencies or MTREEs Error: Owned by dependency: @dirrmtry share/locale/af/LC_MESSAGES Error: Owned by dependency: @dirrmtry share/locale/af Error: Owned by dependency: @dirrmtry %%PERL5_MAN3%% - Detect files in plist that do not exist in the stagedir. Pkgng already did this, but now we have it unified with this check. @comment lines are not considered for this; they will not ignore an error. ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: foo Error: Missing: @dirrmtry bar - Change orphaned output due to several new errors introduced: ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: foo - Send errors to stderr check_leftovers.sh: - Prefer longer values for PLIST_SUB. bsd.options.mk: - Fix NOPORTDOCS/WITHOUT_NLS/NOPORTEXAMPLES not unsetting their respective OPTIONS. - Mark NOPORTDOCS/NOPORTEXAMPLES deprecated and hook them into the OPTIONS_WARNING to tell users the new format. - check-stagedir.sh refactoring: - Switch to using new PLIST_SUB_SED - Use ! instead of , in sed(1) regex to allow files/dirs with ',' - Rework PORTEXAMPLES/PORTDOCS handling so it acts on PLIST_SUB_SED'd value and not absolutes. Also simplify the regex for these a bit to allow reuse. - No longer need DOCSDIRS/EXAMPLESDIR in env - Wrap long lines - Unset some vars in env when they are done being used to free space for larger sed vars - Cleanup redundant sed regexes - Add a Scripts/plist_sub_sed_sort.sh to prefer longer values when substituting over shorter values. - To make check-plist ignore a file *as an orphan* do one of the following: 1. Install it 2. post-install: ${RM} ${STAGEDIR}file 3. Put the file behind an OPTION with a PLIST_SUB: %%OPTION%%file 4. Add to plist as a @comment [2]: @comment file @comment @dirrmtry dir Reviewed by: mat (much earlier version) Discussed with: mat, antoine, bapt, swills (various bits) With hat: portmgr PR: ports/185561 [1] Reported By: Alexander Yerenkow <yerenkow@gmail.com> [1] Tested with: pkg and pkg_install Reported by: many (false-positives) [2] This is a partial solution, we may still need a plist.ignore too. It doesn't make much sense to add files in main pkg-plist we don't care about, but maybe it does since you'll see and reconsider them being ignored someday. @comment is used as all the OPTION PLIST_SUB deactivations use @comment instead of something like @ignore.
2014-04-19 08:44:56 +02:00
check-orphans: check-plist
.endif
.if !target(stage-qa)
stage-qa:
bsd.stage.mk: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=sh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Reviewed by: bapt Approved by: portmgr (bapt)
2013-10-31 20:07:38 +01:00
@${ECHO_MSG} "====> Running Q/A tests (stage-qa)"
@${SETENV} ${QA_ENV} ${SH} ${SCRIPTSDIR}/qa.sh
.endif