- Raise 'package' phase timeout to 2 hours
- image: Use makefs(2) to generate ufs2 disks
- image: makefs use -s for size
- HTML json top: Remove disconnected code
- HTML json jail: Fix 'latest' not populating always.
- pkgclean: Exit 0 if no packages are left to cleanup.
- pkgclean: Fix -a to work properly
- err: Don't show 'Error' if exiting 0
- bulk/pkgclean: Don't allow specifying '' as a package
- pkgclean -a: Remove pkg repository as well
- pkgclean -a: Add a note about manually removing the PACKAGES directory
- Don't fail if 'ulimit -a' fails somehow
- LOCAL_MTREE_EXCLUDES: Add missing leading '.'
- Update sh from FreeBSD head @ r317912
- jail: Add -D option to keep git history
- jail: Document -D option for jail creation
- Fix ZFS rollback handling to create needed file for checking for race
after a rollback.
- pkg/dist/logclean: Make size calculation account for compressed ZFS
datasets.
- Add a 'logclean' command that cleans build logs older than N days, or
builds over a certain count, by filter.
- Store temp and lock files in $(mktemp -dt poudriere), and remove on exit.
- Change SHASH_VAR_PATH out of umounting MASTERMNT on exit
- USE_TMPFS=yes: extend to data as well as wrkdirs
- JSON jail: Ignore broken builds rather than corrupting the data
- HTML JSON: Fix not writing on exit, and writing to /.data.json[.mini]
- blacklist: Fix loading of JAIL-SETNAME-blacklist
- Don't bother statting or creating pkg cache dirs when deleting them at startup.
- pkgclean: Clean cache files as well
- poudriere.8: Fix reference to native-xtools and document use of /usr/src
- jail -d: Fix not cleaning up jail's temporary mountpoint directory
- jail -uc: Don't leak host /etc/src-env.conf into jail builds
- timestamp: Fix not exiting when stdin closes.
- timestamp: Fix not properly handling newlines.
- Fix umounting when using MDMFS
- Make jail -k try harder to cleanup any messes left behind.
- Add an ALLOW_NETWORKING_PACKAGES to allow networking for select packages
- testport: Support -B
- Only load linux64 if the host is newer than ~10.2 Current that had it
- Determine UMOUNT_NONBUSY at startup, which allows jail -k to benefit
- jail cleanup: Fix only the first 2 builders cleaning sometimes on older
systems.
- build_pkg: Assert that rollbackfs worked
- Rework ZFS rollback race fix by creating the file at markfs time.
- Actually install cache.sh
- Fix very old bug with ZFS rollback not succeeding due to
race with zfs rollback + zfs snapshot.
https://www.illumos.org/issues/7600
- bulk/status summaries: Fix build time being wildly wrong
- MOVED: Convert to using shash
- jail -u: Don't show /compat mount warning.
- Automatically handle making some internal paths relative when
doing 'cd'
- Add a per-run caching mechanism for commands
- Cache ZFS list (for mountpath) lookups.
- jail -u, etc: Fix ZFS snapshots not being created after 33297b28f.
- destroyfs: Avoid zfs_getfs() if not needed
- testport: Support MOVED for main port being tested
- locks: Disable INT/TERM traps while in here
- HTML JSON is now generated with TERM delayed, meaning leftover tempfiles
in the logdir should be fixed and the JSON should be more consistent.
- poudriere: Allow -v to increase VERBOSE since not all sub-commands allow it.
- poudriere: Allow -A to force colors, even if not in a TTY
- HTML JSON: Use relative paths for reading/writing files in logdirs
- prefix_output: Append into proctitle to make debugging simpler
- testport -c: Fix after 88334520d1159.
- Introduce HTML_TYPE to select if HTML files are inline or hosted
- Rewrite parsing of 'bulk -f' and 'bulk PORT1 PORT2' (and testport)
handling of ports into the queue.
- Update sh to FreeBSD head r315005.
- compute_deps_pkg: Give a hint about failed pooldir creation being from
duplicates. This is still undergoing further changes. It's technically
an assertion still that is has proven to be possible.
- Support copying a static ccache into the jail with CCACHE_STATIC_PREFIX.
This is to support using something like devel/ccache-memcached-static
in the jail to support memccached without any bootstrap or cyclic
dependency problems.
- Fix spurious SIGALRM on poudriere processes.
- Add a LOCAL_MTREE_EXCLUDES to allow appending to the list of directories
to ignore in the mtree checks.
- Document RESTRICT_NETWORKING
- Fix regression from last build not causing ZFS+USE_TMPFS=all to
rollback to the clean snapshot before cloning.
- Fix mktemp using TMPDIR='' by default (thus /). This broke
some non-root operations.
- options: Fix harmless warning about missing 'arch' file.
- Fix very old crash with SIGINFO while cleaning the queue after
a port is done building.
- Revert a change for port statistics changes which may have resulted in
warnings such as:
mv: rename /usr/local/poudriere/data/logs/bulk/103i386-default/434652/.tmp..poudriere.stats_ignored to /usr/local/poudriere/data/logs/bulk/103i386-default/434652/.poudriere.stats_ignored: No such file or directory
Note that the change here does not restore ignoring errors like
the previous code did. So some odd crashes may occur, but all
callers of this code were audited and should be ignoring the
error anyhow.
- Fix build on <head
- Fix startup depqueue errors if the listed ports have duplicates.
- Fix NO_ZFS/USE_TMPFS=all jail cloning error:
[00:00:00] ====>> Creating the reference jail...realpath: Illegal option -r
- NO_ZFS now overrides existing ZFS datasets; ZFS won't be used at all if
NO_ZFS is set.
- ZFS datasets are now looked up with 'zfs list' rather than 'mount -t zfs',
which fixes some cases of ZFS datasets being left around after an error.
- Properly ignore /proc without invoking pseudofs kernel code.
- mtree: Remove exclusion of /usr/home
- mtree: Remove exclusion of .p4config
- mtree: Restore ignoring /var/tmp in prebuild/prestage.
- mtree: Don't descend into /dev, avoiding devfs_readdir, etc.
- mtree: Avoid descending into all directories.
- gather_distfiles: Use port_var_fetch to lessen needed make executions
- sh: Make touch a builtin
- Fix commit c0daac2f: /dev/null is a special character device, not directory.
- pwait builtin: Properly restore SIGALRM on exit
- pkgclean/distclean: Properly support PREPARE_PARALLEL_JOBS from
poudriere.conf
- Default PREPARE_PARALLEL_JOBS to 1.25*PARALLEL_JOBS
- Poudriere now sets process title for some operations via setproctitle(2).
- Dependency calculation majorly refactored to fetch more make variables
at once rather than fetching some and then more later. This rework also
speeds up some startups by up to 60% by keeping the compute_deps work
queue full and more parallelized. This is especially seen in meta
port lists. The storage for the make variables fetched is not very
efficient yet and will improve more in the future.
- Add more sh builtins: pwait, mkdir, rename, setproctitle, rmdir, mktemp
- nohang: Do read before logfile comparison.
- update_stats: Stop building from html_json coproc after done building
- More work to use relative paths, thus reducing namecache contention.
- rm is bundled if the base system version does not support rm -x.
- Determine at runtime if umount -n (MNT_NONBUSY) is supported
- buildlog: Properly show limits in jail
- The "balance_pool" operation after a port is done building is now
asynchronous.
- Fix some cases where stats were not updated on early exit.
- Detect race during jail startup; prevent multiple concurrent jail
startups for the same jail+set+portstree.
- Some unexpected older packages may be deleted during startup.
- bulk: Change dependency information to -v rather than -vv
- Limit parallel umounts if batched free vnode lists not supported.
- Only fetch LIB/RUN_DEPENDS and OPTIONS if the CHECK_CHANGED_DEPS/OPTIONS
is set
- Add some missing license headers in HTML code.
- run_hook: cd to / before running hook to avoid umount problems.
- Pass -DNOCLEANDEPENDS when cleaning to not clean dependency dirs.
- Slightly reword some of the port job statuses.
- api: Allow evalling from stdin
- Move /var/db/freebsd-update exemption into common_mtree.
- More tests added
- Fix PORTBUILD_UID lookup when user doesn't exist in jail.
- configure: Don't prefer GCC (poudriere build only)
- Manpage updates
- Remove pkg_install support.
- Remove NO_STAGE support.
- Remove use of umount -f where possible.
- Use relative paths where possible to avoid namecache contention.
- USE_CACHED reworked to use POSIX mq (will soon change again)
- PRIORITY_BOOST is fixed to boost dependencies as well now with
USE_PTSORT enabled (default yes).
- Load linux64 when needed
- Improved ports_env support (less executions during ports interations)
- Show resource limits in build log
- Limit filedescriptors to 1024 by default with MAX_FILES
- Shutdown/unmount build jails in parallel
- Don't leak mount path in build log from ident(1)
- sh: Add some builtins to avoid fork/execs (mostly poudriere C helpers):
cachec, clock_monotonic, date +%s, dirempty, locked_mkdir, realpath,
sleep
- BUILD_AS_NON_ROOT: Remove redundant chown -R.
- Add HTML_JSON_UPDATE_INTERVAL (def: 2) to allow overriding JSON update
interval.
- Add support for USE_TMPFS=no
- jail: install the kernel requested with -K when using src method
- jail -c -m src: Add support for building the src tree with -b.
- jail -u -m src: Update the version for the jail as well.
- jail -c: Allow @rev for releng/X.Y branch
- jail: Switch to http protocol for ftp-archive.
- ports: Do not create a FS when '-m none'
- ports -d: Do not delete a ports tree created with method=none or -.
- ports -d: Confirm if in a TTY
- ports -d: Really delete the zfs dataset when TMPFS_ALL is used
- ports: Add git+http support
- ports/jail: Auto pass PROXY env vars from parent environment
- image: only isos have a restriction in the image name
- image: If the images dir does not exist, attempt to create it
- image: Handle a relative path with -o
- image: Various fixes.
- image: usb|*firmware|rawdisk now require a kernel and image size.
- testport: Fix fdescfs/procfs/linprocfs not being mounted.
- testport: Fix limits not being applied
- testport+SIGINFO: Fix very old deadlock with update_stats.
- pkgclean: Add hook: pkgclean done <DELETED_FILES: 0|1> <BUILT_REPO: 0|1>
- pkgclean: Cleanup corrupt packages
- pkgclean: Act on .latest directory for ATOMIC_PACKAGE_REPOSITORY
- poudriere.conf: Don't suggest changing svn.FreeBSD.org to a mirror anymore.
- QEMU+Arm64: Coppy the latest aarch64-binutils ld into the jail at startup.
- QEMU+Arm64: Link /usr/bin/ld to /nxb-bin/usr/bin/ld.
- QEMU+Arm64: Tell about copying ld
* QEMU+Arm64: Note that this is not working properly yet due to the
aarch64-binutils port not providing a static ld.
- options: Properly setup ARCH/MACHINE in make.conf for cross-builds
Changes:
- jail: Fix usage for -m src=PATH
- jail -v: Note that non-listed methods used the value only for display.
- QEMU: Copy the latest emulator into the jail at startup. This fixes
the jail running into bugs with qemu that are fixed with the latest
qemu. It was only copying it in during creation before.
- bulk: Avoid redundant cleaning message with jailversion cleanup.
- Add and document PORTBUILD_UID and CCACHE_DIR_NON_ROOT_SAFE. [1]
PR: 209786 [1]
Changes:
- Fix incorrect handling of null-mounting /nxb-bin in jail
- Disable BUILD_AS_NON_ROOT if CCACHE_DIR is set [1]
- Fix mount path too long when using linprocfs
PR: 209786 [1]
Changes:
- jail/ports -u: Default git pull to use --rebase.
- destroyfs: Run chflags -R 0 and on a smaller subset of files.
- ports: Ignore failures when removing the ports tree
- jail: Ignore errors trying to remove the jail
- Create port-specific directories in do_portbuild_mounts.
- Rework the cleanup to respect -x inside of the directory.
- Support -(j|p|z)<param> without space for poudriere.conf lookups
- hash_var_name: Quote the pattern to _gsub to avoid globbing local files.
- Update sh from head @ r296813
- Switch BUILD_AS_NON_ROOT as default to on.
Changes:
- Optimize jail startup/cloning/rollback and mtree comparisons.
- api: Make more useful
- check_fs_violation: Use mktemp rather than hard-coded tmpfile
- Fix false-positive 'Leftover processes' warning after disabling jexecd
- Conditionalize the jexec logic around USE_JEXECD and add back missing jkills.
- QEMU: Remove _MAKE_JOBS not present in release-3.1 and noted to break
builds by sbruno
- Kill processes before remounting/rolling back the builder FS, rather than
after.
- jail -c: Support older FREEBSD_HOST=ftp.freebsd.org URL schemes still. [1]
- jail -c: Support pre-distributed MANIFESTs for checksum comparing.
PR: 206848 [1]
Changes:
- jail -c: Always copy in QEMU emulator, even if not building from src.
- jail -c: Ensure MANIFEST is not empty or give an error.
- Clear CMD_ENV to disallow sneaking in environment vars
- Update libnv to the latest head @ r293361
- Update sh from head @ r293359
- Null-mount read-only /rescue and /nxb-bin into the jails as they will not
change.
- Add a MUTABLE_BASE option (default yes) to experiment with null-mounting of
base.
- Prefer https://download.FreeBSD.org for snapshot downloads from FreeBSD.
- Remove duplicate NM from make.nxb.conf.
Changes:
- jail -d: If stdin is a TTY, confirm before removing jail.
- Disable rexec/jexecd for now as it has issues on <head and seems racy on
head as well, possibly leading to "Unable to execute id(1) in jail."
- QEMU: Use host pkg-static in the jail for repo to speed it up. For
this PKG_REPO_FROM_HOST=yes may be needed on mips.
This brings in almost a year of updates that had been held back.
Changes: https://github.com/freebsd/poudriere/wiki/release_notes_32
- New sub-command 'image' which is still in development, for generating
install images.
- Some poudriered fixes, but it is still experimental.
- DISTFILES_CACHES may be 'no' to use distfiles/ from ports tree.
- A cache daemon has been added in, off by default, for experimentation.
- The ports 'environment' feature from Mk/Scripts/functions.sh:export_ports_env
is used for dependency gathering now, which avoids 8 or so fork/execs for
every make -V call.
jail:
- New -K [KERNCONF] flag to build and install a kernel into the jail.
- New -S <path> flag to specify what srcpath to use from the master jail,
rather than /usr/src.
- Git support.
- New -U flag to specify what url to fetch from for git and svn.
- EXTRA_DISTS added to add to the distribution list fetched/extracted, from
poudriere.conf.
- NO_LIB32 added, when set to 'yes' skips installing lib32 distribution.
ports:
- Support '-m none' to disable portsnap usage.
- New -U flag to specify what url to fetch from for git and svn.
Build process handling:
- A jexecd process is spawned in the jail and communicated with via rexec
using a pipe. This is intended to lessen the jail locking contention,
allow more easily cleaning up processes, and limiting of TTY. It is
still considered experimental.
Changes:
- Fix 'builder stop' hook, which never worked.
- Don't transcend mount directories when running rm -rf
- Stop exporting MY_JOBID
- Run 'builder stop' hook always in stop_builder() [this means the hook
runs at startup as well to clean out stale builders].
- Enable post-mount jail hooks.
Unless something major is wrong here, this will be 3.1.10 and the next update
to -devel *will be* master which has a year of pending updates.
Changes:
-Always call make by absolute path of /usr/bin/make.
-Update stale comment about MFS
-Fun things happen with /nonexistent exists. Ensure it doesn't.
-testport: Allow -o to be optional.
-Mark poudriere-queue experimental.
-Using ccache on 10+ is fine so long as we use a PATH.
-XDEV: Don't endlessly append to make.conf on updates.
-Fix leftover lock file in /tmp at cleanup
-Support in-tree ccache for buildworld on recent head
-Always use the FAST_DEPEND option for buildworld.
-Go crazy, use -j for installworld targets on recent head
-Allow comments on the same line with port entries in blacklist
Changes:
- bulk -c: Restore previous behavior with log deletion. Don't wipe out ALL
logs, only delete the ones for the given build. This change
almost inadvertently wiped out all logs and violates POLA at
this point.
- bulk -c: Fix it, it wouldn't start.
- Misc cleanups
Changes:
- jail: check the directory for method src=
- csup has been deorbited from the project
- Test to make sure $SVN_CMD is actually set before trying to use it.
Improve test and error message for detecting if svn is installed.
- jail -x: Run native-xtools from /usr/src for now.
- Remove EOL 8.3 support
- Run 'jail start' hook before any 'make' commands.
- jail -m src: Fix handling of cross-compile by setting proper TARGET/TARGET_ARCH
- Remove use of SVN mirrors.
- Fix need_emulation to only check TARGET_ARCH against kern.supported_archs.
- Let KEEP_OLD_PACKAGES work on 9.x/8.4 with sort(1) lacking -V [1]
- Fix typo in poudriere.8
- Correct typo in poudriere -p help output
PR: 193732 [1]
I had intended to update this to master but have not yet fixed a blocking
bug on it.
Changes:
- status -a: Fix handling of showing older builds.
- QEMU: Add a hack to raise the default build times.
- Don't crash starting up a job, or otherwise, if the time goes backwards.
- blacklist: Fix loading of JAIL-SETNAME-blacklist
- QEMU: Always try loading a qemu-blacklist file
- Add a helper 'clock_monotonic' to use clock_gettime(CLOCK_MONOTONIC)
rather than non-monotonic 'date +%s'
- Use monotonic time for calculating durations
- Add a build.started epoch time.
- Add a build.ended epoch time that is set in the last HTML json generation.
- Update sh to FreeBSD r281982
- Fix build on older FreeBSD due to sh. If F_DUPFD_CLOEXEC is unavailable
then use system sh.
- Ignore /var/db/freebsd-update/ too.
Changes: https://github.com/freebsd/poudriere/wiki/release_notes_317
- Restore /usr/sbin and /sbin to PATH lost in 3.1.5
- HTML: Fix broken log link from 3.1.5
- Avoid copying /var/db/freebsd-update into build jails
- QEMU: Fix handling of elftoolchain install for older src checkouts.
Changes:
- Fix testport -o for invalid origins.
- HTML open tag fixes
- Fix temporary file creation causing bad 'mv: fastcopy' warnings about
JSON files.
Changes:
Fix not setting proper UNAME_m/p for powerpc64 cross build
Fix qemu builds not setting MACHINE/MARCHINE_ARCH in make.conf
QEMU: No need to set MACHINE/MACHINE_ARCH in env.
QEMU: For consistency set UNAME_m/p in jail's login.conf.
Don't let balance_pool() failure crash a builder.
Avoid double ln(1) when called in a clean_pool/stop_build failure from build_pkg
When a build crashes and does not complete, properly cleanup its builder jail.
bulk queue optimizations
Call pkg_cache_data in a separate worker. This allows build_pkg() to finish quicker.
bulk -t/testport: Fix the first package built in a jail not showing its needed shared libraries.
Disable 'set -x' in some noisy functions
Rewrite lock_acquire() to not spam 'sleep 0.1'
SIGINFO: Disable trap when erroring, exiting or in other traps
SIGINFO: Don't allow running twice, and ignore errors.
SIGINFO: Hide html_json's sleep(1) SIGINFO output
Allow TMPFS_DATA to be used for builders too.
Always protect(1) poudriere and its coprocesses from OOM killer.
Create temporary files in [MASTER]MNT/.p/tmp when possible rather than /tmp
Don't ignore SIGINT in bg processes. This should fix cpdup spam on ^C during cloning.
Fix some interleaved messages during parallel processing.
Rework stderr prefixer for parallel jobs.
Don't show 2 errors when failing to get PKGNAME.
Import and prefer /bin/sh from FreeBSD r281718. This simplifies compatibility and allows job control.
Changes:
- jail -cx: Fix incorrect hard linking.
- Support building powerpc from powerpc64 without qemu.
- Document that '.' is not allowed in -jzp
- jail_start: Enforce no '.' in -jzp.
