- Fix file descriptor leak during rc script execution
- Fix setting attributes on directories
- Fix locking on NFS
- Allow pkg annodate -S to run as a non priviledge user
- Extend the audit periodic script to report about expiration/deprecation
Change in hook behavior:
Now the initial building of pkg(8) triggers a success or failure hook
after building. Before ports-mgmt/pkg was the only port that could
never trigger these hooks. The "bulk run start" hook is triggered
afterwords as pkg(8) is a prerequisite to preparing the bulk run.
Fix potential bad unmount of /usr/src
In the case where /usr/src exists, but the profiles uses a system root
that is not "/" and doesn't have an "usr/src" subdirectory, an error
would be emitted as the unmount of the non-existant mount failed. This
uncommon use case has been fixed.
Curses display builder "Elapsed" label changed to "Duration"
This matches the format of the web-based report
DragonFly only: Support File(1) v2.0 version parsing
The output of file(1) version on DragonFly will change with version 2.0.
For example, what is now version 4.0.702 will be displayed as 4.7.2 on
newer versions of file(1). Moreover, the parsing for DragonFly 4.10+
would not have worked. Now both versions are supports and the double-
digit minor versions are properly handled as well.
At the request of a user, further review reveal that the pkg_* tools
aren't actually used. It just awks the INDEX file. The maintainer
forgot the technical details, especially since the SEE ALSO section of
the man page references pkg_* tools.
A minor update is needed to re-support DragonFly and update the man
page. Until this, it's been marked IGNORE for DragonFly.
- Fix fd leak on systems without utimensat (merged in release branch)
- Do not use openssl for sha256
- Improve the default output when fetching data
- Update libfetch to the version of FreeBSD 11
- Update libsbuf to the version of FreeBSD 11
- Fix NetBSD ABI
- Add a fallback for utimensat when reporting EOPNOTSUPP
Fix web report regression caused by cleaning of history files.
The history files were being removed too late. Any ignored/skipped
files erased after writing (but would return with first built/failed
port). If more than 500 ports were ignored+skipped, the web report
would fail to show the history completely and would not recover. The
old history files are now removed before the ignore/skip cascades.
Cosmetic: curses display
Capitalize "Swap" and "Skipped" field labels
Cosmetic: web report
After the build completes, fade out the builders table in 2.5 seconds.
Nobody needs to see a table showing all the builders in "shutdown" mode.
Bandwidth: json files
A history.json file holds up to 500 log entries. 16 bytes of spaces were
removed which reduces the history file size by 8000 bytes. Spaces were
also removed from summary.json which can account for 20%+ of the size.
Only rewrite history files upon change:
The last history.json file was being written/rewritten every 3 seconds
regardless if there were changes to it or not. Now it's only written
if it's changed since the previous write. This might help reduce errors
seen by in nginx access log which are thought to occur when file changes
as the file is being served.
Address "[N] Fetch prebuilt packages" bug:
Once the /usr/local/etc/pkg/repos/00_synth.conf file was created, there
was a good change that fetching prebuilt packages would stop with a
fetch error that the checksum didn't match. This seems to be caused by
pkg(8) using both the FreeBSD and the Synth repositories and often
deferring to the latter. This is incorrect behavior as only the FreeBSD
repository should be used for prefetching. This issue is hopefully
resolved now.
Web report behavior change:
Any existing ??_history.json files are now deleted at the start of
each run. They were only litter as the javascript did not pull in old
files, so this change just removes them.
While here, rename the "Elapsed" column on the builders table to
"Duration" to match the history table lable and also adjust the widths
of the origin and information columns on the history table.
- Drop privileges in many commands
- Drop privileges when fetching a file
- Add resource limitation in sandboxes
- Add support for METALOG
- pkg delete new prints a warning for pattens that matches nothing?
Major new feature: Web-based front-end
A dynamic html report is now generated for each build. The report is
placed at <profile log directory>/Report/index.html.
The report is updated 10 times per minute. The entire build history
is retained and is searchable, but that history is reset at the start
of the next run.
In addition, several fields are clickable and trigger a quick-filter
on the history log including the "Built", "Failed", "Ignored",
"Skipped", "No." column and "ID" column (see tiptool for action hint)
and the "Total" field clears the search pattern quickly.
An earlier but representative version of the web report can be seen at:
https://github.com/jrmarino/synth#web-interface
The Synth web report pulls 2 orders of magnitude less data than the
equivalent poudriere report because the json data is split into
multiple files limited to 500 ports each (e.g. the poudriere data file
might be 1.8 Mb on a big bulk run polled every 8 seconds where as
Synth only pulls the latest segment file (50kb avg) every 6 seconds.
Synth also stops polling when the run is complete.
Minor enhancement: The log files were cleaned up
The data/phase headers and footers were modified so that the build
log appears a lot cleaner now (subjective of course).
from ${OSVERSION}. This should work around a bug in portsnap whereby the
"Fetching N metadata files" step will fail if the identical INDEX files
are generated for different major FreeBSD versions.
It turns out that the C source files were getting built with the headers
from the base ncurses. The recent update in Synth's display exposed a
resulting problem with the add chtype routines, resulting in garbled
output. Since adacurses is a static library, synth requires a revbump.
Handles remaining resizing exceptions and improves display handling.
Yesterday's work handled most of the common display exceptions, but others
were still possible. Now all possible exceptions are handled.
Several improvements were made to the display:
1) lines no longer wrap if the size width is resized too narrow; they
get truncated as always intended
2) Elements such as the elapse timer don't get displayed in the wrong
place when the screen is too narrow (they just don't show)
3) The dashes now get restored if the screen is sized small and then
big again (or started small and then expanded). In many cases those
lines just never came back before.
4) The "full" refresh frequency was increased a period of 30 seconds to
a period of 4 seconds. This has a side benefit to text-mode watchdog
as well since that's the same timer for the log inspection.
5) The history window height ranges from 10 to 50 rows. If the xterm
window starts small, the history will be 10 lines. If it starts
big, the number of lines will be dictated by the original size of
the xterm window. Making the screen small and then bigger again will
reveal the full number of log lines.
Major bug fix: ncurses display resize hang fixed
Until now, resizing the window why synth is running in ncurses mode
caused synth to hang (it would finish the builds it was working on
but the display wouldn't update and no new jobs would start). This
was due to an unhandled exception thown by ncurses binding as a result
of the resize event, and now these are handled.
Minor fix: Ports with @info in pkg-plist now pass in test mode
The mtree exclusion file was improved to allow these leftover info
directories to be ignored (as is done in poudriere. Before only
info/dir was ignored, but the presence of "dir" prevented "info" from
being removed by pkg(8) upon deinstallation.
enhancement: Augment text mode (requested)
Now when a builder starts on a new package, the port origin will be
shown in the running log (before only the completion was logged.)
Fix regression in text-mode caused by activation of watchdog.
The watchdog is checking the lengths of the build logs to figure out if
a builder has stalled. It turns out that the logs were only being
inspected in ncurses display mode, so any port that took longer than
20 minutes to build would be aborted by the watchdog.
While here, bump the *BASE* time limit for the build phase from 20 to
25 minutes based on extreme causes (normally involving gcc or tex ports)
and also bump the check-plist phase limit from 3 minutes to 10 minutes.
Some ports have tens of thousands of files in them which takes a long
time to check under test mode, especially if the server is loaded.
- If a port has another upstream, remove GOOGLE_CODE
- If a port only has GOOGLE_CODE mark it BROKEN
Some ports have a local mirror configured but for security reasons, it
is not considered upstream.
Sponsored by: Absolight
This release changes the watchdog time limits from a fixed value to
one that is affected by the average load. Using fixed values didn't
work well on extremely (and intentionally) loaded servers, e.g. an
overage load of 200 on an server with ncpu of 32.
The base times which were long to account for loading have been reduced,
and a multiplier of average-5 load / ncpu rounded to the nearest tenth
has been implemented (with 1.0 as the floor value). For example, a 4-core
machine with an average 5-minute load of 6.0 will multiply the base
timeout by 1.5. This approach worked well in an extreme test against the
entire ports tree.
This release improves robustness and activates the watchdog.
It leveraes the procctl functionality to ensure all processes spawned
from a builder are reaped, which in turn ensures that tmpfs mounts can
be dismounted. Previously stuck processes could prevent those dismounts,
trapping them as new mounts get placed on top.
This also finally enables the watchdog that will kill runaway builds.
The watchdog has a specific time limit per build phase where it will
kill the build if the log doesn't grow over the previous X minutes.
No activity timeout limits per phase are:
check_sanity : 1 minute
pkg_depends : 3 minutes
fetch : 480 minutes
checksum : 480 minutes (fetches if required)
extract_depends : 3 minutes
extract : 30 minutes
patch_depends : 3 minutes
patch : 3 minutes
build_depends : 5 minutes
build : 20 minutes
run_depends : 10 minutes
stage : 20 minutes
check_plist : 3 minutes
pkg_package : 120 minutes
install_mtree : 3 minutes
install : 10 minutes
deinstall : 10 minutes
A minor change regarding the swap display: If there is no swap installed,
it will now display "n/a" instead of "100%"
This is a minor bug fix version. Changes include:
* Set close-on-exec operation mode on popen. This is required to stop
leaking file descriptors in highly concurrent modes (e.g. 32 builders).
It's not supported on FreeBSD 9 or DragonFly 4.4 (and earlier) so this
modification is removed for those platforms.
* Remove procfs mount for lang/rust and lang/rust-nightly. This was
necessary for DragonFly, but rust has been fixed for DF 4.6 and later
* Add a stage-QA exception for entries left /var/spool/*. The presence
of these are not an identication of a port issue
* Sanitize synthexec to verify file descriptors 0 .. 2 are in use and
automatically close any file descriptor > 2 before execv fork.
Plasma5 ports
At the moment KDE ports use bsd.kde4.mk to handle their dependencies. When
working on the ports for KDE Frameworks and Plasma5 it seemed to be more
reasonable to create a new kde.mk instead of adding an bsd.kde5.mk.
The kde.mk in this review is a stripped down version of the one we are using in
the KDE Test repositories plasma5 branch [1] to only contain the parts relevant
to the current KDE4 ports in the portstree [2].
Changes to the KDE Ports needed by this:
Replace USE_KDE4 by USE_KDE [3]
Add USES=kde:4 [4]
[1] http://src.mouf.net/area51/view/branches/plasma5/KDE/Mk/Uses/kde.mk
[2] The version in the plasma5 branch also handles frameworks/plasma5 and
handles MASTER_SITES via a KDE_DIST variable similar to bsd.qt.mk for Qt
Ports -- I chose to leave this out for now, as the diff is already large
enough.
[3] I chose USE_KDE instead of USE_KDE4, USE_KDE5, USE_KDEX as the version we
want is already specified as argument to kde:<arg>
[4] For KDE Frameworks and Plasma5 ports this would be kde:5
PR: 210667
Approved by: portmgr, mat (mentor), rakuco (mentor)
Reviewed by: mat, rakuco
Differential Revision: https://reviews.freebsd.org/D6961
- sandboxes always drops privileges if run as root and chroot in /var/empty
- sandboxes limits resource usage to make them less useless on system without capsicum
- drops privileges and chroot to /var/empty in various commands if run as root:
* pkg audit
* pkg info
* pkg (r)query
- While here, add NO_ARCH and turn ECHO_MSG's into pkg-message to be visible to package users as well
PR: 211956
Submitted by: sakaue.mamoru@mwghennndo.com (maintainer)
The master port was refactored to make tracking github repositories easier, but
this change was not reflected in the last update to ports-mgmt/portshaker.
During upgrades/reinstall javavmwrapper 2.5 pre deinstall script deletes the pkg
temporary files preventing to finish the upgrade.
A fixed version of javavmwrapper will be made soon, this hack will remain in the
ports tree and not in pkg(8) itself. This hack is made to not bother users is
only affecting upgrade/reinstall phase not proper deinstall
PR: 210313
the variable $unique_list was created with identical content, its value was
only set within the implicit sub-shell (2nd command in a pipe) and lost on
exit from the while loop.
Reported by: Jakub Laach
At one point during development of last changeset, the default.gpr
file needed to have a LOCALBASE correction, but that's no longer true.
This line shouldn't have made it into the previous commit.
This is a bug fix version. Changes include:
* Fix potential exception seen with a large number of builders. The
height of the log area was determined by subtracting the height of
the other two zones from the overall xterm height. This allowed for
the possibility of a negative height for the log area resulting in a
thrown exception. Now the log is defined to be at least 10 lines
log, but ncurses will just truncate as necessary.
* Detection of bad OPSYS value added. There's a rare case where pkg
can corrupt the configuration value of the Operating System. If it
happens on FreeBSD, Synth thinks it is running on DragonFly which
causes all sorts of problems. The corruption can't be auto-fixed,
but Synth will gracefully abort with instructions on how to correct
the issue.
* Fix filtering of port-mgmt/pkg in list of ports. Synth was designed
to always skip building pkg and thus remove it from a given list of
ports, but the filtering didn't work correctly resulting in an
exception with a misleading message.
* Redirect failed jail setup command output to a new log:
/var/log/synth/05_abnormal_cmd.out. Under normal operation, the
log will be empty, but any abnormal output will now be sent there
rather than corrupt the ncurses display as it before now.
* Fix mount cleanup procedure. The output of "df -h" was being
sorted alphabetically which caused mounts-on-mounts (an exception
condition that dillon has been getting with H2 testing) to fail
during dismount. The df output is already in the correct order so
it only needs to be iterated in reverse without the pre-sort.
Change:
* Synth now links curses statically so it should not pull in any
additional packages when the Synth binary package is installed.
The only known bug is the curses display will freeze if running in an
xterm window that gets resized. The cause is still unknown because in
theory Synth should correctly react to a window-resize event. Some
people may have success with ^z followed by fg command, but it's best
to avoid resizing the window at all.
everything at once. Sometime, rename post-install into a options helper
target.
I did not fix ports that were such a mess that I could not figure out
what they really wanted to do. I also did not change ports that had
some version of an auto-plist code in post-install, for the same reason.
With hat: portmgr
Sponsored by: Absolight
indefinite amounts of time, preventing installation or upgrade of
ports with portmaster. Instead of using a pipe to read pkg query
input to be consumed by a shell loop, buffer the query result in a
shell variable and let the loop iterate over the contents of this
variable.
While here, use a call to all_pkgs_by_origin() instead of an inline
versin of equivalent code.
Reviewed by: tz
Approved by: tz
- New maintainer's email
- Enable automatic workarounds of looped dependencies
- Fixe a problem that had emerged when pkg(8) was updated
PR: 210955
Submitted by: maintainer
Makefile:
Bump `PORT_REVISION` for the change.
portfind.c:
get_release(..):
The function was incorrectly modifying a pointer that wasn't the original
calloc'ed pointer, tripping asserts when MALLOC_PRODUCTION wasn't enabled
in jemalloc [*].
- Use one temporary buffer (`release`) instead of two (`release` and
`version`).
- Improve temporary memory idiom for managing memory used with
sysctlbyname(3) by first checking the length, mallocing the buffer,
then filling it with a second call to sysctlbyname(3).
- Use strchr(3) instead of handrolling it in a while-loop and to
avoid the improper free(3) of the memory allocated for `release`.
main(..):
- Use asprintf instead of calloc + sprintf.
- Use constant `pasting` with `INDEX_FILE` instead of passing it in to
asprintf(3).
- Fix error message when unable to open `INDEX_FILE`.
Approved by: brd
Differential Revision: https://reviews.freebsd.org/D7198
PR: 211032 [*]
Reported by: Michael Zhilin <mizhka@gmail.com>
Reviewed by: Michael Zhilin <mizhka@gmail.com>
Sponsored by: EMC / Isilon Storage Division
- Added support dialog.ascii_lines(D4PASCIILINES), thanks to kuriyama
- Now first line can be ITEM_SEPARATOR, reported by: Rotluchs(EFnet)
- Rewrote Makefile, bsd.prog.mk no longer needed, with help from amdmi3
- Fix help message, thanks to wblock
PR: 211035
Submitted by: rum1cro@yandex.ru (maintainer)
- update internal expat to 2.2.0
- fix external solver
- fix build on freebsd pre 9.2
- fix warnings on armv6
- fix solver issue resulting in missing conflicts or reinstalling unneeded
packages
- add a mini summary at the end of the output
- Update internal expat to 2.2.0
- Fix using an external solver
- Fix build on FreeBSD prior to 9.2
- Fix issues on armv6
- Add a mini summary of the operations (removal, installation, upgrades) that
that will be done
- Fix solver issues that could trigger issues with conflicts or reinstall too
many things
- Important fixes in the solver
- When conflicts are found, they are explicitly printed
- Reduce namecache contention a bit
- Temporary files created during extraction are now hidden
- Accept symlinks to directories when pkg is expecting a directory
- Accept symlinks to directory when the package expects a directory
- Cleanup symlink checksuming code
- Fix a bug that results in bus error on very specific cases with lots of
packages installed
- Lots of important fixes in the solver
- Fix pkg upgrade -f
- Reduce namecache contention a bit
- Prefix all temporary files with a . so they are hidden
- Explicitly tell the users which conflicts were found
- Do not fork pkg when being ptraced (only on FreeBSD)
- Implement repo pinning so once installed a package will keep coming from a
given repo
- Fix repository priority handling on directly requested packages
Big thanks az@ and vsevolod@ for the thank less help provided
This release adds detection and handling of circular dependencies.
Two users that tried replacing OpenSSL with LibreSSL found out that
ftp/curl depends on itself when its built with GNUTLS. That's because
Synth goes into an infinity dependency loop, runs out of stack space,
and crashes.
Now Synth can detect a circular dependency and abort gracefully and
informatively, e.g.
> synth status ftp/curl
ftp/curl scan aborted because a circular dependency on ftp/curl was detected.
... backtrace devel/cmake
... backtrace emulators/tpm-emulator
... backtrace security/trousers
... backtrace security/gnutls
... backtrace ftp/curl
While here, build synth with stack-checking on so out-of-stack situations
result in an exception and not an illegal instruction crash.
This release brings a behavioral change to Synth. The action of building
package repositories is much more conservative now; packages that would
fail sanity checks are only removed under the following commands:
* synth prepare-system
* synth upgrade-system
* synth rebuild-repository
* synth everything
That means the following commands will never result in built packages
being removed now:
* synth install
* synth build (followed by "yes" to rebuild repo question)
* synth force (followed by "yes" to rebuild repo question)
Other changes/fixes:
* The PORTSCAN.SEEK_FAILURE exception is usually a false positive and
should be fixed now
* Under test mode leftover check, all changes to /var/db between
installation and deinstallation are ignored rather than being flagged
as a failure (see databases/mysql56-server test failure)
* Add support for DragonFly's /boot/modules.local (similar to FreeBSD's
/boot/modules)
* Bring in ports patch to use installed libraries instead of the bundled
versions
Note that people are using mirror_type: http when they should not
most probably due to bad documentation on our side so we will improve that later
mirror_type: http is only useful if you actually host multiple mirror of your
repos see pkg-repository(5).
Changes:
- Fix: setting date on system not POSIX 2008 compliant (like FreeBSD 8)
- Fix: running pkg in env where the 0 file descriptor is closd
- Fix: make http mirror type respect the specified port
- Fix: chown/chmod on linux
- Fix: pkg ssh
- Improved upgrade chain in multi repo environment
- Update libucl to latest version
Changes:
- Fix: make http mirror type respect the specified port
- Fix: chown/chmod on linux
- Fix: pkg ssh
- Update libucl to latest version
- Reduce adherence on openssl
Changes:
- jail: Fix usage for -m src=PATH
- jail -v: Note that non-listed methods used the value only for display.
- QEMU: Copy the latest emulator into the jail at startup. This fixes
the jail running into bugs with qemu that are fixed with the latest
qemu. It was only copying it in during creation before.
- bulk: Avoid redundant cleaning message with jailversion cleanup.
- Add and document PORTBUILD_UID and CCACHE_DIR_NON_ROOT_SAFE. [1]
PR: 209786 [1]
Changes:
- Fix incorrect handling of null-mounting /nxb-bin in jail
- Disable BUILD_AS_NON_ROOT if CCACHE_DIR is set [1]
- Fix mount path too long when using linprocfs
PR: 209786 [1]
longer. This is a no-op because KDE4_PREFIX is equal to LOCALBASE
Fix up properties for misc/kde4-l10n/files/bsd.l10n.mk to make svn happy.
PR: 209014 (partial)
Submitted by: myself
Approved by: portmgr (bapt)
Differential Revision: https://reviews.freebsd.org/D6542
Changes:
- improvements in the solver regarding multirepo
- fix setting time on pre POSIX 2008 systems (such as freebsd 8)
- fix running pkg when FD 0 has been closed
Changes:
- Reduce memory used while analysing elf files
- Stop checking pre-deinstall script except if DEVELOPER=yes is set
- fix double free in pkg updating
Due to too many bad quality ports, for now we are back in not checking pre/post
deinstall scripts at all except if DEVELOPER_MODE=yes is set, we hope to readd
those checks once the ports have been fixed
New features:
- new vital flag that prevent from removing a package with pkg delete when not specifying -f
- upgrade process now has atomicity per packages meaning now we can safely check the return of the pre* scripts
- handle ctrl-c during upgrade/install/update
- solver improvements
- pkg clean now runs in a capsicum sandbox on supported plateforms
Changes:
- pkg delete now fails if one tries to delete it (the pkg package) without -f. Before it was just warning and skipping it
- pkg install --from-root has been renamed pkg install --recursive to avoir confusion
Fixes:
- correctly handle hardlinked setuid files
- better handling of setuid
- fix elf analysis when some bad libraries have SONAME specified by empty
- fix elf analysis when a binary provides elf_notes but not the expected ones and are still valid freebsd binaries
(this fixes the case when libgdk_pixbuf was not found as a provided lib on i386)
- NAMESERVER option is now working again and inputs are better checked
- lots of bug fixes: potential leaks mostly
Changes:
- jail/ports -u: Default git pull to use --rebase.
- destroyfs: Run chflags -R 0 and on a smaller subset of files.
- ports: Ignore failures when removing the ports tree
- jail: Ignore errors trying to remove the jail
- Create port-specific directories in do_portbuild_mounts.
- Rework the cleanup to respect -x inside of the directory.
- Support -(j|p|z)<param> without space for poudriere.conf lookups
- hash_var_name: Quote the pattern to _gsub to avoid globbing local files.
- Update sh from head @ r296813
- Switch BUILD_AS_NON_ROOT as default to on.
