The primary reason for the creation of v1.66 is to provide support for
pkgsrc on FreeBSD, DragonFly, NetBSD and initial support for Linux and
Solaris. It represents over 100 commits and was successful. Along the
way, some improvements were made for the ports version. A (likely
incomplete) list includes:
- Improve watchdog kill message by including inactive time for trigger
- Add SYNTHPROFILE variable and value to builders' mk.conf which can be
used by custom local makefiles
- Force users to define TERM in environment if it is not present
- Ensure fetch(1) doesn't check for valid SSL certificates. Since all
downloads are hash-checked for authenticity, misconfigured SSL of a
webserver is not a good reason to break fetching.
- New 6th log which documents why existing packages are obsolete (same
as setting WHYFAIL in env.) "06_obsolete_packages.log"
- New SELFTEST environment command for test mode which forces new "test"
phase to execute (for ports with regression tests)
- Updated man page
- Stop dropping privileges for fetching, it causes more issues than it solves
- Fix segfault when ABI is defined by empty
- Fix reporting of deprecated packages
- Provide instructions on how to unset vital flages
- Fix libfetch for NetBSD
- Fix @config potential double free
- Always warn about the deprecated @dirrm/@dirrmtry
- Always warn about the deprecated @exec
- Rework pkg register to share more code with pkg install
* It makes it more robust to libarchive regressions [1]
PR: 214381 [1], 215029 [2]
Reported by: lampa@fit.vutbr.cz [1]
Exp-run: antoine [2]
Switch from perl threads to Parallel::ForkManager. With perl 5.24, threads
used in distilator produce a many random warnings hinting at the fact that the
modules used to check the links are not (and never were) thread safe.
Remove WWW. There is no public website available.
PR: 208506 (based on)
Notified by: mat
MFH: 2016Q4
- In case of errors during pkg update exit with a non 0 status
- Fix pkg register issues with some version of libarchive:
* hardlinks not being made hardlinks sometime
* issues when PREFIX is a symlink to another directory
- Use portable mechanism to find the number of CPUs which allows pkg repo to be
faster on linux
- Add progress when fetching (number of files to fetch)
- Improved messages about locked packages
- Return 1 when a user rejects the proposed plan
- When dealing with configuration files '@config' never overwrites non tracked files
- Warn everyone about deprecation of @dirrm and @exec
- Deduplicate metadata loading code
- pkg register now understands context aware messages
- Add a fallback for utimensat when reporting EOPNOTSUPP
- replace libsbuf with utstring for portability
- Overwrite some sqlite vfs functions to allow capsicumisation of pkg
- move more pkg audit details into the library
- allow to run pkg query without any privileges
- extend the audit periodic script to report deprecated packages
- Fix fd leak to rcscripts
- stop dropping privieges during fetching it causes more issues than it solves
- speed walking though [reverse]dependencies
- Update blake2b to latest version
- Add blake2s to the list of supported checksums
- Internal digest now default on blake2b on 64 bits and blake2s on 32 bits.
- Fix segfault when ABI is defined but empty
- Do not use the shared library provides/requires on non FreeBSD/Linux
- Define _NETBSD_SOURCES on NetBSD to allow libfetch to use funopen
- Document vital flags
- Fix potential double free when dealing with @config keyword
* Allow https URLs for the WWW line in pkg-descr. Why are we only NOW doing
this? [1]
* Check TIMESTAMP to make sure it's in the present/past, but not more than 30
days in the past. [2]
PR: 212091 [2]
Submitted by: amdmi3 [1]
Differential Revision: D8462 [1]
Skip new "base" category during ports scanning
The new base category was never meant to be part of the build so the
category makefile was intentionally left off so poudriere and friends
would skip it.
Unfortunately, this idea didn't account for Synth which has a different
behavior: If no category makefile exists, *ALL* subdirectories are
scanned instead of none of them. (DPorts doesn't use category Makefiles
because by definition all ports are valid; there are no "unhooked" ones)
The proper solution would be to include a practically blank
ports/head/base/Makefile to maintain consistency with the rest of
the ports categories, but synth now actively excludes the base category.
Change name of 05 log
The command output log name has changed from "05_abnormal_cmd.out" to
"05_abnormal_command_output.log". The former tended to be considered
a binary file by webservers and would download rather than display.
Final tally added to end of main 00_last_results.log file
At the end of a bulk run, the final tally (queue size, built, failed,
skipped, ignored) would be displayed in curses mode. This change adds
the same useful information to the end the main bulk run log.
- Fix file descriptor leak during rc script execution
- Fix setting attributes on directories
- Fix locking on NFS
- Allow pkg annodate -S to run as a non priviledge user
- Extend the audit periodic script to report about expiration/deprecation
Change in hook behavior:
Now the initial building of pkg(8) triggers a success or failure hook
after building. Before ports-mgmt/pkg was the only port that could
never trigger these hooks. The "bulk run start" hook is triggered
afterwords as pkg(8) is a prerequisite to preparing the bulk run.
Fix potential bad unmount of /usr/src
In the case where /usr/src exists, but the profiles uses a system root
that is not "/" and doesn't have an "usr/src" subdirectory, an error
would be emitted as the unmount of the non-existant mount failed. This
uncommon use case has been fixed.
Curses display builder "Elapsed" label changed to "Duration"
This matches the format of the web-based report
DragonFly only: Support File(1) v2.0 version parsing
The output of file(1) version on DragonFly will change with version 2.0.
For example, what is now version 4.0.702 will be displayed as 4.7.2 on
newer versions of file(1). Moreover, the parsing for DragonFly 4.10+
would not have worked. Now both versions are supports and the double-
digit minor versions are properly handled as well.
At the request of a user, further review reveal that the pkg_* tools
aren't actually used. It just awks the INDEX file. The maintainer
forgot the technical details, especially since the SEE ALSO section of
the man page references pkg_* tools.
A minor update is needed to re-support DragonFly and update the man
page. Until this, it's been marked IGNORE for DragonFly.
- Fix fd leak on systems without utimensat (merged in release branch)
- Do not use openssl for sha256
- Improve the default output when fetching data
- Update libfetch to the version of FreeBSD 11
- Update libsbuf to the version of FreeBSD 11
- Fix NetBSD ABI
- Add a fallback for utimensat when reporting EOPNOTSUPP
Fix web report regression caused by cleaning of history files.
The history files were being removed too late. Any ignored/skipped
files erased after writing (but would return with first built/failed
port). If more than 500 ports were ignored+skipped, the web report
would fail to show the history completely and would not recover. The
old history files are now removed before the ignore/skip cascades.
Cosmetic: curses display
Capitalize "Swap" and "Skipped" field labels
Cosmetic: web report
After the build completes, fade out the builders table in 2.5 seconds.
Nobody needs to see a table showing all the builders in "shutdown" mode.
Bandwidth: json files
A history.json file holds up to 500 log entries. 16 bytes of spaces were
removed which reduces the history file size by 8000 bytes. Spaces were
also removed from summary.json which can account for 20%+ of the size.
Only rewrite history files upon change:
The last history.json file was being written/rewritten every 3 seconds
regardless if there were changes to it or not. Now it's only written
if it's changed since the previous write. This might help reduce errors
seen by in nginx access log which are thought to occur when file changes
as the file is being served.
Address "[N] Fetch prebuilt packages" bug:
Once the /usr/local/etc/pkg/repos/00_synth.conf file was created, there
was a good change that fetching prebuilt packages would stop with a
fetch error that the checksum didn't match. This seems to be caused by
pkg(8) using both the FreeBSD and the Synth repositories and often
deferring to the latter. This is incorrect behavior as only the FreeBSD
repository should be used for prefetching. This issue is hopefully
resolved now.
Web report behavior change:
Any existing ??_history.json files are now deleted at the start of
each run. They were only litter as the javascript did not pull in old
files, so this change just removes them.
While here, rename the "Elapsed" column on the builders table to
"Duration" to match the history table lable and also adjust the widths
of the origin and information columns on the history table.
- Drop privileges in many commands
- Drop privileges when fetching a file
- Add resource limitation in sandboxes
- Add support for METALOG
- pkg delete new prints a warning for pattens that matches nothing?
Major new feature: Web-based front-end
A dynamic html report is now generated for each build. The report is
placed at <profile log directory>/Report/index.html.
The report is updated 10 times per minute. The entire build history
is retained and is searchable, but that history is reset at the start
of the next run.
In addition, several fields are clickable and trigger a quick-filter
on the history log including the "Built", "Failed", "Ignored",
"Skipped", "No." column and "ID" column (see tiptool for action hint)
and the "Total" field clears the search pattern quickly.
An earlier but representative version of the web report can be seen at:
https://github.com/jrmarino/synth#web-interface
The Synth web report pulls 2 orders of magnitude less data than the
equivalent poudriere report because the json data is split into
multiple files limited to 500 ports each (e.g. the poudriere data file
might be 1.8 Mb on a big bulk run polled every 8 seconds where as
Synth only pulls the latest segment file (50kb avg) every 6 seconds.
Synth also stops polling when the run is complete.
Minor enhancement: The log files were cleaned up
The data/phase headers and footers were modified so that the build
log appears a lot cleaner now (subjective of course).
from ${OSVERSION}. This should work around a bug in portsnap whereby the
"Fetching N metadata files" step will fail if the identical INDEX files
are generated for different major FreeBSD versions.
It turns out that the C source files were getting built with the headers
from the base ncurses. The recent update in Synth's display exposed a
resulting problem with the add chtype routines, resulting in garbled
output. Since adacurses is a static library, synth requires a revbump.
Handles remaining resizing exceptions and improves display handling.
Yesterday's work handled most of the common display exceptions, but others
were still possible. Now all possible exceptions are handled.
Several improvements were made to the display:
1) lines no longer wrap if the size width is resized too narrow; they
get truncated as always intended
2) Elements such as the elapse timer don't get displayed in the wrong
place when the screen is too narrow (they just don't show)
3) The dashes now get restored if the screen is sized small and then
big again (or started small and then expanded). In many cases those
lines just never came back before.
4) The "full" refresh frequency was increased a period of 30 seconds to
a period of 4 seconds. This has a side benefit to text-mode watchdog
as well since that's the same timer for the log inspection.
5) The history window height ranges from 10 to 50 rows. If the xterm
window starts small, the history will be 10 lines. If it starts
big, the number of lines will be dictated by the original size of
the xterm window. Making the screen small and then bigger again will
reveal the full number of log lines.
Major bug fix: ncurses display resize hang fixed
Until now, resizing the window why synth is running in ncurses mode
caused synth to hang (it would finish the builds it was working on
but the display wouldn't update and no new jobs would start). This
was due to an unhandled exception thown by ncurses binding as a result
of the resize event, and now these are handled.
Minor fix: Ports with @info in pkg-plist now pass in test mode
The mtree exclusion file was improved to allow these leftover info
directories to be ignored (as is done in poudriere. Before only
info/dir was ignored, but the presence of "dir" prevented "info" from
being removed by pkg(8) upon deinstallation.
enhancement: Augment text mode (requested)
Now when a builder starts on a new package, the port origin will be
shown in the running log (before only the completion was logged.)
