Over the past several months portscout.freebsd.org appears to have been
more frequently NOT finding updates, particularly for ports that use
CHEESESHOP (PyPI) as their MASTER_SITES.
Portscout has also never worked for ports using GitHub for distribution
files due to the following:
a) Portscout, prior to 'guessing', requests a randomly named file
from the Site and expects a 4xx (404) in response. If it doesn't
receive a 4xx response, it increments a 'lie counter' and does not
check the site again in the next run.
b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk
construct a URL that ends in a a dummy query paramater (for the
filename), so that fetch saves the correct filename to DISTDIR.
This means for any DISTFILE name provided, a 200 OK response is
returned
These two factors unfortunately leave us in a position where there is no
good way to workaround this in the ports framework, including overriding
DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various
reasons (not matching distinfo, file conflicts in DISTDIR, etc)
Fortunately, the portroach project (OpenBSD's fork of portscout)
contains a site handler for GitHub and PyPI (among others) already [1].
These site handlers use API endpoints at GitHub and PyPI that respond
JSON respectively, providing a faster and more accurate way to determine
the latest version of a package, without having to go through the
'guessing' process.
This commit:
- Adds GitHub and PyPI site handlers, and modifies or extends them to
accept/match our MASTER_SITES URL's.
- Adds authenticated API request support and two settings for the
GitHub site handler
- Add p5-JSON to RUN_DEPENDS (needed by new site handlers)
- Add HTTPS option for supporting https:// MASTER_SITES. Currently
portscout does not check (fails) https:// MASTER_SITES [2]
- Take MAINTAINER'ship
- Adds badly needed logging/debugging messages to key parts of the
process retaining the conditional logic that ties the verbosity to
"quiet" or "debug" portscout.conf settings.
- Renables the SQLITE3 option (previously commented out) and renames it
to SQLITE (the standard, as per bsd.options.desk.mk)
- Creates a DATABASE option group allowing either/or SQLITE or
POSTGRESQL to be selected
- Switches option conditionals where possible to options helpers
- Backport a fix for maintainer matching/mapping [3]
- Adds LICENSE (BSD2CLAUSE)
- Updates and sorts pkg-plist
This change was tested again ports maintained by me, and resulted in
'new versions' being found and reported for 42 of my ports (of 123).
[1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler
[2] Reported by: truckman
[3] 2f6ee134dd
PR: 203996
Approved by: maintainer (timeout, 15 days)
Differential Revision: https://reviews.freebsd.org/D3996
bf483b8f10