Otherwise if you use the dbscripts for core/extra for community repos,
or vice versa, you'll get an error saying that you don't have
permissions to move packages from or to the given repositories, which
can be confusing if you don't immediately notice your mistake.
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
We tried to optimize out checking packages which aren't even split
packages, by skipping the ones with no pkgbase. But pacman 5.1 always
writes the pkgbase in order to simplify parsing of the .PKGINFO, so this
was no longer reliable. Instead, check how many pkgnames are in the
corresponding PKGBUILD. This is the master source of truth anyway.
Use the --fast-read option to quit as soon as the file is first found. A
package file should never contain two of these anyway, but even if it
did, we simply want to know that one exists!
makepkg guarantees its internal .METADATA files are tar'ed up first, so
this should usually be quite rapidly found. For very large packages,
crawling the entire package file to check for a later entry overriding
the first one, is... pointless... and worst of all, slow.
This reflects the identical mode used for extracting the pkginfo file,
which was inconsistently implemented from the initial feature addition.
I've cleaned older packages from the archive, but sometimes we do not
rebuild packages in a long long time. We still keep have them in the
repository, but this check does not look for the package there, thus
when trying to db-update, the user sees an error. We fix this by also
looking at currently live packages instead of only relying on the
archive.
This is mostly a hotfix until a better solution is created. Depending on
when/how the ftpdir-cleanup cronjob removes such packages, users may
still see errors when an old package is updated and the cronjob removes
it from the repository before db-update is run.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
A semi-common pattern is for one maintainer to stage a rebuild of a
package due to e.g. cascading repository-wide python/boost/whatever
rebuilds, and then for the original maintainer of the package to not
notice and update the package in the stable repo, leaving an out of date
rebuild in staging or testing.
Then the the out of date package gets moved and ends up breaking things,
possibly via a package downgrade, possibly via breaking compatibility
with a much more targeted rebuild uploaded all at once. Ultimately,
Things Happen™ and the repository hierarchy gets broken.
Prevent this by enforcing for all packages that exist in
multiple levels of the repo: staging -> testing -> stable
That updates to one must come with an update to all the others.
Whenever adding new package files to the pool of distributed packages,
copy the file into a longterm archive. This is the first step to merging
the functionality of archivetools, as this implements the shared pool
while also guaranteeing that all packages are archived at the time of
entry rather than once per day if they still exist.
Introduce "db-functions-$VCS" which will eventually contain all
VCS-specific code, and make this configurable in config.
Move private arch_svn function and svn acl handling here, and introduce
a new source_pkgbuild function to handle discovering PKGBUILDs from the
configured VCS and sourcing them to extract metadata.
The PKGBUILD is the only file we ever check out from version control,
and only ever to scrape information from it, except for when we actually
want to db-move a whole directory (which is by necessity considerably
dependent on the VCS in use).
source_pkgbuild is inspired by commits from the dbscripts rewrite,
authored by Florian Pritz <bluewind@xinu.at>
This silences a useless error message that confuses the user.
bsdtar doesn't like it when the stream gets closed before it finishes
which may be the case when grep found its match on potentially huge
archives. Instead of suppressing the whole stderr , we find all matches
with grep, then use a second pass with `tail` to find only the last
match, which ensures the stream remains open for bsdtar but we may still
catch and see useful messages on stderr.
This works because tail has the useful property of not closing early.
Search for unquoted variables using the command:
grep -Prn --exclude-dir=.git '(?<!["=]|\[\[ |\[\[ -[zn] )\$(?!{?#|\(|\? )'
and ignore a bunch of false positives.
You may verify that the only differences between the lines removed and
lines added are double-quotes:
diff -u \
<(git show|sed -n 's/^-//p'|grep -v '^-- a/'|sed 's/"//g') \
<(git show|sed -n 's/^+//p'|grep -v '^++ b/'|sed 's/"//g')
- db-functions: getpkgfile: The .sig file extension should be part of
the filename parameter, not part of the message format string.
- db-functions: arch_repo_modify: Shouldn't use ${action}
string-interpolation in the message format string. Since the
entire message is a command, and we're using @Q to escape arguments
anyway, go ahead and just construct the entire command as a single
string that way, and feed it to '%s'.
Using the following command to find unquoted variables (and ignoring
more than a a few false positives),
grep -Prn --exclude-dir=.git '(?<!["=]|\[\[ |\[\[ -[zn] )\$(?!{?#|\(|\? )'
one is lead to find a few cleanups that are something other than "add
double-quotes". That's what these are. We'll leave dumb adding of
double-quotes for another commit.
Most of these are still fixing quoting issues, just with a better fix.
- parse_pkgbuilds.sh: Avoid having to escape quotes in `eval` strings
by using `declare -p`. Updates the logic copied from makepkg, with the
latest logic copied from makepkg. See
https://git.archlinux.org/pacman.git/commit/?id=9e52a36794552b77ecf26f7f34b226d096978f1e
- sourceballs: Avoid using ary=($string) to do field separation by
using `read` and test that multiple licenses actually work as
expected.
- sourceballs: Replace `[[ -z ${ary[*]} ]]` with test for the array
length
- db-functions: Replace mangling echo field separators using sed, with
printf formatters
- db-functions: Replace for/echo loop to print an array line by line,
with `printf '%s\n'`
- db-functions: set_repo_permissions: Line up error messages, quote
"$group"
- db-move: Replace `$(echo ${array[@]})` with `${array[*]}`
- testing2x: Use `"$@"` instead of `$*` when looping over an array
Also, not really quoting related but on the same line as a quoting
issue, optimize:
- db-functions: Replace
[[ -n "$(... | sort | uniq -D)" ]]
with
! ... | awk 'a[$0]++{exit 1}'
(By default, prefer `grep &>/dev/null`)
`grep -q` may exit as soon as it finds a match; this is a good optimization
for when the input is a file. However, if the input is the output of
another program, then that other program will receive SIGPIPE, and further
writes will fail. When this happens, it might (bsdtar does) print a
message about a "write error" to stderr. Which is going to confuse and
alarm the user.
In one of the cases (in common.bash, in the test suite), this had
already been mitigated by wrapping bsdtar in "echo "$(bsdtar ...)", as
Bash builtin echo doesn't complain if it gets SIGPIPE. However, that
means we're storing the entire output of bsdtar in memory, which is
silly. Additionally, the way it was implemented is also wrong;
because it was being used with `grep -qv` instead of just `grep -q`,
it *always* found a non-matching line (even something inconsequential
like `%NAME%`), and *never* triggered a test failure.
Looking at a few of these cases, it might also make sense to switch to
using `bsdtar tf` instead of `bsdtar xf` when checking membership, but
that's work for another day.
It is a method of notifying text-editors that a file is in Bash syntax
without giving it a propper shebang (which would be confusing, as it
would suggest that the file should be executable), as well as working
across virtually all text-editors (unlike "-*- Mode: Bash -*-" or
whatever).
These are things that were (IMO) missed in 5afac1e. I found them using:
git grep -E '(plain|msg|msg2|warning|error|die) "[^"]*\$'
I went a little above-and-beyond for escaping strings for the error
messages in db-functions' arch_repo_add and arch_repo_remove. The
code should explain itself, but I wanted to point it out, as it's more than
the usual "slap %s in there, and move the ${...} to the right".
Use less duplicated logic, opening the way to more elegantly change
the interface for the repo-add/repo-remove scripts which are themselves
symlinked together and mostly have the same interface.
The current glob `*.pkg.tar.?z` is both less restrictive and more
restrictive than makepkg, as it accepts any valid unicode character.
To be more exact, it's almost completely orthogonal to the one in makepkg.
makepkg only accepts .tar.gz, .tar.bz2, .tar.xz, .tar.lzo, .tar.lrz, and
.tar.Z and most of those fail to match against a two-char compression type.
dbscripts accepts .pkg.tar.💩z which incidentally is what I think of
cherry-picking xz and gz as supported methods.
Since this can be anything makepkg.conf accepts, it needs to be able to
match all that, unless we decide to perform additional restrictions in
which case we should still explicitly list each allowed extension. Using
bash extended globbing allows us to do this relatively painlessly.
Document the fact that this has *always* been some sort of glob, and
update the two cases where this was (not!) being evaluated by bash
[[ ... ]], to use a not-elegant-at-all proxy function is_globfile() to
evaluate globs *before* testing if they exist.
Don't bother emitting errors. bash doesn't show globbing errors if it
cannot read a directory to try globbing there. And the former code never
aborted on errors anyway, as without `set -o pipefail` the sort command
swallowed the return code.
PKGEXT is a makepkg variable referring to a fixed filename suffix, but
we were also using it to mean a bash glob referring to candidate
filenames. This is wrong, so rename it to PKGEXTS which is more
descriptive of its purpose.
Exclude the testsuite from this change, as the testsuite actually uses
PKGEXT for its intended purpose. Fix the testsuite to consistently use
PKGEXT, as it hardcoded the file extension in several cases, and pin
its value to .pkg.tar.xz
Fix numerous instances of POSIX `[ ... ]`, including reliance on ugly
deprecated constructs like POSIX `-a`. Since we require bash regardless,
it makes sense to take full advantage of it.
bash `[[ ... ]]` does not require quoting variables as the shell
natively recognizes them as variables rather than expanded strings.
Use shell arithmetic rather than test, when checking numerical values.
This meant only the first array element was checked in
check_repo_permission(). Although arguably this should never cause real
issues as something else would have to be broken if multiple
architectures have different permissions, we should catch this now
anyway.
libmakepkg messaging functions provide automatic access to gettext
(which we do not currently make use of) in addition to cleanly
separating data from message strings.
In order for this to work properly, pass argv correctly from die() to
libmakepkg's error()
double shame!
This will detect both, packages with missing .BUILDINFO (built in
non updated environments) and packages that are not built in a
chroot at all (indicated by non default builddir).
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
Signed-off-by: Pierre Schmitz <pierre@archlinux.de>
This allows for faster checking if an update might be needed and helps
to let reporead run when something changed.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
ACLs from the WORKDIR were used, leading to problems as these weren't
world-readable. Fix follows ftpdir-cleanup.
Signed-off-by: Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
Signed-off-by: Pierre Schmitz <pierre@archlinux.de>
Morten Linderud
Eli Schwartz
Johannes Löthberg
Florian Pritz
anthraxx
Luke Shumaker
Pierre Schmitz
Dave Reisner
Jan Alexander Steffens (heftig)