As each job deletes lots of rows in the table, vacuum it to free up space for
the next job.
I don't think this is something to do widely, but I wanted to try it out for
the tables which have a lot of churn.
From the normalized one, to the one actually contained within glibc. Recent
versions of glibc also contain symlinks linking the normalized codeset to the
locales with the .UTF-8 ending, but older ones do not.
Maybe handling codeset normalisation for queries would be good, but the locale
values ending in .UTF-8 are more compatible and allow the code to be
simplified. For querying, maybe there should be a locales table which handles
different representations.
Use the deduplicated list of packages when fetching lint warnings, to avoid
duplicates. This was first seen in the following Guix commit
843344273c6a587b8e6c84d8406500fd64d8908a.
In recent versions of Guix on the Guix Data Service side, I don't think these
are system-error's, but a different type. For compatibility, just catch any
error.
PostgreSQL seems to be unhappy with some data, I guess because it gets a bit
jumbled. Rather than failing the job, or getting stuck not inserting logs, try
and capture the error, log the details, and then keep going.
The derivation checker currently opens a store connection on its own, but by
passing the already open connection in, it won't have to do that, and
hopefully this will make checking all the packages faster.
Rather than just the native system. I'm not quite sure of the value here, as I
guess system tests should behave the same regardless of the way the software
is compiled, but this seems like it could be useful, and being explicit about
the system the derivation is for is good.
The switch away from catch broke this, I obviously still don't quite get how
with-exception-handler works. Therefore, use it twice as that seems to help.
I think there's still the potential for a race condition when working with the
cached checkout of the repository, but this lock was far too wide. The
sensitive section is just the latest-channel-instances call, as it adds the
relevant files to the store, and that's what's returned.
Switch to using a sequence for the ids in the log parts table, and spawn a
thread to listen for output from the inferior processes, and enter it in to
the database.
Yet again...
This makes the channel-derivations for each system accessible within the
load-new-guix-revision procedure, in preparation for storing them in the
database.
Stop using the system values as targets, and remove package derivation entries
where this is the case.
Switch the non-cross derivation case to have a target of "", rather than
matching the system, as this makes more sense, and is more consistent now that
the target values no longer match the system values.
Hardcode some more correct target values, and use these instead. Hopefully
this can be better integrated with Guix in the future.
This commit also includes a migration attempting to shrink some indexes.
In an attempt to stop the derivations from being garbage collected between the
time they're generated, and when they're potentially read by the Guix Data
Service.
Previously this would just compare on the version if the name was the same,
but there are package definitions that share the name and version (itstool is
one example).
To try and make this more stable, to avoid weird errors, and unstable
comparisons between revisions, use the line number when deduplicating
packages.
I think this might be happening when packages are filtered out as
duplicates (by name and version), but then a reference to a duplicate occurs
somewhere, like in a lint warning.
This seems to work better for both generating the non-cross and cross
derivations. Previously, using the package-transitive-supported-systems
approach didn't generate some cross derivations.
Use the second argument to package-transitive-supported-systems to correctly
identify the different bootstrap path for non x86_64 and i686-linux. The
previous implementation did work, but only up until a merge of core-updates
changed the bootstrap approach.
Previously, they were nix-base32-string encoded, but the representation in the
derivations is base16, so it doesn't make sense to use a different
representation in the database.
Therefore, add some code that runs before the start of each job to convert the
data in the database. It was easier to do this in Guile with the existing
support for working with these bytevector representations. After some
migration period, the code for converting the old hashes can be removed.
Previously, all the entries for the branch were deleted, but not, only the
entries for the branch, that are present in the revision that was loaded will
be deleted. This is more efficient, as it avoids deleting and recreating
entries in the table that shouldn't have changed.
There's already the package_versions_by_guix_revision_range table, but I think
it would be also useful to be able to see how derivations change over time.
Between b13b9384bc43bf93c754c037956c8ef9a99c2b41 and
601171a9bc7ca6e4acb932895a07c0ca9aedfdac, this method failed, so catch the
error to allow loading the affected revisions.
Ideally this would go in to the database somehow as well, but the only idea I
have for that is pass in a pipe, and then spawn a thread to read from the
other end of that pipe in a loop to send the output to the database.
That hasn't been written yet, so just send the output to stderr for now.
Add a new event 'retry', and run jobs where the number of retry events is
greater or equal to the number of failure events.
Also add an index to the git_branches table to make the finding jobs query a
bit faster.
This isn't new information, but derived from information already in the
database. It's collected here to make querying faster.
The table is updated when each new revision is entered.
Add tests around the package module, extract out the use of the
inferior-package record assessors so that they aren't part of the tests, and
switch across the package-metadata module to use
insert-missing-data-and-return-all-ids.
To associate a set of lint checkers with a specific revision. While there is
the association through the lint warnings, that only works for checkers where
there are lint warnings for that revision.
Therefore, to allow finding all the checkers for a specific revision, also
associate them directly with the revision.
When loading data from an inferior Guix, first build it's latest version of
glibc-locales, and include that in the environment from the inferior.
This improves locale support, which is currently relevant for extracting lint
warnings.
Without this change, you'd only be able to switch to locales supported by the
glibc-utf8-locales package, assuming that the right version is available.
This commit adds the relevant tables and code to store lint warnings in the
database.
Currently, only lint checkers which don't require access to the network will
be run, as this allows the processing to happen without network access. Also,
this functionality won't work in older versions of Guix which don't expose the
lint warnings in a compatible way.
Christopher Baines
Danjela Lura