guix/data-service
2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/data-service.git synced 2023-12-14 03:23:03 +01:00
Code Issues Releases Wiki Activity
1258 commits 1 branch 0 tags 1.8 MiB
Commit graph

1258 commits

This branch
This branch
All branches
Author SHA1 Message Date
Christopher Baines f2e2ca7a07 Use the connection pool for build events 
I'm not sure why this wasn't the way it worked previously. This should avoid
unconstrained use of PostgreSQL connections.
 2023-08-23 12:19:22 +01:00
Christopher Baines 9f102dbd39 Add code to delete nars entries 2023-08-01 14:13:10 +01:00
Christopher Baines 1461aa037f Make database changes to help with deleting entries in the nars table 2023-08-01 14:12:48 +01:00
Christopher Baines 7495085f63 Delete unreferenced derivations in batches 
To avoid a long blocking query.
 2023-08-01 10:16:31 +01:00
Christopher Baines 1c75394187 Fix issues with with-resource-from-pool 2023-07-26 10:11:00 +01:00
Christopher Baines abe50b0443 Fix the load revision system sorting 
As previously it only applied to system tests.
 2023-07-25 17:58:19 +01:00
Christopher Baines bbc53deb1f Rewrite deleting unreferenced derivations 
Use fibers more, leaning in on the non-blocking use of Squee for parallelism.
 2023-07-25 17:57:00 +01:00
Christopher Baines 348fe36b55 Sort systems and targets for more deterministic behaviour 
As the ordering from Guix seems to be non-deterministic.
 2023-07-21 12:10:55 +01:00
Christopher Baines ef73305250 Further tweak loading package derivations 
There's an issue where sometimes for i686-linux and armhf-linux, only a few
package derivations are computed.

This commit tries to simplify the code, and adds some conditional logging for
the guix package, which might help reveal what's going on.
 2023-07-20 16:46:53 +01:00
Christopher Baines 6e4d436d75 Fix some uses of select-build-server-urls-by-id 2023-07-20 16:12:00 +01:00
Christopher Baines ce1480edf4 Fix the repositories page 2023-07-19 17:40:41 +01:00
Christopher Baines de6d38016d Tweak which duplicate packages to select 
This helps with replacements, as the original package is usually higher up in
the file.
 2023-07-17 13:28:42 +01:00
Christopher Baines ed974ebf3b Tweak loading package derivations 
Make sure to log any errors, and also use a more efficient approach sending
less data to the inferior.
 2023-07-17 13:27:21 +01:00
Christopher Baines 9dec45d2eb Make fetching metrics work even when having database problems 2023-07-14 10:36:02 +01:00
Christopher Baines f9bb60ab4a Drop the resource pool timeout 
Better to timeout early.
 2023-07-14 10:35:44 +01:00
Christopher Baines 22ed81fbcc Reduce the number of threads used by fibers 
As one thread per core is probably unnecessary.
 2023-07-10 18:56:31 +01:00
Christopher Baines 2971322b27 Set thread names 
To help with debugging
 2023-07-10 18:56:31 +01:00
Christopher Baines 75ef4cffd3 Expose resource pool stats 2023-07-10 18:56:31 +01:00
Christopher Baines 899bd1387e Support getting resource pool stats 2023-07-10 18:56:31 +01:00
Christopher Baines 7251c7d653 Stop using a pool of threads for database operations 
Now that squee cooperates with suspendable ports, this is unnecessary. Use a
connection pool to still support running queries in parallel using multiple
connections.
 2023-07-10 18:56:31 +01:00
Christopher Baines 672ee6216e Try to fix backfilling blocked_builds 2023-07-02 10:42:49 +01:00
Christopher Baines d12ae0079e Filter out duplicate ids for blocking builds 2023-07-02 10:42:24 +01:00
Christopher Baines 742949cc97 Improve data deletion 2023-07-01 12:01:13 +01:00
Christopher Baines 044a905c1d Provide more information about revision processing errors 
In the compare package derivations response.
 2023-06-15 11:36:46 +01:00
Christopher Baines f7b823cfd9 Fix fetch-narinfo-files to work in the specific outputs case 2023-06-09 16:32:54 +01:00
Christopher Baines 04e90a17b0 Better report exceptions in query-build-server-substitutes 2023-06-09 16:11:06 +01:00
Christopher Baines 29d49ba31a Detach the database setup from the main guix-data-service process 
This will allow restarting them independently, leaving it up to the operator
to ensure that all processes are compatible.
 2023-06-09 16:11:06 +01:00
Christopher Baines 5c9ec28cb5 Query for outputs when build events arrive 
This will keep the substitute information more up to date.
 2023-06-09 16:11:06 +01:00
Christopher Baines 7524d23b44 Make the build event handling code more generic 
So that triggering a check for substitutes can be integrated in.
 2023-06-06 12:32:53 +01:00
Christopher Baines 68850065d7 Extend the invalid parameter handling for more types 2023-05-22 17:34:39 +01:00
Christopher Baines 5f21458a8f Have render-compare/package-derivations provide invalid param info 
So that this can be used by the qa-frontpage.

This should be improved and generalised.
 2023-05-22 17:25:44 +01:00
Christopher Baines 54c7a1a880 Fix ignoring canceled builds 
The previous changes only affected searching for package derivations, and they
also didn't work.
 2023-05-18 12:31:58 +01:00
Christopher Baines 4208b5f148 Ignore canceled builds when querying package derivations 
This will help when using this to submit builds, since you won't end up
ignoring derivations with canceled builds.
 2023-05-18 11:25:14 +01:00
Christopher Baines 61f6619313 Add Guile internal time metrics 
As this will help identify when the service restarts.
 2023-05-14 22:45:19 +01:00
Christopher Baines bbacbcd2aa Ensure the known and unknown keys appear 2023-05-09 13:45:36 +01:00
Christopher Baines c50f6aeb34 Remove redundant match-lambda in select-package-output-availability-for-revision 2023-05-09 13:43:32 +01:00
Christopher Baines 47c482bdcc Set lock_timeout for some data deletion transactions 
As these can cause deadlocks. This will probably cause errors, so some
retrying will need to be added.
 2023-05-09 08:55:09 +01:00
Christopher Baines 3734a85650 Use the package_derivations system id in a query 
Rather than the derivations system id, as this helps PostgreSQL run the query
faster.
 2023-05-04 08:32:43 +01:00
Christopher Baines 639c6ff183 Further tweak fetching narinfos 
Move the batching to the database, which should reduce memory usage while
removing the limit on the number of fetched narinfos.
 2023-04-28 22:33:41 +02:00
Christopher Baines 8a19bcba13 Tweak fetching narinfos 
Currently I'm seeing failures due to guile-gnutls not supporting suspendable
ports (write_wait_fd), so batch the requested outputs to try and avoid this.
 2023-04-28 20:42:31 +02:00
Christopher Baines 688f4cd79d Set request timeouts for the thread pools 
The request timeout should ensure that the operations don't back up if the
thread pool is overloaded.
 2023-04-27 14:58:47 +02:00
Christopher Baines 638e0442c3 Support request timeouts in the thread pool 2023-04-27 14:58:44 +02:00
Christopher Baines 5bb7cf0c1c Change par-mapper' to use fetch-result-of-defered-thunks 
So the fetch-result-of-defered-thunk procedure can be removed.
 2023-04-27 11:59:01 +02:00
Christopher Baines 9f080524bc Split the thread pool used for database connections 
In to two thread pools, a default one, and one reserved for essential
functionality.

There are some pages that use slow queries, so this should help stop those
pages block other operations.
 2023-04-27 10:31:09 +02:00
Christopher Baines 4fa7a3601e Include distribution counts table in data deletion 2023-04-07 11:21:28 +01:00
Christopher Baines 1a0eaeb672 Improve performance of select-fixed-output-package-derivations-in-revision 2023-03-11 18:19:19 +00:00
Christopher Baines a20b929e90 Fix query in get-count-for-next-level 2023-03-09 14:05:41 +00:00
Christopher Baines 519f0c6f67 Defer backfilling derivation distribution counts until later 
After the migrations have run.
 2023-03-09 09:39:47 +00:00
Christopher Baines 319439d05a Reorder some vacuum operations when loading revisions 2023-03-09 09:17:07 +00:00
Christopher Baines 659dcc6a0d Avoid a recursive CTE for finding blocked builds where possible 
Use the new approach of looking up the distribution of the derivations, and
building a non recursive query specifically for this revision. This should
avoid PostgreSQL picking a poor plan for performing the query.
 2023-03-09 08:36:51 +00:00