guix/data-service
2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/data-service.git synced 2023-12-14 03:23:03 +01:00
Code Issues Releases Wiki Activity
1264 commits 1 branch 0 tags 1.8 MiB
Commit graph

1264 commits

This branch
This branch
All branches
Author SHA1 Message Date
Christopher Baines d72b9a5239 Remove logging from enqueue-job-for-email 
As getmail treats output to stderr as the operation failing.
 2023-10-11 14:17:11 +01:00
Christopher Baines b0404d62fd Fix typo 2023-10-11 14:04:40 +01:00
Christopher Baines 10bad53ad5 Support polling git repositories for new branches/revisions 
This is mostly a workaround for the occasional problems with the guix-commits
mailing list, as it can break and then the data service doesn't learn about
new revisions until the problem is fixed.

I think it's still a generally good feature though, and allows deploying the
data service without it consuming emails to learn about new revisions, and is
a step towards integrating some kind of way of notifying the data service to
poll.
 2023-10-09 22:19:02 +01:00
Christopher Baines 9bb8f84741 Add commit message entry about UDP push notifications for commits 2023-09-27 15:36:00 +01:00
Christopher Baines 893cccf8e5 Add metrics for ports and file descriptors 2023-09-07 11:16:45 +01:00
Christopher Baines e8e1459769 URI encode some derivation names in links 
To handle characters like +.
 2023-09-07 11:14:11 +01:00
Christopher Baines f2e2ca7a07 Use the connection pool for build events 
I'm not sure why this wasn't the way it worked previously. This should avoid
unconstrained use of PostgreSQL connections.
 2023-08-23 12:19:22 +01:00
Christopher Baines 9f102dbd39 Add code to delete nars entries 2023-08-01 14:13:10 +01:00
Christopher Baines 1461aa037f Make database changes to help with deleting entries in the nars table 2023-08-01 14:12:48 +01:00
Christopher Baines 7495085f63 Delete unreferenced derivations in batches 
To avoid a long blocking query.
 2023-08-01 10:16:31 +01:00
Christopher Baines 1c75394187 Fix issues with with-resource-from-pool 2023-07-26 10:11:00 +01:00
Christopher Baines abe50b0443 Fix the load revision system sorting 
As previously it only applied to system tests.
 2023-07-25 17:58:19 +01:00
Christopher Baines bbc53deb1f Rewrite deleting unreferenced derivations 
Use fibers more, leaning in on the non-blocking use of Squee for parallelism.
 2023-07-25 17:57:00 +01:00
Christopher Baines 348fe36b55 Sort systems and targets for more deterministic behaviour 
As the ordering from Guix seems to be non-deterministic.
 2023-07-21 12:10:55 +01:00
Christopher Baines ef73305250 Further tweak loading package derivations 
There's an issue where sometimes for i686-linux and armhf-linux, only a few
package derivations are computed.

This commit tries to simplify the code, and adds some conditional logging for
the guix package, which might help reveal what's going on.
 2023-07-20 16:46:53 +01:00
Christopher Baines 6e4d436d75 Fix some uses of select-build-server-urls-by-id 2023-07-20 16:12:00 +01:00
Christopher Baines ce1480edf4 Fix the repositories page 2023-07-19 17:40:41 +01:00
Christopher Baines de6d38016d Tweak which duplicate packages to select 
This helps with replacements, as the original package is usually higher up in
the file.
 2023-07-17 13:28:42 +01:00
Christopher Baines ed974ebf3b Tweak loading package derivations 
Make sure to log any errors, and also use a more efficient approach sending
less data to the inferior.
 2023-07-17 13:27:21 +01:00
Christopher Baines 9dec45d2eb Make fetching metrics work even when having database problems 2023-07-14 10:36:02 +01:00
Christopher Baines f9bb60ab4a Drop the resource pool timeout 
Better to timeout early.
 2023-07-14 10:35:44 +01:00
Christopher Baines 22ed81fbcc Reduce the number of threads used by fibers 
As one thread per core is probably unnecessary.
 2023-07-10 18:56:31 +01:00
Christopher Baines 2971322b27 Set thread names 
To help with debugging
 2023-07-10 18:56:31 +01:00
Christopher Baines 75ef4cffd3 Expose resource pool stats 2023-07-10 18:56:31 +01:00
Christopher Baines 899bd1387e Support getting resource pool stats 2023-07-10 18:56:31 +01:00
Christopher Baines 7251c7d653 Stop using a pool of threads for database operations 
Now that squee cooperates with suspendable ports, this is unnecessary. Use a
connection pool to still support running queries in parallel using multiple
connections.
 2023-07-10 18:56:31 +01:00
Christopher Baines 672ee6216e Try to fix backfilling blocked_builds 2023-07-02 10:42:49 +01:00
Christopher Baines d12ae0079e Filter out duplicate ids for blocking builds 2023-07-02 10:42:24 +01:00
Christopher Baines 742949cc97 Improve data deletion 2023-07-01 12:01:13 +01:00
Christopher Baines 044a905c1d Provide more information about revision processing errors 
In the compare package derivations response.
 2023-06-15 11:36:46 +01:00
Christopher Baines f7b823cfd9 Fix fetch-narinfo-files to work in the specific outputs case 2023-06-09 16:32:54 +01:00
Christopher Baines 04e90a17b0 Better report exceptions in query-build-server-substitutes 2023-06-09 16:11:06 +01:00
Christopher Baines 29d49ba31a Detach the database setup from the main guix-data-service process 
This will allow restarting them independently, leaving it up to the operator
to ensure that all processes are compatible.
 2023-06-09 16:11:06 +01:00
Christopher Baines 5c9ec28cb5 Query for outputs when build events arrive 
This will keep the substitute information more up to date.
 2023-06-09 16:11:06 +01:00
Christopher Baines 7524d23b44 Make the build event handling code more generic 
So that triggering a check for substitutes can be integrated in.
 2023-06-06 12:32:53 +01:00
Christopher Baines 68850065d7 Extend the invalid parameter handling for more types 2023-05-22 17:34:39 +01:00
Christopher Baines 5f21458a8f Have render-compare/package-derivations provide invalid param info 
So that this can be used by the qa-frontpage.

This should be improved and generalised.
 2023-05-22 17:25:44 +01:00
Christopher Baines 54c7a1a880 Fix ignoring canceled builds 
The previous changes only affected searching for package derivations, and they
also didn't work.
 2023-05-18 12:31:58 +01:00
Christopher Baines 4208b5f148 Ignore canceled builds when querying package derivations 
This will help when using this to submit builds, since you won't end up
ignoring derivations with canceled builds.
 2023-05-18 11:25:14 +01:00
Christopher Baines 61f6619313 Add Guile internal time metrics 
As this will help identify when the service restarts.
 2023-05-14 22:45:19 +01:00
Christopher Baines bbacbcd2aa Ensure the known and unknown keys appear 2023-05-09 13:45:36 +01:00
Christopher Baines c50f6aeb34 Remove redundant match-lambda in select-package-output-availability-for-revision 2023-05-09 13:43:32 +01:00
Christopher Baines 47c482bdcc Set lock_timeout for some data deletion transactions 
As these can cause deadlocks. This will probably cause errors, so some
retrying will need to be added.
 2023-05-09 08:55:09 +01:00
Christopher Baines 3734a85650 Use the package_derivations system id in a query 
Rather than the derivations system id, as this helps PostgreSQL run the query
faster.
 2023-05-04 08:32:43 +01:00
Christopher Baines 639c6ff183 Further tweak fetching narinfos 
Move the batching to the database, which should reduce memory usage while
removing the limit on the number of fetched narinfos.
 2023-04-28 22:33:41 +02:00
Christopher Baines 8a19bcba13 Tweak fetching narinfos 
Currently I'm seeing failures due to guile-gnutls not supporting suspendable
ports (write_wait_fd), so batch the requested outputs to try and avoid this.
 2023-04-28 20:42:31 +02:00
Christopher Baines 688f4cd79d Set request timeouts for the thread pools 
The request timeout should ensure that the operations don't back up if the
thread pool is overloaded.
 2023-04-27 14:58:47 +02:00
Christopher Baines 638e0442c3 Support request timeouts in the thread pool 2023-04-27 14:58:44 +02:00
Christopher Baines 5bb7cf0c1c Change par-mapper' to use fetch-result-of-defered-thunks 
So the fetch-result-of-defered-thunk procedure can be removed.
 2023-04-27 11:59:01 +02:00
Christopher Baines 9f080524bc Split the thread pool used for database connections 
In to two thread pools, a default one, and one reserved for essential
functionality.

There are some pages that use slow queries, so this should help stop those
pages block other operations.
 2023-04-27 10:31:09 +02:00