guix/data-service
2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/data-service.git synced 2023-12-14 03:23:03 +01:00
Code Issues Releases Wiki Activity
1173 commits 1 branch 0 tags 1.8 MiB
Commit graph

1173 commits

This branch
This branch
All branches
Author SHA1 Message Date
Christopher Baines ca1e4819b6 Fix closing thread postgresql connections 2022-11-17 16:32:04 +00:00
Christopher Baines 9fc5821180 Include more information about invalid query parameters 
In the /compare response.

This should enable qa.guix.gnu.org to detect when the base revision for a
comparison is unknown.
 2022-11-17 16:18:28 +00:00
Christopher Baines cc61bb5f13 Drop the chunk size when gathering lint warnings 
To try and bring the peak memory usage down.
 2022-11-14 09:26:59 +00:00
Christopher Baines ab7df4c6e5 Include blocked_builds information in comparison responses 
This will make it easier to tell when a scheduled build is yet to start, and
can't start due to a missing dependency.
 2022-11-14 09:24:49 +00:00
Christopher Baines 8294accffe Remove Build status field from blocking builds page 
As this is unused.
 2022-11-12 11:54:50 +00:00
Christopher Baines c46ee47632 Make backfilling blocked_builds a bit smarter 
And drop the chunk size.
 2022-11-12 11:53:14 +00:00
Christopher Baines ed114265cd Handle deleting from blocked_builds when builds are scheduled 
As scheduling a build might unblock others.
 2022-11-12 11:42:26 +00:00
Christopher Baines b9305d81a4 View scheduled builds like succeeded builds in terms of blocking 
This means that an output is viewed to not be blocking if it has a scheduled
build, just as if it has a succeeded build. Also, scheduling builds will
unblock blocked builds.

This is helpful as it means that it reduces noise for blocking builds.
 2022-11-12 11:33:37 +00:00
Christopher Baines 7731c6f340 Tweak backfilling the blocked builds 2022-11-12 10:57:53 +00:00
Christopher Baines 8e9ab68d14 Use latest_build_status rather than build_status 
In various places in the blocked-builds module.
 2022-11-12 10:57:26 +00:00
Christopher Baines a34bf4defc Spawn specific PostgreSQL connections for the blocked_builds updates 
So that the queries don't get cancelled by the statement timeout.
 2022-11-12 10:46:46 +00:00
Christopher Baines fc5f562731 Add index on derivation_outputs id and derivation_id fields 
This might help with doing recursive queries on the derivations graph.
 2022-11-12 10:42:04 +00:00
Christopher Baines fb9d99a076 Add extended statistics on package_derivations 
This helps row count estimates when filtering on system_id and target.
 2022-11-12 10:40:43 +00:00
Christopher Baines 48d8ee885a Have insert-blocked-builds cache when the partitions exist 
To make it more efficient.
 2022-11-11 11:29:45 +00:00
Christopher Baines 0f22e3ab40 Rework insert-blocked-builds to make it more efficient 
This also fixes a typo in the partition name.
 2022-11-11 11:29:37 +00:00
Christopher Baines 4101c6b897 Stop using exception handling when inserting blocked_builds entries 
As it doesn't work in a transaction.
 2022-11-11 10:51:12 +00:00
Christopher Baines 01e54c36db Add missing test description 2022-11-11 10:35:09 +00:00
Christopher Baines 989916b740 Add a blocking builds page 2022-11-11 10:35:09 +00:00
Christopher Baines 1fb291be40 Add support for incrementally tracking blocked builds 
This will hopefully provide a less expensive way of finding out if a scheduled
build is probably blocked by other builds failing or being canceled.

By working this out when the build events are recieved, it should be more
feasible to include information about whether builds are likely blocked or not
in various places (e.g. revision comparisons).
 2022-11-11 10:35:09 +00:00
Christopher Baines 95064d39a3 Log heap size when computing system tests 2022-11-06 08:53:04 +01:00
Christopher Baines 1e2826e095 Add more granular logging for computing system test derivations 2022-11-06 08:47:58 +01:00
Christopher Baines d06230fcf4 Close postgresql connections when the thread pool thread is idle 
I think the idle connections associated with idle threads are still taking up
memory, so especially now that you can configure an arbitrary number of
threads (and thus connections), I think it's good to close them regularly.
 2022-10-23 11:28:37 +01:00
Christopher Baines aaec813cba Fix JSON response for unknown revisions 2022-10-17 16:07:46 +01:00
Christopher Baines 8dd164665a Expose Guile GC metrics 
As the guix-data-service process seems to be using excessive amounts of
memory, and this will be useful to track it.
 2022-10-09 11:13:28 +01:00
Christopher Baines ff77bbea7e Make it possible to increase the number of thread pool threads 
And double the default to 16.
 2022-10-02 15:08:18 +01:00
Christopher Baines ce2e13aa45 Log delays in the thread pool channel 
As I think with lots of requests, this could become a bottleneck.
 2022-10-02 14:57:02 +01:00
Christopher Baines 640386a84d Insert guix revision lint warnings in chunks 
To avoid long running queries.
 2022-09-17 08:53:23 +02:00
Christopher Baines 78a5abee21 Improve chunking when inserting derivation inputs 
Chunk the values inserted in the query, rather than the derivations involved,
as this is more consistent.
 2022-09-17 08:53:23 +02:00
Christopher Baines 7050ea749f Reduce some chunk sizes 2022-09-17 00:40:51 +02:00
Christopher Baines 35cf9ba1bc Chunk inserting guix revision package derivation entries 2022-09-15 16:25:41 +02:00
Christopher Baines 8ef896b103 Further reduce some chunk sizes 2022-09-15 16:25:31 +02:00
Christopher Baines 37e8af60fb Chunk the data for some queries in insert-missing-data-and-return-all-ids 
This helps to avoid queries getting logged as slow just because of the amount
of data.
 2022-09-15 09:18:19 +01:00
Christopher Baines d791c2bd8a Format some queries generated in insert-missing-data-and-return-all-ids 2022-09-14 15:42:16 +01:00
Christopher Baines f41bfcf8b6 Reduce some chunk sizes 
As these queries are still slow enough to be logged.
 2022-09-14 15:42:00 +01:00
Christopher Baines 189b9a4e86 Implement the JSON response for package reproducibility 2022-09-14 10:52:04 +01:00
Christopher Baines ec3dba4894 Implement the JSON response for package substitute availability 2022-09-14 10:37:33 +01:00
Christopher Baines e3062abf33 Speed up finding the locales for a revision 2022-09-14 10:26:03 +01:00
Christopher Baines 12af30c039 Reduce chunk size for inserting dervation inputs 
As this query can take some time.
 2022-09-14 09:48:59 +01:00
Christopher Baines 77c4e1cb63 Reduce the chunk size for querying related derivation ids 
And include the chunk size in the log message.
 2022-09-13 21:00:04 +01:00
Christopher Baines ad39492cf0 Support per repository configuration for fetching substitutes 
This is so that data.qa.guix.gnu.org can be configured only to query the
branches from the main repository.
 2022-09-09 12:39:38 +01:00
Christopher Baines 45e25c2436 Make caching exceptions when querying for substitutes more reliable 2022-09-09 12:33:15 +01:00
Christopher Baines 076331325a Log more information about heap size when loading derivation info 
To better understand the memory usage when this is happening.
 2022-09-05 14:23:10 +01:00
Christopher Baines b3d59c650a Use much smaller chunks when trying to run the derivation linter 
Since larger chunks still ran in to inferior memory usage problems.
 2022-09-05 14:22:38 +01:00
Christopher Baines aa8c9dbffa Compute lint warnings for packages in chunks 
In an attempt to reduce the peak memory usage, and avoid running in to the:

  Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS

issue.
 2022-09-05 08:40:27 +01:00
Christopher Baines 1043b51bff Include lint warnings in the JSON /compare response 2022-08-22 17:42:37 +01:00
Christopher Baines ee73d2cc98 Return 503 for the /healthcheck if the service is starting up 
As this is a little clearer.
 2022-07-15 09:24:46 +01:00
Christopher Baines 1962b1f61c Fix missing NULL variable 2022-07-09 17:14:55 +01:00
Christopher Baines 39487cd7e6 Improve deleting derivations 
Drop the batch size to get rid of warnings about memory usage and improve the
logging by adding duration information.
 2022-07-08 20:55:58 +01:00
Christopher Baines 788571f53f Set builds derivation output details set id if information available 
The build event information can now contain the derivation outputs, as well as
the name of the derivation. This allows the Guix Data Service to join builds
up with derivations, even if it doesn't know about the derivation being built.
 2022-07-08 13:48:08 +01:00
Christopher Baines 6da5e8e67b Sort derivation output details ids 
To ensure that direct array comparison can be used in the query.
 2022-07-08 13:47:52 +01:00