guix/data-service
2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/data-service.git synced 2023-12-14 03:23:03 +01:00
Code Issues Releases Wiki Activity
1123 commits 1 branch 0 tags 1.8 MiB
Commit graph

1123 commits

This branch
This branch
All branches
Author SHA1 Message Date
Christopher Baines 11ec8a8064 Make the revision derivation counts numbers 2022-01-22 18:47:38 +00:00
Christopher Baines cbef46e682 Include the commit hash in the revision JSON response 
For the latest processed revision, this is useful for looking up which is the
latest processed revision.
 2022-01-22 17:18:05 +00:00
Christopher Baines a7c9daab6a Process derivations in chunks 
Which should reduce the peak memory usage.
 2022-01-14 15:25:53 +00:00
Christopher Baines 5a1b6d41eb Fix chunk-for-each! calling proc when there's nothing to do 2022-01-14 15:25:35 +00:00
Christopher Baines 5ae8b796a7 Rename chunk-map! to chunk-for-each! 
As that better reflects what it does.
 2022-01-14 15:25:13 +00:00
Christopher Baines fcf1177d2c Don't count exiting with a status of 0 as a failure 2022-01-14 15:24:12 +00:00
Christopher Baines 21cb33a859 Re-write insert-derivation-inputs in a more memory efficient manor 
Previously it would compute a long list of strings, potentially more than
100,000 elements long, then split this string up and insert it in chunks. Only
then could memory be freed.

This new approach builds the strings in batches for the insertion query, then
moves on to the next batch. This should mean that more memory can be freed and
reused along the way.
 2022-01-12 18:18:15 +00:00
Christopher Baines 6102553d94 Record job failure on non-0 exit status 
This is helpful when the jobs fail through Guile running out of memory for
example.
 2022-01-12 18:17:52 +00:00
Christopher Baines f1d8d76c4d Improve the process jobs script signal handling 
The return value of sleep is unreliable (see guile bug #53139), so use a
signal handler instead.
 2022-01-09 10:30:03 +00:00
Christopher Baines 7436283989 Increase the default job processing timeout 2022-01-09 10:12:06 +00:00
Christopher Baines 3840f588e5 Improve logging for system test derivation issues 2022-01-09 10:11:53 +00:00
Christopher Baines 78b8b4b422 Update the guix-dev.scm file 2022-01-09 10:11:45 +00:00
Christopher Baines 8beab2511c Query substitutes for latest processed revisions periodically 
This is a step towards having up to date substitute availability data.
 2021-11-16 19:08:46 +00:00
Christopher Baines ba9bcbf735 Use a bigger start size for the hash table 
This might help when there's lots of derivations to insert.
 2021-10-03 15:28:40 +01:00
Christopher Baines b28d338de7 Insert derivations in chunks 
To avoid making a very large query when inserting lots of derivations.
 2021-10-03 14:54:43 +01:00
Christopher Baines 0796cb3bd3 Add a chunk procedure 
Just a variant of chunk! which doesn't modify the provided list.
 2021-10-03 14:54:10 +01:00
Christopher Baines af0a06d147 Log the time to read missing derivations from the store 2021-10-03 12:59:26 +01:00
Christopher Baines 3627d36d77 Select existing derivations in chunks 
To avoid one massive query.
 2021-10-03 12:59:02 +01:00
Christopher Baines 857b4e32d5 Insert derivation inputs in chunks 
To avoid one massive query.
 2021-10-03 12:56:23 +01:00
Christopher Baines d5ab67000e Add a chunk! untility 
For splitting a list in to multiple chuncks, satisfying some max length.
 2021-10-03 12:55:21 +01:00
Christopher Baines 211da6868f Handle the case where there are no missing file names 
In update-derivation-ids-hash-table!.
 2021-09-25 00:09:08 +01:00
Christopher Baines 3081887b90 Optimise inserting derivation inputs 
Rather than querying for the output ids one by one and then running an insert
query for each derivation, perform the task with a single insert query.
 2021-09-24 18:22:28 +01:00
Christopher Baines abff41f9ae Neaten up formatting in select-derivation-output-id 2021-09-24 17:26:48 +01:00
Christopher Baines 947cabb685 Replace derivation-file-names->vhash 
Rather than creating vhashes, just update the hash table that is used as a
cache, and query that. This should speed things up and reduce memory usage
when loading derivations.
 2021-09-24 17:14:40 +01:00
Christopher Baines 261552bd5e Speed up inserting missing derivation sources 
Split the recursive part of the query from the non-recursive part, since
PostgreSQL doesn't do a great job of estimating the number of rows which will
come back from the recursive part, and thus generates a bad plan.
 2021-09-24 12:55:45 +01:00
Christopher Baines 8b34126d22 Fix the select-source-files-missing-nars query 
Previously, the VALUES list wasn't being generated properly.
 2021-09-16 17:01:34 +01:00
Christopher Baines f2b9663cf2 Make sure to add missing source file nars when inserting derivations 
Source file nars weren't always recorded, so this will help backfill that
data (which enables providing substitutes for those derivations).
 2021-09-14 21:19:48 +01:00
Christopher Baines 52094f5f19 Handle #f package descriptions 
One was added in e81cf4e79a6e297db0ae2a9c39eab495e7e204f0.
 2021-08-09 09:43:03 +01:00
Christopher Baines 5cefa250d8 Add an index on derivation_output_details.id 
Where hash is not null, since this helps with the fixed output package
derivations query.
 2021-07-11 15:25:12 +01:00
Christopher Baines d1a2a7125c Fix a regression with running sqitch 
Introduced in 0dc05982cd.
 2021-07-11 12:40:48 +01:00
Christopher Baines af209170f7 Track package replacements 
Start at least looking for package replacements, and storing the
details (particularly the derivation). I'm looking at doing this so that build
servers using the Guix Data Service can build these derivations.
 2021-07-11 11:57:05 +01:00
Christopher Baines b4188bda9d Run sqitch in the change mode 
Since this rolls back migrations less, which is good when the rollback bit
isn't always implemented.
 2021-07-04 10:43:13 +01:00
Christopher Baines 0dc05982cd Try to adapt the PostgreSQL paramstring to use with sqitch 2021-06-16 13:44:00 +01:00
Christopher Baines 673d762e68 Remove redundant derivation-sources call 2021-06-09 16:38:33 +01:00
Christopher Baines 1a21bc40a8 Pass #:system to channel-instances->manifest 
This is better than setting the %current-system, since more of the process
will run as native code.
 2021-06-09 10:59:31 +01:00
Christopher Baines 8a48960fa7 Use derivations.system_id in select-builds-with-context 
As that table is always included in the query.
 2021-06-09 10:35:29 +01:00
Christopher Baines f421faa790 Fix a uri encoding issue with linking to outputs 2021-06-09 10:28:11 +01:00
Canan Talayhan 9c1b989d20 Add index for package_metadata 
This index speeds up inserting new entries to the package_metadata table.

Signed-off-by: Christopher Baines <mail@cbaines.net>
 2021-05-27 21:26:56 +01:00
Christopher Baines 854f2cbba9 Fix a URI decoding issue 
This caused a problem viewing this derivation
/gnu/store/65r4488732qy6ilrh1ryb8xwmip1qnir-tklib-0.6.tar.gz?uuid=tklib-0-6.drv
 2021-05-27 09:38:41 +01:00
Christopher Baines f4a87ec059 Add some indexes 
This is from reading the slow query log on data.guix.gnu.org.
 2021-05-17 18:40:45 +01:00
Christopher Baines b8bfdcf622 Remove the "derivations using output" data 
From the HTML and JSON responses, since this data isn't very useful, and the
query to collect the small amount which is displayed is very slow. This could
return at some point, but in a proper paginated form which isn't slow.
 2021-05-17 18:35:19 +01:00
Christopher Baines da0ee9dff0 Use filter-map rather than filter and map when processing linters 
I guess this is a good change in general, but this seems to avoid a long
stack, which when a linter crashes, and the inferior tries to return the
exception details, and apparently hang the inferior/client as the reply isn't
written/read.
 2021-05-16 20:54:07 +01:00
Christopher Baines 2a4b16f5e4 Fix incorrect query parameters in a couple of builds queries 
The parameters must be strings rather than numbers.
 2021-05-12 08:28:06 +01:00
Christopher Baines cdb41cfa3c Fix query typo in branches-by-package-version 2021-05-12 08:27:48 +01:00
Christopher Baines d13a7187af Fix request->path-components-and-mime-type 
Previously, it wouldn't pull the mime types out of the request properly, I
think it does a better job of that now.
 2021-05-12 08:26:15 +01:00
Luciana Lima Brito 767e60b2b3 Change data handling when comparing derivations 
comparison.scm: return query data for derivation comparison as an alist,
instead of list.
html.scm: Access derivation differences data using assq-ref.
controller.scm: remove mapping for outputs/inputs/sources.
utils.scm: add group-to-alist/vector function.

Signed-off-by: Christopher Baines <mail@cbaines.net>
 2021-04-27 21:18:51 +01:00
Christopher Baines e38bddcae5 Reformat a couple of package related queries 2021-04-24 06:00:47 +01:00
Christopher Baines 1f078e1af0 Fix more things for the systems table introduction 2021-04-23 23:10:28 +01:00
Christopher Baines c252d1cb30 Actually add the system module 2021-04-23 18:42:40 +01:00
Christopher Baines 869d2522e5 Add the system module to Makefile.am 2021-04-23 18:07:55 +01:00