0e3f65062a
Compute more cross derivations
2022-03-11 10:07:08 +00:00
fe556f4a4d
Deduplicate inferior packages including replacements
...
Previously, duplicates could creep through if the duplicate wasn't exported,
and only found as a replacement. Now they're filtered out.
This isn't ideal, as duplicates aren't always mistakes, it would be useful
still to capture this package, but having multiple entries for the same
name+version causes the comparison functionality to break.
2022-03-04 14:22:10 +00:00
0cc749a1fa
Simplify deduplicate-inferior-packages
...
Use the a-version and b-version variables, rather than calling the functions
again.
2022-03-04 14:20:41 +00:00
Maxime Devos
8ab72e633a
Include the nl_NL.UTF-8 locale when processing revisions
...
It's 100% translated according to
<https://translate.fedoraproject.org/projecs/guix/guix/nl/ >.
* guix-data-service/model/package-metadata.scm
(locales): Add nl_NL.utf-8
Signed-off-by: Christopher Baines <mail@cbaines.net>
2022-03-02 21:05:30 +00:00
3a90798567
Address a few issues in the load new guix revision tests
2022-03-02 18:23:26 +00:00
f86657915e
Try to further speed up inserting missing derivation source files
...
Switch from using a recursive query to doing a breath first search through the
graph of derivations, as I think PostgreSQL wasn't doing a great job of
planning the recursive queries (it would overestimate the rows involved, and
prefer sequential scans for the derivation_outputs table).
2022-03-02 18:00:36 +00:00
c5b504e94a
Speed up the finding of missing sources
...
Use larger batches and more efficient duplicate deletion.
2022-03-01 20:57:26 +00:00
6cd3541d1a
Add a faster delete-duplicates function
...
Which is useful when deleting duplicates from large lists.
2022-03-01 20:34:06 +00:00
4a9d45aa16
Skip dropping the log part sequence if there's a lock
...
So that the job completes. The sequence can be deleted later.
2022-02-21 13:02:57 +00:00
a0a7d66b1d
Move logging cleanup tasks to after the transaction commits
...
As I think some operations (like the database backup) can block the DROP
SEQUENCE bit, so at least this approach means that the main transaction should
commit and then the sequence is eventually dropped.
2022-02-19 09:54:39 +00:00
17167ef3e4
Change how package supported systems are handled
...
This code is a bit tricky, since it should be compatible with old and new guix
revisions. I think these changes stop computing package derivations for
invalid systems, while hopefully not breaking anything.
2022-02-18 12:21:08 +00:00
4a1088c216
Switch to guile-fibers@1.1
...
I think the main change required is just to stop accessing the now missing
current-fiber parameter.
2022-02-09 17:20:54 +00:00
8ed53099ba
Fix some URL encoding issues around derivation names
2022-01-31 20:27:11 +00:00
86cd96ea87
Rename data to nars for package derivation outputs
...
As this is clearer.
2022-01-31 20:26:47 +00:00
752d21ad05
Improve the way the field fields work on various pages
...
Remove the brackets from the values since this makes the set of values more
consistent, and don't display the no additional fields value on the page.
2022-01-31 20:25:06 +00:00
f1418c4e88
Support querying package derivation outputs without the nars
...
Since this speeds up the response if you don't need the nar information.
2022-01-31 20:24:27 +00:00
11ec8a8064
Make the revision derivation counts numbers
2022-01-22 18:47:38 +00:00
cbef46e682
Include the commit hash in the revision JSON response
...
For the latest processed revision, this is useful for looking up which is the
latest processed revision.
2022-01-22 17:18:05 +00:00
a7c9daab6a
Process derivations in chunks
...
Which should reduce the peak memory usage.
2022-01-14 15:25:53 +00:00
5a1b6d41eb
Fix chunk-for-each! calling proc when there's nothing to do
2022-01-14 15:25:35 +00:00
5ae8b796a7
Rename chunk-map! to chunk-for-each!
...
As that better reflects what it does.
2022-01-14 15:25:13 +00:00
fcf1177d2c
Don't count exiting with a status of 0 as a failure
2022-01-14 15:24:12 +00:00
21cb33a859
Re-write insert-derivation-inputs in a more memory efficient manor
...
Previously it would compute a long list of strings, potentially more than
100,000 elements long, then split this string up and insert it in chunks. Only
then could memory be freed.
This new approach builds the strings in batches for the insertion query, then
moves on to the next batch. This should mean that more memory can be freed and
reused along the way.
2022-01-12 18:18:15 +00:00
6102553d94
Record job failure on non-0 exit status
...
This is helpful when the jobs fail through Guile running out of memory for
example.
2022-01-12 18:17:52 +00:00
f1d8d76c4d
Improve the process jobs script signal handling
...
The return value of sleep is unreliable (see guile bug #53139 ), so use a
signal handler instead.
2022-01-09 10:30:03 +00:00
7436283989
Increase the default job processing timeout
2022-01-09 10:12:06 +00:00
3840f588e5
Improve logging for system test derivation issues
2022-01-09 10:11:53 +00:00
78b8b4b422
Update the guix-dev.scm file
2022-01-09 10:11:45 +00:00
8beab2511c
Query substitutes for latest processed revisions periodically
...
This is a step towards having up to date substitute availability data.
2021-11-16 19:08:46 +00:00
ba9bcbf735
Use a bigger start size for the hash table
...
This might help when there's lots of derivations to insert.
2021-10-03 15:28:40 +01:00
b28d338de7
Insert derivations in chunks
...
To avoid making a very large query when inserting lots of derivations.
2021-10-03 14:54:43 +01:00
0796cb3bd3
Add a chunk procedure
...
Just a variant of chunk! which doesn't modify the provided list.
2021-10-03 14:54:10 +01:00
af0a06d147
Log the time to read missing derivations from the store
2021-10-03 12:59:26 +01:00
3627d36d77
Select existing derivations in chunks
...
To avoid one massive query.
2021-10-03 12:59:02 +01:00
857b4e32d5
Insert derivation inputs in chunks
...
To avoid one massive query.
2021-10-03 12:56:23 +01:00
d5ab67000e
Add a chunk! untility
...
For splitting a list in to multiple chuncks, satisfying some max length.
2021-10-03 12:55:21 +01:00
211da6868f
Handle the case where there are no missing file names
...
In update-derivation-ids-hash-table!.
2021-09-25 00:09:08 +01:00
3081887b90
Optimise inserting derivation inputs
...
Rather than querying for the output ids one by one and then running an insert
query for each derivation, perform the task with a single insert query.
2021-09-24 18:22:28 +01:00
abff41f9ae
Neaten up formatting in select-derivation-output-id
2021-09-24 17:26:48 +01:00
947cabb685
Replace derivation-file-names->vhash
...
Rather than creating vhashes, just update the hash table that is used as a
cache, and query that. This should speed things up and reduce memory usage
when loading derivations.
2021-09-24 17:14:40 +01:00
261552bd5e
Speed up inserting missing derivation sources
...
Split the recursive part of the query from the non-recursive part, since
PostgreSQL doesn't do a great job of estimating the number of rows which will
come back from the recursive part, and thus generates a bad plan.
2021-09-24 12:55:45 +01:00
8b34126d22
Fix the select-source-files-missing-nars query
...
Previously, the VALUES list wasn't being generated properly.
2021-09-16 17:01:34 +01:00
f2b9663cf2
Make sure to add missing source file nars when inserting derivations
...
Source file nars weren't always recorded, so this will help backfill that
data (which enables providing substitutes for those derivations).
2021-09-14 21:19:48 +01:00
52094f5f19
Handle #f package descriptions
...
One was added in e81cf4e79a6e297db0ae2a9c39eab495e7e204f0.
2021-08-09 09:43:03 +01:00
5cefa250d8
Add an index on derivation_output_details.id
...
Where hash is not null, since this helps with the fixed output package
derivations query.
2021-07-11 15:25:12 +01:00
d1a2a7125c
Fix a regression with running sqitch
...
Introduced in 0dc05982cd
2021-07-11 12:40:48 +01:00
af209170f7
Track package replacements
...
Start at least looking for package replacements, and storing the
details (particularly the derivation). I'm looking at doing this so that build
servers using the Guix Data Service can build these derivations.
2021-07-11 11:57:05 +01:00
b4188bda9d
Run sqitch in the change mode
...
Since this rolls back migrations less, which is good when the rollback bit
isn't always implemented.
2021-07-04 10:43:13 +01:00
0dc05982cd
Try to adapt the PostgreSQL paramstring to use with sqitch
2021-06-16 13:44:00 +01:00
673d762e68
Remove redundant derivation-sources call
2021-06-09 16:38:33 +01:00
Christopher Baines
Maxime Devos