Christopher Baines
189b9a4e86
Implement the JSON response for package reproducibility
2022-09-14 10:52:04 +01:00
Christopher Baines
ec3dba4894
Implement the JSON response for package substitute availability
2022-09-14 10:37:33 +01:00
Christopher Baines
e3062abf33
Speed up finding the locales for a revision
2022-09-14 10:26:03 +01:00
Christopher Baines
12af30c039
Reduce chunk size for inserting dervation inputs
...
As this query can take some time.
2022-09-14 09:48:59 +01:00
Christopher Baines
77c4e1cb63
Reduce the chunk size for querying related derivation ids
...
And include the chunk size in the log message.
2022-09-13 21:00:04 +01:00
Christopher Baines
ad39492cf0
Support per repository configuration for fetching substitutes
...
This is so that data.qa.guix.gnu.org can be configured only to query the
branches from the main repository.
2022-09-09 12:39:38 +01:00
Christopher Baines
45e25c2436
Make caching exceptions when querying for substitutes more reliable
2022-09-09 12:33:15 +01:00
Christopher Baines
076331325a
Log more information about heap size when loading derivation info
...
To better understand the memory usage when this is happening.
2022-09-05 14:23:10 +01:00
Christopher Baines
b3d59c650a
Use much smaller chunks when trying to run the derivation linter
...
Since larger chunks still ran in to inferior memory usage problems.
2022-09-05 14:22:38 +01:00
Christopher Baines
aa8c9dbffa
Compute lint warnings for packages in chunks
...
In an attempt to reduce the peak memory usage, and avoid running in to the:
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
issue.
2022-09-05 08:40:27 +01:00
Christopher Baines
1043b51bff
Include lint warnings in the JSON /compare response
2022-08-22 17:42:37 +01:00
Christopher Baines
ee73d2cc98
Return 503 for the /healthcheck if the service is starting up
...
As this is a little clearer.
2022-07-15 09:24:46 +01:00
Christopher Baines
1962b1f61c
Fix missing NULL variable
2022-07-09 17:14:55 +01:00
Christopher Baines
39487cd7e6
Improve deleting derivations
...
Drop the batch size to get rid of warnings about memory usage and improve the
logging by adding duration information.
2022-07-08 20:55:58 +01:00
Christopher Baines
788571f53f
Set builds derivation output details set id if information available
...
The build event information can now contain the derivation outputs, as well as
the name of the derivation. This allows the Guix Data Service to join builds
up with derivations, even if it doesn't know about the derivation being built.
2022-07-08 13:48:08 +01:00
Christopher Baines
6da5e8e67b
Sort derivation output details ids
...
To ensure that direct array comparison can be used in the query.
2022-07-08 13:47:52 +01:00
Christopher Baines
db37d9f6a8
Split out inserting derivation output details sets
...
So that this can be used when inserting builds.
2022-07-08 13:47:52 +01:00
Christopher Baines
811256a920
Split out inserting into derivation_output_details
...
So that this can be done when inserting builds.
2022-07-08 13:47:52 +01:00
Christopher Baines
8e23d38660
Handle migrations and server startup better
...
The server part of the guix-data-service doesn't work great as a guix service,
since it often fails to start if the migrations take any time at all.
To address this, start the server before running the migrations, and serve the
pages that work without the database, plus a general 503 response. Once the
migrations have completed, switch to the normal behaviour.
2022-06-17 13:13:21 +01:00
Christopher Baines
d19eb07138
Add an index on git_commits
...
And also remove the duplicates that have crept in.
2022-06-17 12:54:43 +01:00
Christopher Baines
3b5950024b
Add an index on package_derivations_by_guix_revision_range
...
This helps render the package version range related pages.
2022-06-17 12:51:10 +01:00
Christopher Baines
22c2ed2fa7
Fix ambiguous id column in delete-guix-revisions query
2022-06-16 12:46:32 +01:00
Christopher Baines
754f64718f
Fix DELETE query in delete-revisions-from-branch
2022-06-16 12:38:51 +01:00
Christopher Baines
be45e4251e
Fix ambiguous id column in delete-from-git-commits
2022-06-16 12:30:08 +01:00
Christopher Baines
71aaf1016b
Remove duplicate AND from delete-from-git-commits query
2022-06-16 12:25:47 +01:00
Christopher Baines
5b82be803c
Fix the select-guix-revision-for-branch-and-datetime query
...
I think it was broken with the git_branches/git_commits switch.
2022-05-30 23:29:42 +01:00
Christopher Baines
4b8846a861
Remove cross derivation targets that don't make sense
...
This matches the previous behaviour without using the platform data.
2022-05-27 10:09:09 +01:00
Christopher Baines
fb8353559f
Take advantage of the new (guix platform) module
...
This means there's less reliance on the hardcoded lists of systems and targets
and mappings between them.
2022-05-26 00:24:55 +01:00
Christopher Baines
786a5fa041
Improve the package derivation by guix revision range insert queries
2022-05-26 00:23:56 +01:00
Christopher Baines
e0f5ad3f51
Add missing comma in query
2022-05-24 13:56:48 +01:00
Christopher Baines
b73cfa5d49
Fix delete-guix-revision-package-derivation-entries
...
For the table schema change.
2022-05-24 12:22:52 +01:00
Christopher Baines
512b43ec73
Fix expected exec-query value when no branch exists
2022-05-24 08:43:32 +01:00
Christopher Baines
d4bb0ffaaa
Fix more issues with the git_commits introduction
2022-05-23 22:49:51 +01:00
Christopher Baines
6d403cbc8d
Allow filtering package derivations based on build server builds
...
This means you can query for derivations where builds exist or don't exist on
a given build server.
I think this will come in useful when submitting builds from a Guix Data
Service instance.
2022-05-23 22:39:32 +01:00
Christopher Baines
40035bcf9a
Fix more of rebuilding one of the tables
2022-05-23 21:18:59 +01:00
Christopher Baines
3596eceed5
Add missing git_branches query part to insert query
2022-05-23 20:22:54 +01:00
Christopher Baines
e5cb793d4e
Raise a clearer exception when a linter crashes
2022-05-23 19:19:57 +01:00
Christopher Baines
64be52844e
Partition the package_derivations_by_guix_revision_range table
...
And create a proper git_branches table in the process.
I'm hoping this will help with slow deletions from the
package_derivations_by_guix_revision_range table in the case where there are
lots of branches, since it'll separate the data for one branch from another.
These migrations will remove the existing data, so
rebuild-package-derivations-table will currently need manually running to
regenerate it.
2022-05-23 19:10:25 +01:00
Christopher Baines
89545caa3f
Fix branch links on the repository page
...
Thanks to Tobias for reporting.
2022-05-20 12:18:58 +01:00
Christopher Baines
198b6ef719
Only clear the %store-table when it's defined
2022-05-17 12:06:09 +01:00
Christopher Baines
5727703d84
Clear cached store connections when fetching lint warnings
...
As I'm seeing the inferior process crash with [1] just after fetching the
derivation lint warnings.
This change appears to help, although it's probably just a workaround. When
there's more packages/derivations, the caches might need clearing while
fetching the derivation lint warnings, or this will need to be split across
multiple processes.
1: Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
2022-05-13 12:03:43 +01:00
Christopher Baines
ff116d5e64
Clear out cached store connections in the load revision inferior
...
These cached store connections have caches associated with them, that take up
lots of memory, leading to the inferior crashing. This change seems to help.
2022-05-07 09:55:13 +01:00
Christopher Baines
9607bcedfe
Move the builds.derivation_output_details_set_id update
...
To the end of the main revision processing transaction.
Currently, I think there are issues when this query does update some builds,
as those rows in the build table remain locked until the end of the
transaction. This then causes build event submission to hang. Moving this part
of the revision loading process to the end of the transaction should help to
mitigate this.
2022-04-16 18:47:54 +01:00
Christopher Baines
27c34a9ca5
Fix a couple of things on the package substitute availability page
...
When there's a target, render the heading neatly, and include the target
parameter in the URLs.
2022-03-12 10:49:19 +00:00
Christopher Baines
c0679185b8
Fix an issue with the /builds page
2022-03-11 13:27:59 +00:00
Christopher Baines
097e22ab5e
Close the load revision inferior prior to inserting data
...
This means that the lock can be acquired after closing the inferior, freeing
the large amount of memory that the inferior process is probably using.
2022-03-11 13:27:55 +00:00
Christopher Baines
df4e0a7a61
Add to the hardcoded list of valid targets
...
Since the hardcoded list in the load-new-guix-revision code has been updated.
2022-03-11 11:50:10 +00:00
Christopher Baines
0f07826a20
Extract out the code for starting an inferior
2022-03-11 11:22:08 +00:00
Christopher Baines
8379427cbb
Process each system target pair individually
...
As the cross targets take quite some time.
2022-03-11 11:12:18 +00:00
Christopher Baines
9db755f27d
Disable value history in the inferior repl
...
This might help reduce memory usage a little.
2022-03-11 11:11:53 +00:00