guix/data-service
2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/data-service.git synced 2023-12-14 03:23:03 +01:00
Code Issues Releases Wiki Activity
247 commits 1 branch 0 tags 1.8 MiB
Commit graph

247 commits

This branch
This branch
All branches
Author SHA1 Message Date
Christopher Baines
af581784b2 Log the guix store path being used 
In the load-new-guix-revision module.
 2019-09-01 13:12:10 +01:00
Christopher Baines
f118ed0bf9 Improve handling of errors when changing locale 
In the load-new-guix-revision module.
 2019-09-01 13:12:10 +01:00
Christopher Baines
3a79449974 Display counts of lint warnings on the revision page 2019-09-01 13:12:10 +01:00
Christopher Baines
75f2c19fc8 Add lint-warning-count-by-lint-checker-for-revision 
To return counts of lint warnings, grouped by lint checker for a specific
revision.
 2019-09-01 13:12:10 +01:00
Christopher Baines
6b9977f62e Store lint warnings in the database 
This commit adds the relevant tables and code to store lint warnings in the
database.

Currently, only lint checkers which don't require access to the network will
be run, as this allows the processing to happen without network access. Also,
this functionality won't work in older versions of Guix which don't expose the
lint warnings in a compatible way.
 2019-09-01 13:12:10 +01:00
Christopher Baines
bf469504eb Add buffering to the log handling for jobs 
To increase the log output speed, avoiding inserting one character at a time
in to the database.
 2019-08-31 12:10:54 +01:00
Christopher Baines
8c741c569b Add insert-missing-data-and-return-all-ids to the model utils module 
This should help greatly with populating the database with new entries, and
greatly reduce code duplication.
 2019-08-31 12:09:54 +01:00
Christopher Baines
657c72c203 Use $STR$ for better quoting in quote-string 2019-08-31 12:09:31 +01:00
Christopher Baines
c7baf53c3f Add value->sql-boolean to (guix-data-service model utils) 2019-08-31 12:07:58 +01:00
Christopher Baines
66df42ae4f Fix insert-guix-revision to return an id 
Rather than an id wrapped in a list.
 2019-08-31 12:07:22 +01:00
Christopher Baines
d35cc85d52 Improve error handling for processing emails 2019-08-31 12:06:36 +01:00
Christopher Baines
84ad3656df Alphabetically order the derivation comparison page 2019-08-06 10:33:00 +01:00
Christopher Baines
89b498aad0 Remove the build status filter from the compare derivations page 
As there isn't sufficient data to make this work yet.
 2019-08-06 10:24:19 +01:00
Christopher Baines
3849bb79c5 Support filtering by systems and targets when comparing derivations 2019-08-06 10:22:57 +01:00
Christopher Baines
36a16d356f Improve derivation comparison to show more changes 
In cases where the version is changed for example, the relevant derivations
will now show up, whereas previously they did not.
 2019-08-06 09:55:03 +01:00
Christopher Baines
82c3e8942b Only show the Compare button when it'll work 
On the branch page.
 2019-08-05 21:24:07 +01:00
Christopher Baines
9e80bda4f9 Avoid erroring when processing emails again 
These changes allow processing emails again, and just creating job and branch
entries where data is missing.
 2019-08-05 20:59:48 +01:00
Christopher Baines
712dd377a1 Remove test data after the tests have ran 
The load-new-guix-revision tests aren't transactional, so truncate some tables
after running the tests.
 2019-08-05 20:45:40 +01:00
Christopher Baines
f553a7490d Change the git_branches table primary key 
To allow for having branches in multiple git repositories.
 2019-08-05 20:02:06 +01:00
Christopher Baines
4b59108fdd Properly handle not finding a matching derivation 2019-08-05 19:45:10 +01:00
Christopher Baines
70126f7ed0 Set the Vary header to a sensible default value 
As most pages vary based on the Accept header.
 2019-08-05 19:43:50 +01:00
Christopher Baines
8f7398b814 Tweak the branch table layout 2019-08-05 11:36:57 +01:00
Christopher Baines
002ae7685c Clear the previous log entry when starting a job 2019-08-04 09:42:21 +01:00
Christopher Baines
84197686ce Fix some duplicated values in tables 
The licenses table, along with the package_metadata table had duplicate
values. This could happen as the unique constraints on those tables didn't
properly account for the nullable fields.

The duplicates in those tables also affected the license_sets, packages,
package_derivations tables in a similar way. Finally, the
guix_revision_package_derivations table was also affected.

This commit adds a migration to fix the data, as well as the constraints. THe
code to populate the licenses and package_metadata tables is also updated.
 2019-08-04 09:39:40 +01:00
Christopher Baines
1724bc485f Link to the comparison page from the branch page 
To allow easily comparing revisions.
 2019-07-29 21:11:45 +01:00
Christopher Baines
ef123dcded Link to the package pages from the comparison page 2019-07-29 21:10:41 +01:00
Christopher Baines
91e0527b4d Link to cgit on the comparison page 2019-07-29 20:40:56 +01:00
Christopher Baines
b075a71d1e Add a function to get the cgit URL bases for revisions 
To enable adding a link to cgit to the comparison page.
 2019-07-29 20:39:57 +01:00
Christopher Baines
f2c7cc83ed Fix the JSON compare output 2019-07-29 19:48:57 +01:00
Christopher Baines
08d9cbbfe3 Neaten up how branches in Git repositories are linked to 2019-07-27 09:38:50 +01:00
Christopher Baines
a2caa3ea30 Link to the Statistics and Jobs pages from the index page 2019-07-27 09:38:36 +01:00
Christopher Baines
75226a1499 Fix the latest-processed-revision pages 2019-07-27 09:12:25 +01:00
Christopher Baines
7f8ffb4bea Show links to the cgit pages for repositories on the index page 2019-07-22 21:11:09 +01:00
Christopher Baines
05db0a87f9 Switch the index page to show branches for a repository 
As that's probably more useful than recent revisions and jobs.
 2019-07-22 20:51:54 +01:00
Christopher Baines
c23822e14d Extract out the branches with most recent commits table 
As I'm thinking about using this on the index page.
 2019-07-22 20:36:40 +01:00
Christopher Baines
219b1fd4ad Extract out view-revision/jobs-and-events 
This is common to both view-revision and unknown-revision.
 2019-07-22 20:29:38 +01:00
Christopher Baines
7a70e9ca6f Extract out view-revision/git-repositories 
As this was duplicated in the functions for viewing known and unknown
revisions.
 2019-07-22 20:29:29 +01:00
Christopher Baines
91be72df03 Display the jobs related to a revision on the revision page 2019-07-22 20:00:11 +01:00
Christopher Baines
1f1deac296 Link from a revision to the git repository page 2019-07-19 21:29:56 +01:00
Christopher Baines
1f977f6c12 Improve how repositories and branches are handled 
Make the link between repositories and branches clearer, replacing the
/branches and /branch pages by /repository/ and /repository/*/branch/* pages.
 2019-07-19 21:22:15 +01:00
Christopher Baines
6dd52f08ed Remove restrictions in select-job-for-update 
This allows running jobs manually that have failed.
 2019-07-19 20:25:53 +01:00
Christopher Baines
71607eff66 Close the logging port once it's been used 2019-07-19 20:25:13 +01:00
Christopher Baines
dab984b1d5 Fix match expression 
Specify 'GET rather than GET, to actually match the method, rather than
binding it to GET.
 2019-07-19 20:20:52 +01:00
Christopher Baines
fac3db2d14 Remove localhost from the Sqitch config 
This means it'll connect over the socket, like the application.
 2019-07-16 18:20:56 +01:00
Christopher Baines
413ef9ea6e Prioritise processing the latest revisions for branches 
Reserve some capacity to process revisions which are the tip of a branch. This
should reduce the time between new revisions appearing, and then being
processed.
 2019-07-16 18:20:02 +01:00
Christopher Baines
6904547bf2 Record jobs as failed when they are killed 
So that they aren't retried again and again.
 2019-07-14 14:43:47 +01:00
Christopher Baines
3dfa9212f1 Kill long running load new revision jobs 
There are some revisions of Guix which take forever to process (or days at
least). To avoid jobs being processed forever, kill them after they've been
running for a while (default 24 hours).
 2019-07-12 23:00:44 +01:00
Christopher Baines
83ef624b97 Switch to processing jobs in parallel 
This should speed up processing new revisions, reduce latency between finding
out about new revisions and processing them, as well as help manage memory
usage, by processing each job in a process that then exits.
 2019-07-12 23:00:44 +01:00
Christopher Baines
09d927cb99 Add a process-job command 
This allows easily processing an individual job by id. This may be useful to
use manually, but also when processing jobs in parallel, as forking doesn't
work well with the libpq library used by squee.
 2019-07-12 19:57:24 +01:00
Christopher Baines
a6bd46b829 Obtain a lock when inserting the data for a new revision 
This is working towards running the jobs in parallel. Each job looks at the
records in the database, and adds missing ones. If other jobs, running in
different transactions insert the same missing records at the same time, this
could cause an error.

Therefore, to just avoid this problem, lock before inserting the data. This
will allow the jobs to be processed in parallel, and it shouldn't have too
much of an effect on performance, as the slow bit is outside of the
transaction.
 2019-07-12 19:57:13 +01:00