2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/data-service.git synced 2023-12-14 03:23:03 +01:00
Commit graph

64 commits

Author SHA1 Message Date
Christopher Baines
375a6a37dc Support not querying pending builds
As this can take some time.
2020-11-01 22:52:53 +00:00
Christopher Baines
f485423d5a Allow only fetching builds for a specific system 2020-11-01 22:49:49 +00:00
Christopher Baines
6a7f6b5a0e Fix create small backup issue with latest_build_status 2020-10-23 20:01:43 +01:00
Christopher Baines
3225766207 Make it easier to get to a repl 2020-10-10 13:44:37 +01:00
Christopher Baines
18b6dd9e6d Stop opening a PostgreSQL connection per request
This was good in that it avoided having to deal with long running connections,
but it probably takes some time to open the connection, and these changes are
a step towards offloading the PostgreSQL queries to other threads, so they
don't block the threads for fibers.
2020-10-03 09:22:29 +01:00
Christopher Baines
39b5df04eb Remove development code from the process job script 2020-09-28 08:29:20 +01:00
Christopher Baines
033858410b Add a JSON page for repository branches 2020-09-27 16:32:56 +01:00
Christopher Baines
fb180e1ebd Replace debug-set! with setenv COLUMNS
As that actually seems to work.
2020-09-26 16:42:18 +01:00
Christopher Baines
53341c70fc Change the locale codeset representation
From the normalized one, to the one actually contained within glibc. Recent
versions of glibc also contain symlinks linking the normalized codeset to the
locales with the .UTF-8 ending, but older ones do not.

Maybe handling codeset normalisation for queries would be good, but the locale
values ending in .UTF-8 are more compatible and allow the code to be
simplified. For querying, maybe there should be a locales table which handles
different representations.
2020-09-26 11:45:57 +01:00
Christopher Baines
e38db9eed9 Set the locale at the start of the process jobs script
This might help with the odd [1] errors regarding PostgreSQL queries.

1: invalid byte sequence for encoding "UTF8":
2020-09-20 11:11:03 +01:00
Christopher Baines
a0e098a6ce Increase the stack trace width when processing jobs
As this might result in more useful error messages.
2020-09-20 10:59:22 +01:00
Christopher Baines
b6754c8a4c Add a lookup_builds field to the build_servers table
This is to allow for build servers where only the substitutes should be
queried, and it shouldn't be assumed that they're running Cuirass.
2020-05-24 17:02:53 +01:00
Christopher Baines
9c72fc23dc Move around --no-tablespaces
Turns out, at the moment, this is ineffective when combined with the archive
formats, like the custom format in use. Therefore, move it to the pg_restore
command, where hopefully it'll work.
2020-05-16 08:42:00 +01:00
Christopher Baines
796c129a36 Don't include tablespace assignments in the backup dump
This is a comprimise, as this won't help restoring the backup in situations
you want tablespaces, but I'm currently viewing tablespaces as a deployment
concern, so maybe the right thing to do is exclude them. This approach will at
least keep the same behaviour in terms of restoring the backups locally.

This will fix the small dump creation process on data.guix.gnu.org, which is
currently broken because of the tablespace assignments when trying to restore
the backups.
2020-05-14 20:49:46 +01:00
Christopher Baines
6baef6ae25 Split out querying of build servers and substitute servers
These are related things, but somewhat separate. This change should make it
easier to deal with changes regarding querying build servers, and querying
substitute servers.
2020-05-03 13:23:43 +01:00
Christopher Baines
a0263a0eae Set a statement timeout of 60 seconds for web requests
This will help stop queries running for an unnecessarily long time, longer
than NGinx will wait for example.
2020-04-24 09:00:20 +01:00
Christopher Baines
5081a64c1f Rebuild the package derivation ranges table for the small backup
This is better than just deleting the entries that don't match up with the
remaining revisions, but also not very useful for local development (due to
the lack of data).
2020-03-31 20:46:18 +01:00
Christopher Baines
d1c243f7fd Give the temporary database more working memory
In the hope that this makes the script faster.
2020-03-26 20:21:47 +00:00
Christopher Baines
3017765f0c Use EXPLAIN ANALYZE for the creation of tmp_derivations
In the create-small-backup script, as this is quite a slow part, it's useful
to get more information.
2020-03-26 20:21:14 +00:00
Christopher Baines
9a79a5d747 Handle a couple more tables in create-small-backup
derivation_output_details_sets, and derivations_by_output_details_set. This
required moving around some of the code.
2020-03-26 20:20:29 +00:00
Christopher Baines
d0eff9da5d Use the --no-comments option to pg_dump
Hopefully this will help with the pg_restore in the create-small-backup
script:

  pg_restore: [archiver (db)] Error while PROCESSING TOC:
  pg_restore: [archiver (db)] Error from TOC entry 2875; 0 0 COMMENT EXTENSION plpgsql
  pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
      Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
2020-03-25 20:47:53 +00:00
Christopher Baines
8af7130239 Handle channel instances in create-small-backup
Otherwise this table is empty.
2020-03-25 18:27:01 +00:00
Christopher Baines
b99854924a Handle system test derivations in create-small-backup
Otherwise this table is empty.
2020-03-25 18:26:33 +00:00
Christopher Baines
ca0d3ee754 Stop using package_versions_by_guix_revision_range
It's been replaced by the package_derivations_by_guix_revision_range table.
2020-03-24 20:44:57 +00:00
Christopher Baines
cf4082dbeb Avoid failures related to renice and ionice
These parts of the backup scripts are optional, so don't fail if they don't
work.
2020-03-20 20:40:33 +00:00
Christopher Baines
ded4df6632 Move and improve the "starting the server" message
Move it after the output relating to narinfo signing, and include the host.
2020-03-14 13:14:29 +00:00
Christopher Baines
a03e1601de Improve handling of errors
Adjust the previously unused error page code, and start to use it. Only show
the error if configured to do so, to avoid leaking secret information.
2020-03-14 12:46:02 +00:00
Christopher Baines
baeae56de4 Don't use TRUNCATE CASCADE in the create small backup script
As it makes it clearer what tables will be truncated.
2020-03-13 18:38:42 +00:00
Christopher Baines
6ce96ad55b Trip the derivation output details table in the small data dump 2020-03-13 18:38:29 +00:00
Christopher Baines
b64e6b19c2 Trim derivation source file tables in the small data dump 2020-03-13 18:37:46 +00:00
Christopher Baines
77caafb019 Add scripts for generating database dumps 2020-03-02 21:44:29 +00:00
Christopher Baines
65f2f21d3a Support customising the latest branch revision max processes
This makes it possible to set a higher or lower value depending on what you
want.
2020-02-28 20:58:21 +00:00
Christopher Baines
9c7310f8e3 Make it possible to query builds servers for specific outputs 2020-02-16 09:54:27 +00:00
Christopher Baines
33749786e4 Add verbose output to the query-build-servers script 2020-02-15 19:54:42 +00:00
Christopher Baines
381acf312e Warn if process-branch-updated-mbox won't match any emails 2020-02-01 14:03:26 +01:00
Christopher Baines
7342280ead Guard against failures when loading the narinfo signing public key 2019-12-26 10:49:14 +00:00
Christopher Baines
54b3d8d93a Fix the option alist element names for the narinfo signing key 2019-12-26 10:47:29 +00:00
Christopher Baines
66e886a6b4 Serve narinfo files for derivations 2019-12-26 09:34:43 +00:00
Christopher Baines
96305ec406 Fix the query-build-servers script when no commits are provided 2019-12-23 19:27:23 +00:00
Christopher Baines
c516154816 Fix the default max-processes for the process-jobs script 2019-12-22 23:26:45 +00:00
Christopher Baines
c229ace8a5 Allow customising the maximum number of processes 2019-12-16 19:57:56 +00:00
Christopher Baines
c90f411e5e Add options to the query-build-servers script
So you can select to query specific build servers.
2019-12-12 20:07:22 +00:00
Christopher Baines
7cc5c02cdd Query build servers for narinfo files
As well as making it possible to only query build servers for builds relating
to a specific revision.
2019-11-30 10:59:16 +00:00
Christopher Baines
5663235048 Rework the builds and build_status tables as well as related code
Allow for build status information to be submitted by POST request. This
required some changes to the builds and build_status tables, as for example,
the Cuirass build id may not be available, and the derivation may not be know
yet, so just record the derivation file name.
2019-11-24 20:18:08 +00:00
Christopher Baines
0ffd8caeeb Add a script to help manage build servers
This computes and displays the tokens needed to send build events to the Guix
Data Service.
2019-11-23 11:50:51 +00:00
Christopher Baines
977609c66e Rewrite localhost for sqitch
As this means that like psql, the unix socket will be used.
2019-10-01 19:03:34 +01:00
Christopher Baines
ea253c4515 Add a new script to process emails in an mbox file
This can be used with the mbox files for the guix-commits mailing list to add
older emails in to the database.
2019-09-28 23:55:11 +01:00
Christopher Baines
759b75257e Switch to processing emails as bytevectors
This is better, as different parts of the email might be encoded differently,
and guile-email will take care of this if handed a bytevector.
2019-09-26 18:08:12 +01:00
Christopher Baines
d35cc85d52 Improve error handling for processing emails 2019-08-31 12:06:36 +01:00
Christopher Baines
09d927cb99 Add a process-job command
This allows easily processing an individual job by id. This may be useful to
use manually, but also when processing jobs in parallel, as forking doesn't
work well with the libpq library used by squee.
2019-07-12 19:57:24 +01:00