Commit Graph

811 Commits

Author SHA1 Message Date
Christopher Baines 7db6a8fa03 Guard against errors when inserting log data
PostgreSQL seems to be unhappy with some data, I guess because it gets a bit
jumbled. Rather than failing the job, or getting stuck not inserting logs, try
and capture the error, log the details, and then keep going.
2020-04-07 17:38:57 +01:00
Christopher Baines 8e9f1af89b Use the config options when creating the default paramstring 2020-04-07 17:37:03 +01:00
Christopher Baines 04bc3af53a Remove some done items from the README
The derivations for channel instances (guix pull) and system tests are now
captured.
2020-04-05 21:09:23 +01:00
Christopher Baines 3285080da5 Fix another typo in the README 2020-04-05 21:09:15 +01:00
Vincent Legoll b2ec547434 Fix typos in README
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-04-05 21:07:35 +01:00
Danjela lura e514471fc9 Add option to change language of lint messages for the revision page
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-04-04 18:31:26 +01:00
Christopher Baines a50bc3342f Fix the new package-versions-for-branch query
This recent change simply didn't work, the ordering was bad and the window
function wasn't properly defined. It now should hopefully work, although
there's an interesting case where different versions are available for
different systems/targets, which isn't handled particularly well.
2020-03-31 20:47:53 +01:00
Christopher Baines 5081a64c1f Rebuild the package derivation ranges table for the small backup
This is better than just deleting the entries that don't match up with the
remaining revisions, but also not very useful for local development (due to
the lack of data).
2020-03-31 20:46:18 +01:00
Christopher Baines 94030b736c Improve the insert-guix-revision-package-derivation-entries query
Somewhat untested improvements, but these make the query a bit more rigorous
in the case of multiple branches and git repositories.
2020-03-31 20:43:34 +01:00
Christopher Baines 63d7af9340 Improve filtering for no target
Previously, the select option label was empty that's not particularly
informative. These changes also fix the next page link behaviour for the
target parameter.
2020-03-31 18:58:36 +01:00
Danjela lura 6ee9a674cd Store output of most-recent-commits-for-branch in a local variable
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-03-29 21:12:35 +01:00
Christopher Baines 776d0462e9 Fix a missing ; in a Sqitch migration 2020-03-29 12:05:00 +01:00
Christopher Baines 8d109bccb5 Handle NULL better insert-missing-data-and-return-all-ids queries
Actually check if fields can be NULL, and if they can be then include some
extra conditions for the comparison.

This will at least make the queries smaller, I'm not sure if it will have an
effect on performance.
2020-03-29 10:29:46 +01:00
Christopher Baines bddf7a4240 Add a function to fetch information about a table
Mostly so whether a field can contain NULL values can be determined.
2020-03-29 10:29:08 +01:00
Christopher Baines b30904e1af Make jobs unique by commit
Given that a commit, regardless of what repository it comes from should
contain the same exactly the same data, just track jobs by commit.
2020-03-28 19:59:35 +00:00
spf50 f980b6c2ac Include data in JSON response for the branch page
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-03-28 14:15:00 +00:00
Christopher Baines d1c243f7fd Give the temporary database more working memory
In the hope that this makes the script faster.
2020-03-26 20:21:47 +00:00
Christopher Baines 3017765f0c Use EXPLAIN ANALYZE for the creation of tmp_derivations
In the create-small-backup script, as this is quite a slow part, it's useful
to get more information.
2020-03-26 20:21:14 +00:00
Christopher Baines 9a79a5d747 Handle a couple more tables in create-small-backup
derivation_output_details_sets, and derivations_by_output_details_set. This
required moving around some of the code.
2020-03-26 20:20:29 +00:00
Christopher Baines 00bc6535f9 Add an unimplemented JSON response for the branch page 2020-03-26 09:14:00 +00:00
Christopher Baines d0eff9da5d Use the --no-comments option to pg_dump
Hopefully this will help with the pg_restore in the create-small-backup
script:

  pg_restore: [archiver (db)] Error while PROCESSING TOC:
  pg_restore: [archiver (db)] Error from TOC entry 2875; 0 0 COMMENT EXTENSION plpgsql
  pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
      Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
2020-03-25 20:47:53 +00:00
Christopher Baines 0c4e6a2fe4 Create an index on narinfo_fetch_records
This greatly improves the performance of the derivation-outputs page.
2020-03-25 19:11:03 +00:00
Christopher Baines 8af7130239 Handle channel instances in create-small-backup
Otherwise this table is empty.
2020-03-25 18:27:01 +00:00
Christopher Baines b99854924a Handle system test derivations in create-small-backup
Otherwise this table is empty.
2020-03-25 18:26:33 +00:00
Christopher Baines 60df7ecbf9 Remove redundant DISTINCT from the derivation range INSERT query
This was redundant and slow, so don't do it.
2020-03-24 21:37:18 +00:00
Christopher Baines 994318782a Pass #:store to checkers that support it
The derivation checker currently opens a store connection on its own, but by
passing the already open connection in, it won't have to do that, and
hopefully this will make checking all the packages faster.
2020-03-24 21:36:13 +00:00
Christopher Baines ca0d3ee754 Stop using package_versions_by_guix_revision_range
It's been replaced by the package_derivations_by_guix_revision_range table.
2020-03-24 20:44:57 +00:00
Christopher Baines 102f7a0536 Implement version history using the derivations table
Rather than having two big tables looking at the history, just use the
derivations table as it has all the information.

This will allow deleting the package_versions_by_guix_revision_range table
which should help save time when importing revisions, and reduce the size of
the database.
2020-03-24 20:17:18 +00:00
Christopher Baines 96bf658dfc Fix the date ordering on the package output history page 2020-03-24 20:17:00 +00:00
Christopher Baines f87583016e Sort the lint-warning-message-ids when querying
This might solve errors where the Guix Data Service is trying to insert a
lint_warning_message_set that already exists.
2020-03-23 09:13:14 +00:00
Christopher Baines e18f3861f8 Say that there are no lint warnings
Rather than having an empty table.
2020-03-21 11:01:16 +00:00
Christopher Baines e108ed34e2 Change the sort order for derivations on the package version page
Put the non-cross built derivations first.
2020-03-21 10:54:57 +00:00
Christopher Baines 7d2309d344 Add a page for package output history on a branch
This complements the existing pages for the version history, and derivation
history. As well as the new page, the buttons and styling of the two existing
pages has been made to match better to enable easier navigation between the
pages.
2020-03-21 10:38:20 +00:00
Christopher Baines f4583e5fe6 Add a new function to get the outputs for a package on a branch
Similar to the one above for derivations, this just looks at outputs. This
filters out equivalent derivations, which can be useful.
2020-03-21 10:37:22 +00:00
Christopher Baines d12909d572 Fix some tab issues in the Makefile.am file 2020-03-20 20:43:02 +00:00
Christopher Baines cf4082dbeb Avoid failures related to renice and ionice
These parts of the backup scripts are optional, so don't fail if they don't
work.
2020-03-20 20:40:33 +00:00
Christopher Baines c44297b615 Generate and store system test derivations for all supported systems
Rather than just the native system. I'm not quite sure of the value here, as I
guess system tests should behave the same regardless of the way the software
is compiled, but this seems like it could be useful, and being explicit about
the system the derivation is for is good.
2020-03-20 08:28:55 +00:00
Christopher Baines e0f920bb14 Fix marking jobs as failed when exceptions are raised
The switch away from catch broke this, I obviously still don't quite get how
with-exception-handler works. Therefore, use it twice as that seems to help.
2020-03-19 20:43:37 +00:00
Christopher Baines ded4df6632 Move and improve the "starting the server" message
Move it after the output relating to narinfo signing, and include the host.
2020-03-14 13:14:29 +00:00
Christopher Baines 865e856e8f Provide more information if the server can't bind the port
As this could be a common problem.
2020-03-14 13:13:48 +00:00
Christopher Baines ebb8ab8066 Improve the 404 pages
Render some HTML rather than the plain response.
2020-03-14 12:55:34 +00:00
Christopher Baines a03e1601de Improve handling of errors
Adjust the previously unused error page code, and start to use it. Only show
the error if configured to do so, to avoid leaking secret information.
2020-03-14 12:46:02 +00:00
Christopher Baines 33958eac79 Increase the batch size for fetching builds/narinfo files
To increase the likelyhood that all the builds and narinfo files for the
latest revisions are fetched.
2020-03-14 12:35:34 +00:00
Christopher Baines 6e81acf0a5 Set an order for select-outputs-without-known-nar-entries
Pulling out the recent entries first.
2020-03-14 12:35:34 +00:00
Christopher Baines baeae56de4 Don't use TRUNCATE CASCADE in the create small backup script
As it makes it clearer what tables will be truncated.
2020-03-13 18:38:42 +00:00
Christopher Baines 6ce96ad55b Trip the derivation output details table in the small data dump 2020-03-13 18:38:29 +00:00
Christopher Baines b64e6b19c2 Trim derivation source file tables in the small data dump 2020-03-13 18:37:46 +00:00
Christopher Baines 24e7df541b Query for more recent derivation outputs by default
Rather than looking for the oldest unknown outputs, as the new ones are
generally more useful.
2020-03-12 20:47:43 +00:00
Christopher Baines 885053e107 Remove peek call that snuck in to the builds module 2020-03-02 21:48:14 +00:00
Christopher Baines 77caafb019 Add scripts for generating database dumps 2020-03-02 21:44:29 +00:00