Commit Graph

59 Commits

Author SHA1 Message Date
Christopher Baines 10bad53ad5 Support polling git repositories for new branches/revisions
This is mostly a workaround for the occasional problems with the guix-commits
mailing list, as it can break and then the data service doesn't learn about
new revisions until the problem is fixed.

I think it's still a generally good feature though, and allows deploying the
data service without it consuming emails to learn about new revisions, and is
a step towards integrating some kind of way of notifying the data service to
poll.
2023-10-09 22:19:02 +01:00
Christopher Baines 1fb291be40 Add support for incrementally tracking blocked builds
This will hopefully provide a less expensive way of finding out if a scheduled
build is probably blocked by other builds failing or being canceled.

By working this out when the build events are recieved, it should be more
feasible to include information about whether builds are likely blocked or not
in various places (e.g. revision comparisons).
2022-11-11 10:35:09 +00:00
Christopher Baines 64be52844e Partition the package_derivations_by_guix_revision_range table
And create a proper git_branches table in the process.

I'm hoping this will help with slow deletions from the
package_derivations_by_guix_revision_range table in the case where there are
lots of branches, since it'll separate the data for one branch from another.

These migrations will remove the existing data, so
rebuild-package-derivations-table will currently need manually running to
regenerate it.
2022-05-23 19:10:25 +01:00
Christopher Baines 869d2522e5 Add the system module to Makefile.am 2021-04-23 18:07:55 +01:00
Christopher Baines 663bd1411a Add a /package/NAME page
This might be useful for working out when a non-master branch contains a newer
version of a package, or someone has sent in a patch for a newer version
already.
2021-03-14 21:48:43 +00:00
Christopher Baines c596a1c6a9 Add a Prometheus metrics page, with some database metrics
The database size is growing, but it's hard to know what parts are growing the
fastest. These metrics will hopefully help with understanding that.
2020-09-06 13:14:31 +01:00
Christopher Baines 6baef6ae25 Split out querying of build servers and substitute servers
These are related things, but somewhat separate. This change should make it
easier to deal with changes regarding querying build servers, and querying
substitute servers.
2020-05-03 13:23:43 +01:00
Christopher Baines d12909d572 Fix some tab issues in the Makefile.am file 2020-03-20 20:43:02 +00:00
Christopher Baines 77caafb019 Add scripts for generating database dumps 2020-03-02 21:44:29 +00:00
Christopher Baines cb353d2c82 Make the test output more verbose 2020-02-27 21:11:27 +00:00
Christopher Baines 49d10cfe14 Add a new utils module, containing a with-time-logging syntax rule 2020-02-24 21:31:01 +00:00
Christopher Baines 510e13f175 Add a page listing the available database dumps 2020-02-22 23:24:24 +00:00
Christopher Baines 27904e8fd7 Fix some comments
Copied over from the Mumi code.
2020-02-16 22:29:11 +00:00
Christopher Baines 773e5a9c38 Add a module to handle deleting data
This, along with the way of specifying which branches are processed is a way
to manage the data stored within the Guix Data Service.

This only goes so far, it doesn't delete derivations, but it does delete some
of the information related to a revision.
2020-02-15 11:36:31 +00:00
Christopher Baines 9be7dbac0b Start storing channel instance derivations
These are the ones that relate to Guix pull.
2020-02-11 08:56:24 +00:00
Christopher Baines 8f0d5b51b1 Store information about system tests 2020-02-03 09:57:02 +01:00
Christopher Baines 485b79a6de Add a page to show the public key used to sign substitutes 2019-12-27 13:03:17 +00:00
Christopher Baines a8d58e0541 Sort the web modules in the SOURCES list in Makefile.am 2019-12-26 09:28:12 +00:00
Christopher Baines f78e3126e5 Render nar files for derivations
In the same manor that Guix publish does. This is working towards being able
to serve substitutes for derivations.
2019-12-26 09:25:27 +00:00
Christopher Baines 639f24bb22 Extract management of package_derivations_by_guix_revision_range
... entries in to a separate module, to split the code up a little further.
2019-12-22 16:09:32 +00:00
Christopher Baines 2d87bd6340 Add a page for each signing key 2019-12-12 20:07:22 +00:00
Christopher Baines 8ff27eaa14 Add a new html-utils module
To store code used for rendering HTML across multiple controllers.
2019-12-12 20:07:22 +00:00
Christopher Baines b6194e7b3d Begin to add support for importing narinfo files
This commit adds the tables, as well as code to support extracting data from
narinfo files.
2019-11-30 10:51:58 +00:00
Christopher Baines 32e1fee7ea Add the build-server-token-seed.scm file to Makefile.am 2019-11-26 07:17:09 +00:00
Christopher Baines 2a3ee20a99 Extract the builds page in to it's own set of modules
To allow for expanding it, without cluttering the root controller.
2019-11-24 20:42:37 +00:00
Christopher Baines 5663235048 Rework the builds and build_status tables as well as related code
Allow for build status information to be submitted by POST request. This
required some changes to the builds and build_status tables, as for example,
the Cuirass build id may not be available, and the derivation may not be know
yet, so just record the derivation file name.
2019-11-24 20:18:08 +00:00
Christopher Baines 0ffd8caeeb Add a script to help manage build servers
This computes and displays the tokens needed to send build events to the Guix
Data Service.
2019-11-23 11:50:51 +00:00
Christopher Baines 3ef99acc79 Add a new channel-news module, along with tables the relevant data 2019-11-21 19:56:19 +00:00
Christopher Baines aee903bc8c Add missing files to the SOURCES list in Makefile.am 2019-10-18 18:44:18 +01:00
Christopher Baines 1ea0c9ba26 Sort the SOURCES in the Makefile.am 2019-10-18 18:13:34 +01:00
Christopher Baines c9c7666b49 Don't automatically shutdown the test database
As this leaves a process running, it's better just to stop the database
manually.
2019-10-11 08:43:21 +01:00
Christopher Baines 06af9ce6de Fix the environment variables in the check-with-tmp-database target
So that they're actually set for sqitch.
2019-10-11 08:36:31 +01:00
Christopher Baines eb1709c8e4 Improve the check-with-tmp-database target
To better manage the environment, and stop the database upon completion of the
tests.
2019-10-11 08:32:48 +01:00
Christopher Baines c40747a5ee Add a check-with-tmp-database target
This is aimed at testing using pg_tmp, to create a temporary test
database (and instance of PostgreSQL). This is both useful for testing all the
migrations, but should also be useful for running the tests within the guix
package.
2019-10-08 19:45:59 +01:00
Christopher Baines 44c0c9d0dd Remove non-existent test from Makefile.am 2019-10-01 19:03:17 +01:00
Christopher Baines ea253c4515 Add a new script to process emails in an mbox file
This can be used with the mbox files for the guix-commits mailing list to add
older emails in to the database.
2019-09-28 23:55:11 +01:00
Christopher Baines f5ff0f8638 Hack building a HTML version of the README in to Makefile.am
This is to be able to serve a readable version of the README on the site,
allowing those without Emacs to more easily read it.
2019-09-21 12:40:46 +01:00
Christopher Baines 34ed73cdc1 Fix the godir in Makefile.am
I think the .go files should be in lib, rather than share.
2019-09-21 11:30:25 +01:00
Christopher Baines 1a55022524 Improve the package and package-metadata modules
Add tests around the package module, extract out the use of the
inferior-package record assessors so that they aren't part of the tests, and
switch across the package-metadata module to use
insert-missing-data-and-return-all-ids.
2019-09-05 16:07:23 +02:00
Christopher Baines 6b9977f62e Store lint warnings in the database
This commit adds the relevant tables and code to store lint warnings in the
database.

Currently, only lint checkers which don't require access to the network will
be run, as this allows the processing to happen without network access. Also,
this functionality won't work in older versions of Guix which don't expose the
lint warnings in a compatible way.
2019-09-01 13:12:10 +01:00
Christopher Baines 09d927cb99 Add a process-job command
This allows easily processing an individual job by id. This may be useful to
use manually, but also when processing jobs in parallel, as forking doesn't
work well with the libpq library used by squee.
2019-07-12 19:57:24 +01:00
Christopher Baines 88922d9f11 Tweak the Makefile.am to install the right things
Install the assets and sqitch files, as they are needed. Remove the test
related sources.
2019-06-08 10:27:07 +01:00
Christopher Baines 544dc1558f Add a new test to cover some of the loading new revisions code 2019-06-02 22:02:12 +01:00
Christopher Baines f03a750705 Add some missing test files to Makefile.am 2019-06-02 16:19:06 +01:00
Christopher Baines 16799a34a9 Store license information for packages
And display this on the package page.

This uses a couple of new tables, and an additional field in the
package_metadata table.

Currently, the order of the licenses in the package definition isn't stored,
as I'm not sure the order in the list is significant.
2019-05-15 08:05:14 +01:00
Christopher Baines 7341d17254 Store and display the location of packages
Store the location a package can be found at, and display this on the package
page.

If available, link off to the git repository containing the package.
2019-05-13 21:07:48 +01:00
Christopher Baines 8eac26b17d Remove the sha1_hash from the package_metadata table
I'm thinking about adding more fields to this table, and the sha1_hash values
will make this tricker.

Therefore, remove the value, and adjust the existing code to cope. This commit
also adds a new test which coveres some of the changed functionality.
2019-05-12 18:08:17 +01:00
Christopher Baines 0ab1c71722 Add a mock-inferior module
This to help test code which uses the (guix inferior) module.
2019-05-12 17:32:37 +01:00
Christopher Baines d212682061 Add a (tests driver) module
Using the test driver from build-aux/test-driver.scm, as this shows the test
errors when tests fail.
2019-05-12 17:32:21 +01:00
Christopher Baines f2e123b7ac Add a new query-parameters module
The query parameters feed in to the results shown, but also forms on
pages. Validation is important to avoid errors and security issues, but it's
also important to provide appropriate feedback to the user.

This module provides some utilities and structure around handling query
parameters.
2019-05-11 16:45:59 +01:00