2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/guix-cuirass.git synced 2023-12-14 06:03:04 +01:00
Commit graph

120 commits

Author SHA1 Message Date
Mathieu Othacehe 0ffcb80ebb
Optimize database queries.
Avoid full scans of Builds table that can be very time consuming by rewriting
some queries and using new indexes.

* src/sql/upgrade-12.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_evaluation_index, Evaluations_status_index,
Evaluations_specification_index): New indexes.
* src/cuirass/database.scm (db-get-evaluations-build-summary,
db-get-evaluation-summary): Rewrite queries to avoid full Builds table scan
and use the new indexes.
2020-09-28 17:49:55 +02:00
Mathieu Othacehe b310f17aaf
Add SQL query logging support.
* bin/cuirass.in (show-help): Document "--log-queries" option.
(%options): Add it.
(main): Enable query logging if the above option is set.
* src/cuirass/database.scm (db-log-queries): New procedure.
* src/cuirass/logging.scm (query-logging-port): New parameter.
(log-query): New procedure.
2020-09-24 10:24:02 +02:00
Mathieu Othacehe 43ed85479f
database: Export procedures.
* src/cuirass/database.scm (last-insert-rowid, expect-one-row, %sqlite-exec):
Export them.
2020-09-14 14:32:07 +02:00
Mathieu Othacehe f5b0d39328
Add a status field to Evaluation table.
The Evaluation table currently has an 'in_progress' field. Distinction between
succeeded and failed evaluations are based on the presence of Builds records
for the evaluation. It it also not possible to distinguish aborted evaluations
from failed evaluations.

Rename 'in_progress' field to 'status'. The 'status' field can be equal to
'started', 'succeeded', 'failed' or 'aborted'.

* src/cuirass/database.scm (evaluation-status): New exported enumeration.
(db-set-evaluations-done, db-set-evaluation-done): Remove them.
(db-abort-pending-evaluations, db-set-evaluation-status): New exported procedures.
(db-add-evaluation, db-get-builds, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluation-summary): Adapt to use
'status' field instead of 'in_progress' field.
* src/cuirass/templates.scm (evaluation-badges): Ditto.
* src/schema.sql (Evaluations): Rename 'in_progress' field to 'status'.
* src/sql/upgrade-10.sql: New file.
* bin/cuirass.in (main): Use "db-abort-pending-evaluations" instead of
"db-set-evaluations-done".
* src/cuirass/base.scm (evaluate): Use "db-set-evaluation-status" instead of
"db-set-evaluations-done".
(build-packages): Use "db-set-evaluation-status" instead of
"db-set-evaluation-done".
* tests/database.scm (sqlite-exec): Adapt accordingly.
* tests/http.scm (evaluations-query-result): Ditto.
2020-09-10 15:16:15 +02:00
Mathieu Othacehe d9879583af
Display evaluation date and duration.
src/cuirass/database.scm (db-get-evaluations): Add support for "timestamp",
"checkouttime" and "evaltime" fields,
(db-get-evaluation-summary): ditto.
src/cuirass/templates.scm (nearest-exact-integer, seconds->string): New
procedures,
(evaluation-build-table): print evaluation date and duration.
2020-09-06 14:23:01 +02:00
Mathieu Othacehe 154232bc76
Save evaluations and checkouts timestamps.
src/cuirass/base.scm (fetch-input): Add the commit timestamp to the returned
association list,
(process-specs): Pass a timestamp taken at procedure start and another one
taken after inputs are fetched to "db-add-evaluation" procedure. Once the
evaluation is over, call "db-set-evaluation-time" to save the evaluation
completion time.
src/cuirass/database.scm (db-set-evaluation-time): New procedure,
(db-add-checkout): Handle the "timestamp" field,
(db-add-evaluation): add "checkouttime" and "evaltime" arguments. Modify the
associated SQL query accordingly.
(db-get-builds): Use "Builds.timestamp" instead of "timestamp" as this field
is also part of the Evaluations table.
src/schema.sql (Checkouts): Add "timestamp" field,
(Evaluations): add "timestamp", "checkouttime" and "evaltime" fields.
src/sql/upgrade-9.sql: New file.
tests/database.scm (sqlite-exec): Adapt Evaluations table insertions to include
"timestamp", "checkouttime" and "evaltime" required fields.
2020-09-06 13:03:08 +02:00
Mathieu Othacehe 153b49c952
database: Warn if the worker is busy for more than 5 seconds.
* src/cuirass/utils.scm (with-operation, get-message-with-timeout): New
procedures,
(call-with-worker-thread): add timeout and timeout-proc arguments.
* src/cuirass/database.scm (with-db-worker-thread): Pass a 5 seconds timeout
to call-with-worker-thread, and print a debug message on timeout expiration.
2020-08-01 13:18:38 +02:00
Mathieu Othacehe 17395e85d2
Fix spec reading when restarting builds.
When "spawn-builds" is called to restart builds, the spec is not known,
preventing build products from being created as reported here:

https://issues.guix.gnu.org/42523

Fix this issue by reading the specification in database in
"set-build-successful!" procedure.

* src/cuirass/database.scm (db-get-specification): New exported procedure,
(db-get-specifications): add an optional name argument.
* tests/database.scm (db-get-specification): Add a corresponding test-case.
* src/cuirass/base.scm (set-build-successful!): Remove spec argument and read
it directly from database instead,
(update-build-statuses!): also remove spec argument, adapt
set-build-successful! call accordingly,
(spawn-builds): remove spec argument and adapt handle-build-event and
update-build-statuses! calls accordingly,
(handle-build-event): remove spec argument, adapt
set-build-successful! call accordingly,
(build-packages): remove spec argument, adapt spawn-builds call accordingly,
(process-specs): adapt build-packages call.
2020-07-25 14:22:20 +02:00
Mathieu Othacehe 136a8295e4
Add "latest builds" support.
Add a "status" dropdown menu in the navigation bar. For now this menu only
contains one item, a link to the "latest builds" page at "/status" location.

* src/cuirass/database.scm (db-get-builds): Add support for 'started status.
* src/cuirass/http.scm (url-handler): Add "/status" route.
* src/cuirass/templates.scm (running-builds-table): New procedure,
(html-page): add a dropdown menu containing one item, a link to "/status"
route defined above.
* src/static/css/cuirass.css: Add new class to allow bootstrap dropdown menus
to work without javascript plugin.
2020-07-02 21:07:30 +02:00
Mathieu Othacehe 83e79f38c8
database: Ignore BuildProducts duplicates.
Ignore errors when trying to insert an already existing build product.

* src/cuirass/database.scm (db-add-build-product): Ignore insertion errors.
2020-06-30 15:02:06 +02:00
Mathieu Othacehe f44618fc79
Add support for build products downloading.
* src/sql/upgrade-7.sql: New file.
* Makefile.am: Add it.
* src/cuirass/base.scm (create-build-outputs): New procedure,
(build-packages): call it,
(process-spec): add the new spec argument and pass it to create-build-outputs.
* src/cuirass/database.scm (db-add-build-product, db-get-build-product-path,
db-get-build-products): New exported procedures.
* src/cuirass/http.scm (respond-static-file): Move file sending to ...
(respond-file): ... this new procedure,
(url-handler): add a new "download/<id>" route, serving the requested file
with the new respond-file procedure. Also gather build products and pass them
to "build-details" for "build/<id>/details" route.
* src/cuirass/templates.scm (build-details): Honor the new "products" argument
to display all the build products associated to the given build.
* src/schema.sql (BuildProducts): New table,
(Specifications)[build_outputs]: new field.
* tests/database.scm: Add empty build-outputs spec.
* tests/http.scm: Ditto.
* examples/guix-jobs.scm: Ditto.
* examples/hello-git.scm: Ditto.
* examples/hello-singleton.scm: Ditto.
* examples/hello-subset.scm: Ditto.
* examples/random.scm: Ditto.
* doc/cuirass.texi (overview): Document it.
2020-06-10 16:42:21 +02:00
Christopher Baines 78986d9623 database: Begin tuning db-get-builds for performance.
This commit does several things, the big change is to try and construct a
simpler query for SQLite. I'm not confident that SQLite's query planner can
look past handling the NULL parameters, so I think it could be helpful to try
and create a simpler query, both to avoid that problem if it exists, but also
move the complexity in to Guile code, which I think is a bit more manageable.

The way ordering is handled is also changed. Order is one of the filters,
although it's not a filter, and some of the other filters also influenced the
order. I think there are things still to fix/improve with the handling of
ordering, but at least this commit just has the ordering happen once in the
query.

* src/cuirass/database.scm (filters->order): Remove procedure, inline in to
db-get-builds.
(db-get-builds): Change query generation in an attempt to make it easier to
tune the queries for performance.
2020-06-09 19:58:29 +01:00
Christopher Baines 9265949cf6 cuirass: Perform some database "optimization" at startup.
Add a "optimize" step that occurs when starting up the main Curiass
process. Currently this does two things, but could be extended to do more.

The "PRAGMA optimize;" command prompts SQLite to ANALYZE tables where that
might help. The "PRAGMA wal_checkpoint(TRUNCATE);" command has SQLite process
any unprocessed changes from the WAL file, then truncate it to 0 bytes. I've
got no data to suggest this helps with performance, but I'm hoping that going
from a large WAL file to a small one occasionally might be useful.

* src/cuirass/database.scm (db-optimize): New procedure.
* bin/cuirass.in (main): Run it.
2020-05-25 11:08:20 +01:00
Ludovic Courtès c961de2f63 templates: Evaluation page shows each input and commit.
* src/cuirass/database.scm (db-get-inputs, db-get-checkouts): Export.
* src/cuirass/http.scm (evaluation-html-page): Pass #:checkouts and
 #:inputs to 'evaluation-build-table'.
* src/cuirass/templates.scm (evaluation-build-table): Add #:checkouts
and #:inputs.  Emit a table with "Input" and "Commit" columns.
2020-04-17 15:07:17 +02:00
Ludovic Courtès ae8c067d19 database: Add missing (ice-9 threads) import.
This is required on Guile 3 for 'current-processor-count'.

* src/cuirass/database.scm: Use (ice-9 threads).
2020-04-17 15:07:17 +02:00
Ricardo Wurmus dde1a93564
database: Support "status:" keyword in queries.
* src/cuirass/database.scm (query->bind-arguments): Handle "status" query.
(db-get-builds-by-search, db-get-builds-query-min, db-get-builds-query-max):
Add status filter.
2020-04-15 19:50:25 +02:00
Ludovic Courtès ed0f905cc7 database: 'db-get-builds' returns each build's evaluation ID.
* src/cuirass/database.scm (db-get-builds): Fetch 'Builds.evaluation'
and return it as #:eval-id.
2020-02-24 00:09:22 +01:00
Christopher Baines b9031db946 database: Enable running up to 4 database queries at once.
The number of threads is copied from bin/cuirass.in. When you have at least
two processors, this will allow database queries to be executed in parallel.

With some crude testing using the Apache HTTP server benchmarking tool (ab
from the httpd package), the max request latency does seem to drop when
multiple threads are used, especially when the database queries are slow (I
tested by adding usleep to the worker thread code).

* src/cuirass/database.scm (with-database): Pass #:parallelism to
make-worker-thread-channel.
2020-01-25 22:32:09 +00:00
Christopher Baines e34d773faf Adjust make-worker-thread-channel to take an initializer.
While this is a generic method, and initializer function will give the
flexibility required to create multiple worker threads for performing SQLite
queries, each with it's own database connection (as a result of calling the
initializer once for each thread). Without this change, they'd all have to use
the same connection, which would not work.

* src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take
an initializer, rather than arguments directly.
* src/cuirass/database.scm (with-database): Adjust to call
make-worker-thread-channel with an initializer.
* tests/database.scm (db-init): Change to use make-worker-thread-channel
initializer.
* tests/http.scm (db-init): Change to use make-worker-thread-channel
initializer.
2020-01-25 22:32:09 +00:00
Christopher Baines 0d23a6d374 utils: Change critical section terminology to worker threads.
As far as I'm aware, it's necessary to use a separate thread for interacting
with SQLite as one of the threads used for fibers will be blocked while the
SQLite query is running.

This doesn't mean all queries have to be executed one at a time though,
providing the queries are executed outside the threads used by fibers, and a
single connection isn't used in multiple threads.

These changes start to move in this direction, first by just changing the
terminology.

* src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change
with-db-critical-section to with-db-worker-thread.
* src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to
with-db-worker-thread.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-get-inputs, db-get-specification, db-add-evaluation,
db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output,
db-add-build, db-update-build-status!, db-get-output, db-get-outputs,
db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id,
db-add-event, db-get-events, db-delete-events-with-ids-<=-to,
db-get-pending-derivations, db-get-checkouts, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluations-id-max,
db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change
from using with-db-critical-section to
with-db-worker-thread.
(with-database): Change syntax rule to use make-worker-thread-channel,
renaming from make-critical-section.
* src/cuirass/utils.scm (%critical-section-args): Rename parameter to
%worker-thread-args.
(make-critical-section): Rename to make-worker-thread-channel, and adjust
parameter and docstring.
(call-with-critical-section): Rename to call-with-worker-thread and adjust
parameter.
(with-critical-section): Rename to with-worker-thread, and adjust to call
call-with-worker-thread.
* tests/database.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* tests/http.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
2020-01-25 22:32:09 +00:00
Christopher Baines fa412cdb59 Alter the Builds table to have an id field
The internal rowid's are used for builds as you can request builds by using
the rowid in the URL.

The motivation here is to enable running VACUUM operations in SQLite, without
risking the rowid's for Builds changing. It would be bad if they change, as
they're used in the URL's for builds.

* src/schema.sql (Builds): Add id column.
* src/curiass/dataabse.scm (db-add-build): Change PRIMARYKEY constraint to
UNIQUE constraint.
* src/sql/upgrade-6.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
2020-01-25 22:22:39 +00:00
Christopher Baines 9eb96d1b19 database: Don't return rowid from db-add-input.
As it is unused from where db-add-input is called.

* src/cuirass/database.scm (db-add-input): Don't call and
return (last-insert-rowid).
2020-01-24 19:46:25 +00:00
Christopher Baines 46f73b6b7c Support returning build information by output.
Being able to take a derivation and query the build information is useful, but
in cases where there are multiple derivations that produce the same outputs,
the probability of getting the data back from Cuirass is reduced.

This is because Cuirass might not have build the exact derivation you have,
but a different derivation that produces the same outputs (this can commonly
happen when a related fixed output derivation changes).

Cuirass doesn't store derivations if they produce the same outputs as a
derivation it already knows about, so it can't determine if this is the
case. Therefore, provide a way of querying build results by output, rather
than derivation.

The motivation behind this is to make it easier to import build information in
to the Guix Data Service.

* src/cuirass/database.scm (db-get-output): New procedure.
* src/cuirass/http.scm (respond-output-not-found): New procedure.
(request-path-components): Handle /output/… requests.
* doc/cuirass.texi (Build information): Mention that you can get build
information by output.
2020-01-16 08:33:03 +00:00
Christopher Baines 267649c9f0 Support publishing evaluation events
* src/cuirass/database.scm (db-add-evaluation): Record the creation of new
evaluations as events.
(db-set-evaluation-done): Record when evaluations finish as an event.
* src/cuirass/http.scm (url-handler): Add a new /api/evaluation-events page.
2020-01-16 08:33:03 +00:00
Christopher Baines 12def48b3b Support publishing build events
Add a table to store events, which have a type and a JSON blob. These can be
used to record changes, this commit inserts events when new builds are
created, and when the status of builds change.

The EventsOutbox table is then used to track when events have been sent
out. This is done through the new cuirass-send-events script.

* Makefile.am (bin_SCRIPTS): Add bin/cuirass-send-events.
.gitignore: Add bin/cuirass-send-events.
(dist_pkgmodule_DATA): Add src/cuirass/send-events.scm.
(dist_sql_DATA): Add src/sql/upgrade-5.sql.
(EXTRA_DIST): bin/cuirass-send-events.in.
(bin/cuirass-send-events): New rule.
* bin/cuirass-send-events.in: New file.
* src/cuirass/send-events.scm: New file.
* src/sql/upgrade-5.sql: New file.
* src/cuirass/database.scm (changes-count): New procedure.
(db-update-build-status!): Call db-add-event after updating the build status.
(db-add-event): New procedure.
(db-add-build): Insert an event when a new build is inserted.
(db-delete-events-with-ids-<=-to): New procedure.
* src/schema.sql (Events): New table.
2020-01-16 08:32:52 +00:00
Ricardo Wurmus 8a618322b1
database: Add db-remove-specification.
* src/cuirass/database.scm (db-remove-specification): New procedure.
2019-10-30 09:19:28 +01:00
Ricardo Wurmus fed15b83b0
database: query->bind-arguments: Process ^ and $ characters.
* src/cuirass/database.scm (query->bind-arguments): Handle ^ and $ characters
in query strings.
2019-06-20 23:31:34 +02:00
Ricardo Wurmus c3ff064e97
database: query->bind-arguments: Ignore invalid query terms.
* src/cuirass/database.scm (query->bind-arguments): Ignore invalid query
terms.
2019-06-19 15:37:05 +02:00
Ricardo Wurmus efe7d36e64
http: Show number of builds.
* src/cuirass/database.scm (db-get-evaluation-summary): New procedure.
* src/cuirass/http.scm (url-handler): Display number of builds in tabs.
2019-06-19 10:22:51 +02:00
Ricardo Wurmus 767f34bd68
database: db-get-specifications: Order specs by name.
* src/cuirass/database.scm (db-get-specifications): Order rows by "name".
2019-06-18 23:36:37 +02:00
Ricardo Wurmus 9017b9fbc1
database: Support advanced query strings.
* src/cuirass/database.scm (query->bind-arguments): New procedure.
(db-get-builds-by-search): Support filtering by spec and system.
(db-get-builds-query-min, db-get-builds-query-max): Adjust.
2019-05-21 11:40:28 +02:00
Ricardo Wurmus ce114d8446
cuirass: Add search for builds.
* src/cuirass/database.scm (db-get-builds-by-search, db-get-builds-query-min,
db-get-builds-query-max): New procedures.
* src/cuirass/http.scm (handle-builds-search-request): New procedure.
(url-handler): Handle "search" route.
* src/cuirass/templates.scm (search-form): New variable.
(html-page): Use it.
(build-search-results-table): New procedure.
2019-05-17 22:40:12 +02:00
Ludovic Courtès d0ed3341d7 database: Re-throw the message of 'sqlite-error' exceptions.
Fixes <https://bugs.gnu.org/33407>.
Reported by Björn Höfling <bjoern.hoefling@bjoernhoefling.de>.

* src/cuirass/database.scm (catch-sqlite-error): Add missing MESSAGE
argument to 'throw'.
2018-11-16 21:49:02 +01:00
Ludovic Courtès 09afb02528 database: Factorize 'sqlite-error' handling.
* src/cuirass/database.scm (catch-sqlite-error): New macro.
(db-add-checkout, db-add-output, db-add-build): Use it instead of custom
'catch' block'.
2018-11-16 21:48:28 +01:00
Ludovic Courtès c76edac15d database: Remove duplicate #:in-progress entry.
* src/cuirass/database.scm (db-get-evaluations-build-summary): Remove
duplicate #:in-progress entry.
2018-11-07 15:28:05 +01:00
Clément Lassieur fe2b73c235
database: Return #f when one row is expected and there is none.
* src/cuirass/database.scm (expect-one-row): New procedure.
(db-get-build, db-get-evaluations-id-min, db-get-evaluations-id-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Use
it.
2018-09-30 17:26:12 +02:00
TSholokhova cbf8e13835
templates: Add a navigation bar.
* src/cuirass/database.scm (db-get-evaluation-specification): New exported
procedure.
* src/cuirass/http.scm (respond-html): Allow to pass CODE as argument.
(respond-html-eval-not-found): New procedure.
(url-handler): Fill navigation arguments.  Handle the case where the
evaluation doesn't exist.
* src/cuirass/templates.scm (navigation-items): New procedure.
(html-page): Add navigation bar.

Co-authored-by: Clément Lassieur <clement@lassieur.org>
2018-09-30 17:08:15 +02:00
TSholokhova f51bd3f554
templates: Allow to filter builds by status.
* src/cuirass/database.scm (db-get-builds): Add 'succeeded' and 'failed'
status filters.
(db-get-builds-min, db-get-builds-max): Add support for a new 'status'
argument.
* src/cuirass/http.scm (url-handler): Add a status argument for the '/eval/id'
endpoint.
* src/cuirass/templates.scm (evaluation-badges): Add links to the buttons.
(build-eval-table): Add a 'status' argument.

Co-authored-by: Clément Lassieur <clement@lassieur.org>
2018-09-30 13:10:56 +02:00
Clément Lassieur 4e661552c3
database: Add builds only if one of their outputs is new.
* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'.
* src/cuirass/database.scm (db-add-output): New procedure.
(db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and return #f if
DB-ADD-OUTPUT returned an empty list.
* src/schema.sql (Outputs): Set 'path' as primary key, instead of 'derivation,
name'.
* src/sql/upgrade-4.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-build): Use the #:OUTPUTS key.  Get default
OUTPUTS to depend on DRV.
("db-add-build-with-fixed-output"): New test.
2018-09-29 22:29:06 +02:00
Clément Lassieur 8d40c49170
database: Add a Checkouts table.
It is used to know when a new evaluation must be triggered and to display
input changes.

* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-3.sql'.
* bin/cuirass.in (main): Call DB-SET-EVALUATION-DONE at startup to clear
'in-progress' evaluations.
* bin/evaluate.in (input-checkout, format-checkouts): Rename '#:name' to
'#:input'.
* doc/cuirass.texi (Stamps): Remove section.
(Checkouts): New section.
* src/cuirass/base.scm (fetch-input, fetch-inputs, compile-checkouts): Rename
'#:name' to '#:input'.
(evaluate): Remove the COMMITS argument.  Add an EVAL-ID argument.  Don't call
DB-ADD-EVALUATION because it was called sooner.  Remove the EVAL-ID argument
to AUGMENT-JOB because it's a closure.
(build-packages): Add an EVAL-ID argument.  Call DB-SET-EVALUATION-DONE once
all the derivations are registered.
(process-specs): Replace the stamping mechanism by the primary key constraint
of the Checkouts table: call "evaluate" only when DB-ADD-EVALUATION is true,
which means that at least one checkout was added.  Change the EVALUATE and
BUILD-PACKAGES arguments accordingly.
* src/cuirass/database.scm (db-add-stamp, db-get-stamp): Remove procedures.
(db-set-evaluations-done, db-set-evaluation-done): New exported procedure.
(db-add-checkout): New procedure that returns #f if a checkout with the same
revision already exists.
(db-add-evaluation): Replace the EVAL argument with a SPEC-NAME and a
CHECKOUTS arguments.  Insert the evaluation only if at least one checkout was
inserted.  Return #f otherwise.
(db-get-checkouts): New procedure.
(db-get-evaluations, db-get-evaluations-build-summary): Handle the
'in_progress' column, remove the 'commits' column.  Return the result of
DB-GET-CHECKOUTS as part of the evaluation.
* src/cuirass/templates.scm (input-changes, evaluation-badges): New
procedures.
(evaluation-info-table): Rename "Commits" to "Input changes".  Use
INPUT-CHANGES to display the input changes that triggered the evaluation.  Use
EVALUATION-BADGES to display a message indicating that the evaluation is in
progress.
* src/schema.sql (Stamps): Remove table.
(Checkouts): New table.
(Evaluations): Remove the 'commits' column.  Add an 'in_progress' column.
* src/sql/upgrade-3.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-eval): Remove procedure.
(make-dummy-checkouts): New procedure.
("sqlite-exec"): Remove the 'commits' column.  Add the 'in_progress' column.
("db-update-build-status!", "db-get-builds", "db-get-pending-derivations"):
Update the arguments of DB-ADD-EVALUATION accordingly.
* tests/http.scm (hash-table=?): Add support for lists of hash tables.
(evaluations-query-result): Replace '#:commits' with '#:checkouts'.  Return a
list instead of returning one element, for symmetry.
("fill-db"): Add a new input so that the second checkout can refer to it.
Replace EVALUATION1 and EVALUATION2 with CHECKOUTS1 and CHECKOUTS2.  Update
the arguments of DB-ADD-EVALUATION accordingly.
("/api/queue?nr=100"): Take the CAR of the EVALUATIONS-QUERY-RESULT list to
make it symmetrical with the other argument of HASH-TABLE=?.
2018-08-27 15:38:44 +02:00
Clément Lassieur 4612a3a70f
database: Serialize all database accesses in a thread.
Fixes <https://bugs.gnu.org/32234>.

* bin/cuirass.in (main): Keep only one WITH-DATABASE call around all fibers.
Remove all DB arguments.
* src/cuirass/base.scm (evaluate, update-build-statuses!, spawn-builds,
handle-build-event, build-packages): Remove all DB arguments.
(clear-build-queue, cancel-old-builds): Wrap in WITH-DB-CRITICAL-SECTION,
remove all DB arguments.
(restart-builds): Remove the NON-BLOCKING call, remove all DB arguments.
(process-specs): Remove all DB arguments, remove the WITH-DATABASE call.
* src/cuirass/database.scm (%db-channel): New parameter.
(with-db-critical-section): New macro.
(db-add-input, db-add-specification, db-get-inputs, db-get-specifications,
db-add-evaluation, db-add-build, db-update-build-status!, db-get-outputs,
db-get-builds, db-get-build, db-get-pending-derivations, db-get-stamp,
db-add-stamp, db-get-evaluations, db-get-evaluations-build-summary,
db-get-evaluations-id-min, db-get-evaluations-id-max, db-get-builds-min,
db-get-builds-max): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB arguments.
(with-database): Wrap BODY in PARAMETERIZE form that sets %DB-CHANNEL to the
channel returned by MAKE-CRITICAL-SECTION.
* src/cuirass/http.scm (handle-build-request, handle-builds-request): Remove
all DB arguments.
(url-handler): Remove all DB arguments, remove the DB-CHANNEL state, remove
the WITH-CRITICAL-SECTION calls.
(run-cuirass-server): Remove the DB arguments, remove the
MAKE-CRITICAL-SECTION call.
* src/cuirass/utils.scm (make-critical-section): Replace SPAWN-FIBER with
CALL-WITH-NEW-THREAD.  Wrap body in PARAMETERIZE form that clears
CURRENT-FIBER.
* tests/database.scm (with-temporary-database, "db-add-specification",
"db-add-build", "db-update-build-status!", "db-get-builds",
"db-get-pending-derivations"): Remove the DB arguments.
("db-init"): Set the %DB-CHANNEL parameter to the channel returned by
MAKE-CRITICAL-SECTION, and return #t.
("database"): Set %DB-CHANNEL to #f during cleanup.
* tests/http.scm ("db-init"): Set the %DB-CHANNEL parameter to the channel
returned by MAKE-CRITICAL-SECTION, and return #t.
("cuirass-run", "fill-db"): Remove the DB arguments.
("http"): Set %DB-CHANNEL to #f during cleanup.
2018-08-16 19:19:23 +02:00
Clément Lassieur 4db99f647b
database: Merge Derivations into Builds table.
Fixes <https://bugs.gnu.org/32190>.

* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-2.sql'.
* doc/cuirass.texi (Derivations): Remove section.
(Builds): Update accordingly.  Add columns from the Derivations table.
(Outputs): Replace 'id' with 'derivation'.
* src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.
(build-packages): Add columns that were in the Derivations table before.  Only
build the derivations that were successfully registered, that is, those that
didn't exist in the Builds table.  Give a derivation instead of a build id to
DB-GET-BUILD.  Compute the number of failed jobs based on the derivations that
were added to the table, instead of the jobs.
* src/cuirass/database.scm (db-add-derivation, db-get-derivation): Remove
exported procedures.
(db-add-build): Catch SQLITE_CONSTRAINT_PRIMARYKEY error, which means that two
jobs produced the same derivation, and return #f in that case.  Add columns
that were in the Derivations table before.  Use 'derivation' as primary key
for the Outputs table.
(db-get-outputs): Use 'derivation' as identifier, instead of 'build-id'.
(filters->order): Replace 'id' with 'rowid'.
(db-get-builds): Add a 'derivation' filter.  Replace 'id' with 'rowid'.
Remove the 'INNER JOIN Derivations'.  Replace Derivations with Builds.  Return
'derivation' in first position to make it clear that it's the primary key.
Pass DERIVATION instead of ID to DB-GET-OUTPUTS.
(db-get-build): Allow to take a derivation as argument.  Use NUMBER? to
differentiate between derivations and ids.
(db-get-pending-derivations): Remove the 'SELECT DISTINCT' clause now that
derivations are unique.  Remove the 'INNER JOIN Builds'.
(db-get-evaluations-build-summary, db-get-builds-min, db-get-builds-max):
Replace 'id' with 'rowid'.
* src/schema.sql (Derivations): Remove table.
(Outputs): Replace Builds.id with Builds.derivation.
(Builds): Use 'derivation' as primary key.  Remove the 'id' column.  Add
'job_name', 'system', 'nix_name' columns that were in the Derivations table
before.
(Builds_Derivations_index): Rename to Builds_index.  Update accordingly.
(Derivations_index): Remove index.
* src/sql/upgrade-2.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-job, make-dummy-derivation): Remove
procedures.
(make-dummy-build): Add columns that were in MAKE-DUMMY-DERIVATION.  Get the
DRV parameter to be mandatory because it's a primary key.
(%id): Remove parameter.
("db-add-derivation", "db-get-derivation"): Remove tests.
("db-add-build"): Expect #f, because it adds twice the same derivation.  Pass
the derivation argument to MAKE-DUMMY-BUILD.
("db-update-build-status!"): Rename 'id' to 'derivation'.  Pass the derivation
argument to MAKE-DUMMY-BUILD.  Remove the DB-ADD-DERIVATION call.
("db-get-builds", "db-get-pending-derivations"): Pass the derivation argument
to MAKE-DUMMY-BUILD.  Remove the DB-ADD-DERIVATION calls.
* tests/http.scm ("fill-db"): Remove DERIVATION1 and DERIVATION2, and put
their content in BUILD1 and BUILD2.  Remove the DB-ADD-DERIVATION calls.
2018-08-16 19:19:23 +02:00
Clément Lassieur b4d058fc8d
database: Fix the builds limit issue.
Fixes <https://bugs.gnu.org/32300>.

* src/cuirass/database.scm (filters->order): New procedure.
(db-get-builds): Remove FORMAT-OUTPUT, CONS-OUTPUT, COLLECT-OUTPUTS,
FINISH-GROUP, SAME-GROUP?, GROUP-OUTPUTS procedures.  Remove the 'LEFT JOIN
Outputs' clause.  Use DB-GET-OUTPUTS for each build that was fetched.
2018-08-16 19:19:23 +02:00
Clément Lassieur 31a398f8a3
database: Put database in /var/lib to avoid removal at each boot.
Fixes <https://bugs.gnu.org/32330>.

* src/cuirass/database.scm (%package-database): Change default file name from
/var/run/cuirass/cuirass.db to /var/lib/cuirass/cuirass.db.
2018-08-16 19:19:14 +02:00
Clément Lassieur fb11e79c2e
database: Fix the evaluations limit issue.
* src/cuirass/database.scm (db-get-evaluations-build-summary): Use a 'LEFT
JOIN' instead of a 'JOIN', with 'Evaluations' on the left, so that evaluations
are displayed even when they don't have associated builds.
2018-08-04 16:34:27 +02:00
Clément Lassieur dc27cb369c
database: Remove unused DB-FORMAT-BUILD procedure.
* src/cuirass/database.scm (db-format-build): Remove it.
2018-08-04 16:33:59 +02:00
Clément Lassieur 4c044c658a
Rename 'repo-name' to 'specification'.
* src/cuirass/database.scm (db-get-builds): Rename 'repo-name' to
'specification'.
* src/cuirass/http.scm (build->hydra-build): Idem.
2018-08-04 16:32:22 +02:00
Clément Lassieur d36ec7a385
database: Limit SQL queries to 80 columns. 2018-08-04 16:31:34 +02:00
Clément Lassieur 99241ef1af
http: Change the paramater format from two-elements lists to pairs.
* src/cuirass/database.scm (assqx-ref): Remove exported procedure.
(db-get-builds, db-get-build): Adapt to new format.
* src/cuirass/http.scm (request-parameters): Use (cons key param) instead
of (list key param).
(url-handler): Adapt to new format.
* tests/database.scm ("db-get-builds"): Idem.
2018-07-30 00:26:31 +02:00
TSholokhova 675cd04a85
Add a web interface.
* Makefile.am (dist_sql_DATA): Add static files.
* src/cuirass/database.scm (assqx-ref): Export it.
(db-get-builds): Add 'evaluation' filter and filters for pagination.
(db-get-evaluations-build-summary, db-get-evaluations-id-min,
db-get-evaluations-id-max, db-get-builds-min, db-get-builds-max): New exported
procedures.
* src/cuirass/http.scm (%static-directory): New parameter.
(%page-size, %file-mime-types, %file-white-list): New variables.
(handle-build-request, handle-builds-request): Move the WITH-CRITICAL-SECTION
call out.
(url-handler): Add RESPOND-HTML, RESPOND-STATIC-FILE and RESPOND-NOT-FOUND
procedures.  Call WITH-CRITICAL-SECTION sooner for the '/build',
'/api/latestbuilds' and '/api/queue' routes.  Add '/', '/jobset/<name>',
'/eval/<id>', '/static/<path>' routes.  Use RESPOND-NOT-FOUND when the route
isn't found.
* src/cuirass/templates.scm: New file.
(html-page, specifications-table, evaluation-info-table,
build-eval-table): New exported procedures.
(pagination): New procedure.

Signed-off-by: Clément Lassieur <clement@lassieur.org>
2018-07-29 23:47:00 +02:00