* README: Update dependencies.
* tests/common.scm (pg-tmp): New procedure.
(test-init-db!): Use it.
* tests/database.scm ("db-close"): Do not drop database content.
* tests/http.scm: Ditto.
* tests/metrics.scm: Ditto.
* src/cuirass/database.scm (db-remove-unresponsive-workers): Restart the
builds that are started on unresponsive workers.
* tests/database.scm ("db-remove-unresponsive-workers"): Test it.
* src/sql/upgrade-4.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds)[weather]: New field.
(Builds_weather_evaluation): New index.
* src/cuirass/database.scm (db-get-last-status): New procedure.
(db-update-build-status!): Use it and update the new weather field.
(db-get-builds): Adapt it.
* tests/http.scm (build-query-result): Ditto.
* src/cuirass/database.scm (build-weather): New macro.
(build-status->weather): New procedure.
(db-get-builds): Return the build weather using the new procedure.
* src/cuirass/http.scm (build->hydra-build): Also return the weather.
* src/cuirass/templates.scm (weather-class, weather-title): New procedures.
(build-eval-table): Display the weather.
* tests/database.scm ("db-get-build weather"): New tests.
* tests/http.scm (build-query-result): Adapt it.
* src/sql/upgrade-3.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds)[last_status]: New field.
* src/cuirass/database.scm (db-update-build-status!): Honor it.
(db-get-builds): Return it.
* tests/database.scm ("db-update-build-status!"): New test.
* src/cuirass/database.scm (db-add-worker): Rename ...
(db-add-or-update-worker): ... into this new procedure.
(db-get-worker, db-remove-unresponsive-workers): New procedures.
* src/cuirass/remote-server.scm (%workers): Remove it.
(pop-build): Adapt it.
(remove-unresponsive-workers!): Remove it.
(read-worker-exp): Adapt it.
(zmq-start-proxy): Ditto.
* tests/database.scm ("db-add-worker"): Rename ...
("db-add-or-update-worker"): ... into this new test.
("db-get-worker", "db-remove-unresponsive-workers"): New tests.
* src/cuirass/remote.scm: New file.
* src/cuirass/remote-server.scm: New file.
* src/cuirass/remote-worker.scm: New file.
* bin/remote-server.in: New file.
* bin/remote-worker.in: New file.
* Makefile.am (bin_SCRIPTS): Add new binaries,
(dist_pkgmodule_DATA): add new files,
(EXTRA_DIST): add new binaries,
(bin/remote-server, bin/remote-worker): new targets.
* .gitignore: Add new binaries.
* bin/cuirass.in (%options): Add "--build-remote" option,
(show-help): document it,
(main): honor it.
* src/cuirass/base.scm (with-build-offload-thread): New macro,
(%build-remote?, %build-offload-channel): new parameters,
(make-build-offload-thread): new procedure,
(build-derivations/offload): new procedure,
(restart-builds): use it to offload builds when %build-remote? is set,
(build-packages): ditto.
This is a follow-up of b67f38a7b9.
* src/cuirass/database.scm (catch-sqlite-error): New macro.
(SQLITE_CONSTRAINT_PRIMARYKEY, SQLITE_CONSTRAINT_UNIQUE, %db-writer-channel):
New variables.
* tests/database.scm (with-temporary-database): Set "%db-writer-channel".
(db-add-build-with-fixed-output): Catch sqlite error.
(db-get-pending-derivations): Do not add builds with duplicated outputs.
* tests/metrics.scm ("builds-per-day", "new-derivations-per-day"): Fix tests
results that were depending of the local time.
("evaluation-completion-speed"): Adapt to new unit.
* src/cuirass/metrics.scm (db-average-build-start-time-per-eval,
db-average-build-complete-time-per-eval, db-evaluation-completion-speed,
db-latest-evaluations): New procedures.
(%metrics): Add 'average-eval-build-start-time,
'average-eval-build-complete-time, 'evaluation-completion-speed.
(db-update-metrics): Update evaluation related metrics for the evaluations
added the past three days.
* tests/metrics.scm ("average-eval-build-start-time",
"average-eval-build-complete-time", "evaluation-completion-speed"): Add new
tests.
* tests/metrics.scm (nearest-exact-integer): Remove as unused.
("db-update-metrics"): Test that 'pending-builds metric is updated on
"db-update-metrics" call.
This is a follow-up of 154232bc76.
* tests/http.scm (evaluations-query-result): Add 'timestamp', 'checkouttime'
and 'evaltime' fields.
(fill-db): Pass a matching timestamp to "db-add-evaluation" calls.
The Evaluation table currently has an 'in_progress' field. Distinction between
succeeded and failed evaluations are based on the presence of Builds records
for the evaluation. It it also not possible to distinguish aborted evaluations
from failed evaluations.
Rename 'in_progress' field to 'status'. The 'status' field can be equal to
'started', 'succeeded', 'failed' or 'aborted'.
* src/cuirass/database.scm (evaluation-status): New exported enumeration.
(db-set-evaluations-done, db-set-evaluation-done): Remove them.
(db-abort-pending-evaluations, db-set-evaluation-status): New exported procedures.
(db-add-evaluation, db-get-builds, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluation-summary): Adapt to use
'status' field instead of 'in_progress' field.
* src/cuirass/templates.scm (evaluation-badges): Ditto.
* src/schema.sql (Evaluations): Rename 'in_progress' field to 'status'.
* src/sql/upgrade-10.sql: New file.
* bin/cuirass.in (main): Use "db-abort-pending-evaluations" instead of
"db-set-evaluations-done".
* src/cuirass/base.scm (evaluate): Use "db-set-evaluation-status" instead of
"db-set-evaluations-done".
(build-packages): Use "db-set-evaluation-status" instead of
"db-set-evaluation-done".
* tests/database.scm (sqlite-exec): Adapt accordingly.
* tests/http.scm (evaluations-query-result): Ditto.
src/cuirass/base.scm (fetch-input): Add the commit timestamp to the returned
association list,
(process-specs): Pass a timestamp taken at procedure start and another one
taken after inputs are fetched to "db-add-evaluation" procedure. Once the
evaluation is over, call "db-set-evaluation-time" to save the evaluation
completion time.
src/cuirass/database.scm (db-set-evaluation-time): New procedure,
(db-add-checkout): Handle the "timestamp" field,
(db-add-evaluation): add "checkouttime" and "evaltime" arguments. Modify the
associated SQL query accordingly.
(db-get-builds): Use "Builds.timestamp" instead of "timestamp" as this field
is also part of the Evaluations table.
src/schema.sql (Checkouts): Add "timestamp" field,
(Evaluations): add "timestamp", "checkouttime" and "evaltime" fields.
src/sql/upgrade-9.sql: New file.
tests/database.scm (sqlite-exec): Adapt Evaluations table insertions to include
"timestamp", "checkouttime" and "evaltime" required fields.
Fixes <https://issues.guix.gnu.org/43163>.
* src/cuirass/http.scm (specification->json-object): New procedure,
(url-handler): use it for "/specifications" route to convert specification
objects into a representation suitable for json->scm.
* tests/http.scm ("/specifications"): Test the above route.
Handle requests such as "/build/?nr" by ignoring parameters without any
associated value.
* src/cuirass/http.scm (request-parameters): Ignore silently parameters
without an associated value.
* tests/http.scm: Add a corresponding test case.
When "spawn-builds" is called to restart builds, the spec is not known,
preventing build products from being created as reported here:
https://issues.guix.gnu.org/42523
Fix this issue by reading the specification in database in
"set-build-successful!" procedure.
* src/cuirass/database.scm (db-get-specification): New exported procedure,
(db-get-specifications): add an optional name argument.
* tests/database.scm (db-get-specification): Add a corresponding test-case.
* src/cuirass/base.scm (set-build-successful!): Remove spec argument and read
it directly from database instead,
(update-build-statuses!): also remove spec argument, adapt
set-build-successful! call accordingly,
(spawn-builds): remove spec argument and adapt handle-build-event and
update-build-statuses! calls accordingly,
(handle-build-event): remove spec argument, adapt
set-build-successful! call accordingly,
(build-packages): remove spec argument, adapt spawn-builds call accordingly,
(process-specs): adapt build-packages call.
* src/sql/upgrade-7.sql: New file.
* Makefile.am: Add it.
* src/cuirass/base.scm (create-build-outputs): New procedure,
(build-packages): call it,
(process-spec): add the new spec argument and pass it to create-build-outputs.
* src/cuirass/database.scm (db-add-build-product, db-get-build-product-path,
db-get-build-products): New exported procedures.
* src/cuirass/http.scm (respond-static-file): Move file sending to ...
(respond-file): ... this new procedure,
(url-handler): add a new "download/<id>" route, serving the requested file
with the new respond-file procedure. Also gather build products and pass them
to "build-details" for "build/<id>/details" route.
* src/cuirass/templates.scm (build-details): Honor the new "products" argument
to display all the build products associated to the given build.
* src/schema.sql (BuildProducts): New table,
(Specifications)[build_outputs]: new field.
* tests/database.scm: Add empty build-outputs spec.
* tests/http.scm: Ditto.
* examples/guix-jobs.scm: Ditto.
* examples/hello-git.scm: Ditto.
* examples/hello-singleton.scm: Ditto.
* examples/hello-subset.scm: Ditto.
* examples/random.scm: Ditto.
* doc/cuirass.texi (overview): Document it.
While this is a generic method, and initializer function will give the
flexibility required to create multiple worker threads for performing SQLite
queries, each with it's own database connection (as a result of calling the
initializer once for each thread). Without this change, they'd all have to use
the same connection, which would not work.
* src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take
an initializer, rather than arguments directly.
* src/cuirass/database.scm (with-database): Adjust to call
make-worker-thread-channel with an initializer.
* tests/database.scm (db-init): Change to use make-worker-thread-channel
initializer.
* tests/http.scm (db-init): Change to use make-worker-thread-channel
initializer.
As far as I'm aware, it's necessary to use a separate thread for interacting
with SQLite as one of the threads used for fibers will be blocked while the
SQLite query is running.
This doesn't mean all queries have to be executed one at a time though,
providing the queries are executed outside the threads used by fibers, and a
single connection isn't used in multiple threads.
These changes start to move in this direction, first by just changing the
terminology.
* src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change
with-db-critical-section to with-db-worker-thread.
* src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to
with-db-worker-thread.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-get-inputs, db-get-specification, db-add-evaluation,
db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output,
db-add-build, db-update-build-status!, db-get-output, db-get-outputs,
db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id,
db-add-event, db-get-events, db-delete-events-with-ids-<=-to,
db-get-pending-derivations, db-get-checkouts, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluations-id-max,
db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change
from using with-db-critical-section to
with-db-worker-thread.
(with-database): Change syntax rule to use make-worker-thread-channel,
renaming from make-critical-section.
* src/cuirass/utils.scm (%critical-section-args): Rename parameter to
%worker-thread-args.
(make-critical-section): Rename to make-worker-thread-channel, and adjust
parameter and docstring.
(call-with-critical-section): Rename to call-with-worker-thread and adjust
parameter.
(with-critical-section): Rename to with-worker-thread, and adjust to call
call-with-worker-thread.
* tests/database.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* tests/http.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
