* src/cuirass/remote.scm: New file.
* src/cuirass/remote-server.scm: New file.
* src/cuirass/remote-worker.scm: New file.
* bin/remote-server.in: New file.
* bin/remote-worker.in: New file.
* Makefile.am (bin_SCRIPTS): Add new binaries,
(dist_pkgmodule_DATA): add new files,
(EXTRA_DIST): add new binaries,
(bin/remote-server, bin/remote-worker): new targets.
* .gitignore: Add new binaries.
* bin/cuirass.in (%options): Add "--build-remote" option,
(show-help): document it,
(main): honor it.
* src/cuirass/base.scm (with-build-offload-thread): New macro,
(%build-remote?, %build-offload-channel): new parameters,
(make-build-offload-thread): new procedure,
(build-derivations/offload): new procedure,
(restart-builds): use it to offload builds when %build-remote? is set,
(build-packages): ditto.
* src/sql/upgrade-17.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds): Add "machine" field.
* src/cuirass/database.scm (db-update-build-machine!): New procedure.
* src/cuirass/base.scm (handle-build-event): Use it to record the machine
performing the build.
* bin/cuirass.in (%options): Add "queue-size" argument.
(show-help): Document it.
(main): Honor it.
* src/cuirass/database.scm (%db-writer-queue-size): New parameter,
(with-queue-writer-worker): use it.
Rather than calling sqlite-reset before reading the rows, call sqlite-reset
after. I think this is important to stop SQLite getting stuck because the
statement is still live even though it's not being read from after these
procedures return.
Assuming I've got the ordering right, I think using sqlite-fold-right directly
is simpler code as well.
* src/cuirass/database.scm (db-get-builds-by-search, format-build-products,
db-get-events): Rewrite fetching result rows.
sqlite-prepare will reset cached statements before returning them, so these
sqlite-reset calls are redundant.
* src/cuirass/database.scm (db-get-builds-query-min, db-get-builds-query-max):
Remove sqlite-reset calls.
All these statements are cached, but that's pointless if they're finalized
after use, because they'll need recreating on next use.
* src/cuirass/database.scm (%sqlite-exec, db-get-builds-query-min,
db-get-builds-query-max): Replace sqlite-finalize with sqlite-reset.
When the periodic argument is passed to a job, it will be registered only if
the time difference between the current time and the registration time of the
last build is greater than the specified period. Make sure that the last job
that is searched is part of the same specification.
* src/sql/upgrade-16.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_job_name_timestamp): New index.
* src/cuirass/database.scm (db-get-time-since-previous-build): Add
"specification" argument,
(db-register-builds): pass it.
* src/cuirass/database.scm (db-get-time-since-previous-build): New procedure,
(db-register-builds): if the period argument is set, only register builds
which last registration is older than the specified period.
This is a follow-up of b67f38a7b9.
* src/cuirass/database.scm (catch-sqlite-error): New macro.
(SQLITE_CONSTRAINT_PRIMARYKEY, SQLITE_CONSTRAINT_UNIQUE, %db-writer-channel):
New variables.
* tests/database.scm (with-temporary-database): Set "%db-writer-channel".
(db-add-build-with-fixed-output): Catch sqlite error.
(db-get-pending-derivations): Do not add builds with duplicated outputs.
SQLite only allows one concurrent write query operation. Having multiple
database workers calling "db-update-build-status!", will thus increase worker
starvation. Every write operation will also be done is a single transaction.
For those reasons, create a database worker dedicated to write queries. Have
this worker queue work and issue all the queued work queries in a single
transaction.
* .dir-locals.el: Add with-db-writer-worker-thread.
* src/cuirass/database.scm (with-queue-writer-worker): Rename
"with-registration-workers" macro.
(%db-writer-channel): Rename "%db-registration-channel" variable.
(with-queue-writer-worker): Rename "with-registration-workers".
(db-register-builds): Use "with-db-writer-worker-thread" instead of
"with-db-registration-worker-thread".
(db-update-build-status!): Ditto
* src/cuirass/utils.scm (make-worker-thread-channel): Add "queue-size" and
"queue-proc" arguments.
(call-with-worker-thread): Add "options" argument.
* bin/cuirass.in (main): Use "with-queue-writer-worker" instead of
"with-registration-workers". Modify the macro scope to include all the
possible write operations.
Registering an evaluation can cause a large number of write queries using
individual transactions. This is really sub-optimal as commiting a transaction
will often result in a database sync to the disk.
* src/cuirass/database.scm (db-add-build): Do not create a new transaction for
each new build, instead ...
(db-register-builds): ..., create a single transaction per evaluation
registration.
Make sure that db-get-builds only performs one query and add build products to
the result. Also add indexes so that all "order" filters are covered.
Limit the maximum of build objects returned in "/api/latestbuilds" and
"/api/queue" to 1000.
Fixes: <https://issues.guix.gnu.org/43826>.
* src/sql/upgrade-15.sql: New file.
* Makefile (dist_sql_DATA): Add it.
* src/cuirass/database.scm (db-get-builds-by-search): Add "buildproducts" to
the returned build objects.
(db-get-builds): Remove unused filters. Join Outputs and BuildProducts table
in the query. Add "buildproducts" to the returned build objects.
* src/cuirass/http.scm (build->hydra-build): Use "buildproducts" from
"db-get-builds" procedure.
(url-handler): Forbid limit parameters greater than 1000 and use
"buildproducts" from "db-get-builds" procedure.
* src/schema.sql (Builds_stoptime, Builds_stoptime_id, Builds_status_ts_id):
New indexes.
Searching for something like "%a%" can take up to ten minutes. Remove the
usage of special characters "^" and "$" in queries. Instead, always add a "%"
character at the end of the query. Also add an index on the nix_name field.
Fixes: <https://issues.guix.gnu.org/43791>.
* src/sql/upgrade-13.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_nix_name): New index.
* src/cuirass/database.scm (query->bind-arguments): Remove support for "^" and
"$" special characters. Instead make sure that the query does not contain any
"%" character and prefix the query by "%".
(db-get-builds-by-search): Remove an useless query nesting level.
* src/cuirass/templates.scm (search-form): Adapt the search help message
accordingly.
The number of database worker is limited, and can introduce some
starvation. Print a message each time a two seconds timeout expires while
sending an SQL query to a worker.
* src/cuirass/utils.scm (put-message-with-timeout): New procedure.
(call-with-worker-thread): Rename timeout and timeout-proc arguments to
receive-timeout and receive-timeout-proc arguments. Add two new arguments:
send-timeout and send-timeout-proc. Use put-message-with-timeout to honor
them.
* src/cuirass/database.scm (with-db-worker-thread): Adapt accordingly and pass
send-timeout and send-timeout-proc arguments.
* src/cuirass/database.scm (db-register-builds): Remove store argument and
assume that 'log and 'outputs properties are provided by the evaluation.
* src/cuirass/base.scm (build-packages): Adapt accordingly.
Evaluation registration involves running a large number of SQL queries. This
can cause some database worker starvation as well as some contention. To avoid
this issue, spawn database workers dedicated to evaluation registration.
* src/cuirass/database.scm (%db-registration-channel): New variable.
(with-db-registration-worker-thread, with-registration-workers): New macros.
(with-db-worker-thread-no-timeout): Remove it.
(db-register-builds): Run registration in dedicated database workers using
"with-db-registration-worker-thread" macro.
* bin/cuirass.in (main): Spawn database registration workers by calling
"with-registration-workers" macro.
Make sure that all registration queries are done from within a single database
worker. Otherwise, when builds from multiple evaluations are registered at
the same time, some contention occurs communicating with workers.
* src/cuirass/base.scm (new-outputs?, build-packages): Move build registration
to ...
* src/cuirass/database.scm (db-register-builds): ... this new procedure.
(with-db-worker-thread-no-timeout): New procedure. Use it in
"db-register-builds" to avoid timeout messages.
Avoid full scans of Builds table that can be very time consuming by rewriting
some queries and using new indexes.
* src/sql/upgrade-12.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_evaluation_index, Evaluations_status_index,
Evaluations_specification_index): New indexes.
* src/cuirass/database.scm (db-get-evaluations-build-summary,
db-get-evaluation-summary): Rewrite queries to avoid full Builds table scan
and use the new indexes.
* bin/cuirass.in (show-help): Document "--log-queries" option.
(%options): Add it.
(main): Enable query logging if the above option is set.
* src/cuirass/database.scm (db-log-queries): New procedure.
* src/cuirass/logging.scm (query-logging-port): New parameter.
(log-query): New procedure.
The Evaluation table currently has an 'in_progress' field. Distinction between
succeeded and failed evaluations are based on the presence of Builds records
for the evaluation. It it also not possible to distinguish aborted evaluations
from failed evaluations.
Rename 'in_progress' field to 'status'. The 'status' field can be equal to
'started', 'succeeded', 'failed' or 'aborted'.
* src/cuirass/database.scm (evaluation-status): New exported enumeration.
(db-set-evaluations-done, db-set-evaluation-done): Remove them.
(db-abort-pending-evaluations, db-set-evaluation-status): New exported procedures.
(db-add-evaluation, db-get-builds, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluation-summary): Adapt to use
'status' field instead of 'in_progress' field.
* src/cuirass/templates.scm (evaluation-badges): Ditto.
* src/schema.sql (Evaluations): Rename 'in_progress' field to 'status'.
* src/sql/upgrade-10.sql: New file.
* bin/cuirass.in (main): Use "db-abort-pending-evaluations" instead of
"db-set-evaluations-done".
* src/cuirass/base.scm (evaluate): Use "db-set-evaluation-status" instead of
"db-set-evaluations-done".
(build-packages): Use "db-set-evaluation-status" instead of
"db-set-evaluation-done".
* tests/database.scm (sqlite-exec): Adapt accordingly.
* tests/http.scm (evaluations-query-result): Ditto.
src/cuirass/database.scm (db-get-evaluations): Add support for "timestamp",
"checkouttime" and "evaltime" fields,
(db-get-evaluation-summary): ditto.
src/cuirass/templates.scm (nearest-exact-integer, seconds->string): New
procedures,
(evaluation-build-table): print evaluation date and duration.
src/cuirass/base.scm (fetch-input): Add the commit timestamp to the returned
association list,
(process-specs): Pass a timestamp taken at procedure start and another one
taken after inputs are fetched to "db-add-evaluation" procedure. Once the
evaluation is over, call "db-set-evaluation-time" to save the evaluation
completion time.
src/cuirass/database.scm (db-set-evaluation-time): New procedure,
(db-add-checkout): Handle the "timestamp" field,
(db-add-evaluation): add "checkouttime" and "evaltime" arguments. Modify the
associated SQL query accordingly.
(db-get-builds): Use "Builds.timestamp" instead of "timestamp" as this field
is also part of the Evaluations table.
src/schema.sql (Checkouts): Add "timestamp" field,
(Evaluations): add "timestamp", "checkouttime" and "evaltime" fields.
src/sql/upgrade-9.sql: New file.
tests/database.scm (sqlite-exec): Adapt Evaluations table insertions to include
"timestamp", "checkouttime" and "evaltime" required fields.
* src/cuirass/utils.scm (with-operation, get-message-with-timeout): New
procedures,
(call-with-worker-thread): add timeout and timeout-proc arguments.
* src/cuirass/database.scm (with-db-worker-thread): Pass a 5 seconds timeout
to call-with-worker-thread, and print a debug message on timeout expiration.
When "spawn-builds" is called to restart builds, the spec is not known,
preventing build products from being created as reported here:
https://issues.guix.gnu.org/42523
Fix this issue by reading the specification in database in
"set-build-successful!" procedure.
* src/cuirass/database.scm (db-get-specification): New exported procedure,
(db-get-specifications): add an optional name argument.
* tests/database.scm (db-get-specification): Add a corresponding test-case.
* src/cuirass/base.scm (set-build-successful!): Remove spec argument and read
it directly from database instead,
(update-build-statuses!): also remove spec argument, adapt
set-build-successful! call accordingly,
(spawn-builds): remove spec argument and adapt handle-build-event and
update-build-statuses! calls accordingly,
(handle-build-event): remove spec argument, adapt
set-build-successful! call accordingly,
(build-packages): remove spec argument, adapt spawn-builds call accordingly,
(process-specs): adapt build-packages call.
Add a "status" dropdown menu in the navigation bar. For now this menu only
contains one item, a link to the "latest builds" page at "/status" location.
* src/cuirass/database.scm (db-get-builds): Add support for 'started status.
* src/cuirass/http.scm (url-handler): Add "/status" route.
* src/cuirass/templates.scm (running-builds-table): New procedure,
(html-page): add a dropdown menu containing one item, a link to "/status"
route defined above.
* src/static/css/cuirass.css: Add new class to allow bootstrap dropdown menus
to work without javascript plugin.
* src/sql/upgrade-7.sql: New file.
* Makefile.am: Add it.
* src/cuirass/base.scm (create-build-outputs): New procedure,
(build-packages): call it,
(process-spec): add the new spec argument and pass it to create-build-outputs.
* src/cuirass/database.scm (db-add-build-product, db-get-build-product-path,
db-get-build-products): New exported procedures.
* src/cuirass/http.scm (respond-static-file): Move file sending to ...
(respond-file): ... this new procedure,
(url-handler): add a new "download/<id>" route, serving the requested file
with the new respond-file procedure. Also gather build products and pass them
to "build-details" for "build/<id>/details" route.
* src/cuirass/templates.scm (build-details): Honor the new "products" argument
to display all the build products associated to the given build.
* src/schema.sql (BuildProducts): New table,
(Specifications)[build_outputs]: new field.
* tests/database.scm: Add empty build-outputs spec.
* tests/http.scm: Ditto.
* examples/guix-jobs.scm: Ditto.
* examples/hello-git.scm: Ditto.
* examples/hello-singleton.scm: Ditto.
* examples/hello-subset.scm: Ditto.
* examples/random.scm: Ditto.
* doc/cuirass.texi (overview): Document it.
This commit does several things, the big change is to try and construct a
simpler query for SQLite. I'm not confident that SQLite's query planner can
look past handling the NULL parameters, so I think it could be helpful to try
and create a simpler query, both to avoid that problem if it exists, but also
move the complexity in to Guile code, which I think is a bit more manageable.
The way ordering is handled is also changed. Order is one of the filters,
although it's not a filter, and some of the other filters also influenced the
order. I think there are things still to fix/improve with the handling of
ordering, but at least this commit just has the ordering happen once in the
query.
* src/cuirass/database.scm (filters->order): Remove procedure, inline in to
db-get-builds.
(db-get-builds): Change query generation in an attempt to make it easier to
tune the queries for performance.
Add a "optimize" step that occurs when starting up the main Curiass
process. Currently this does two things, but could be extended to do more.
The "PRAGMA optimize;" command prompts SQLite to ANALYZE tables where that
might help. The "PRAGMA wal_checkpoint(TRUNCATE);" command has SQLite process
any unprocessed changes from the WAL file, then truncate it to 0 bytes. I've
got no data to suggest this helps with performance, but I'm hoping that going
from a large WAL file to a small one occasionally might be useful.
* src/cuirass/database.scm (db-optimize): New procedure.
* bin/cuirass.in (main): Run it.
* src/cuirass/database.scm (db-get-inputs, db-get-checkouts): Export.
* src/cuirass/http.scm (evaluation-html-page): Pass #:checkouts and
#:inputs to 'evaluation-build-table'.
* src/cuirass/templates.scm (evaluation-build-table): Add #:checkouts
and #:inputs. Emit a table with "Input" and "Commit" columns.
The number of threads is copied from bin/cuirass.in. When you have at least
two processors, this will allow database queries to be executed in parallel.
With some crude testing using the Apache HTTP server benchmarking tool (ab
from the httpd package), the max request latency does seem to drop when
multiple threads are used, especially when the database queries are slow (I
tested by adding usleep to the worker thread code).
* src/cuirass/database.scm (with-database): Pass #:parallelism to
make-worker-thread-channel.
While this is a generic method, and initializer function will give the
flexibility required to create multiple worker threads for performing SQLite
queries, each with it's own database connection (as a result of calling the
initializer once for each thread). Without this change, they'd all have to use
the same connection, which would not work.
* src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take
an initializer, rather than arguments directly.
* src/cuirass/database.scm (with-database): Adjust to call
make-worker-thread-channel with an initializer.
* tests/database.scm (db-init): Change to use make-worker-thread-channel
initializer.
* tests/http.scm (db-init): Change to use make-worker-thread-channel
initializer.
As far as I'm aware, it's necessary to use a separate thread for interacting
with SQLite as one of the threads used for fibers will be blocked while the
SQLite query is running.
This doesn't mean all queries have to be executed one at a time though,
providing the queries are executed outside the threads used by fibers, and a
single connection isn't used in multiple threads.
These changes start to move in this direction, first by just changing the
terminology.
* src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change
with-db-critical-section to with-db-worker-thread.
* src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to
with-db-worker-thread.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-get-inputs, db-get-specification, db-add-evaluation,
db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output,
db-add-build, db-update-build-status!, db-get-output, db-get-outputs,
db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id,
db-add-event, db-get-events, db-delete-events-with-ids-<=-to,
db-get-pending-derivations, db-get-checkouts, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluations-id-max,
db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change
from using with-db-critical-section to
with-db-worker-thread.
(with-database): Change syntax rule to use make-worker-thread-channel,
renaming from make-critical-section.
* src/cuirass/utils.scm (%critical-section-args): Rename parameter to
%worker-thread-args.
(make-critical-section): Rename to make-worker-thread-channel, and adjust
parameter and docstring.
(call-with-critical-section): Rename to call-with-worker-thread and adjust
parameter.
(with-critical-section): Rename to with-worker-thread, and adjust to call
call-with-worker-thread.
* tests/database.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* tests/http.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
The internal rowid's are used for builds as you can request builds by using
the rowid in the URL.
The motivation here is to enable running VACUUM operations in SQLite, without
risking the rowid's for Builds changing. It would be bad if they change, as
they're used in the URL's for builds.
* src/schema.sql (Builds): Add id column.
* src/curiass/dataabse.scm (db-add-build): Change PRIMARYKEY constraint to
UNIQUE constraint.
* src/sql/upgrade-6.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
Mathieu Othacehe
Christopher Baines
Mathieu Othacehe
Ludovic Courtès
Ricardo Wurmus