Use the pre-unwind-handler of a catch clause instead of nesting
with-throw-handler inside a catch clause.
* src/web/server/fiberized.scm (with-ignored-disconnects): Remove it,
(client-loop): replace "with-ignored-disconnects" with a catch clause.
EPIPE and ECONNRESET errors are already ignored when thrown by "sendfile" and
"put-bytevector" procedures. It turns out "peek-char" can also cause such
errors. So ignore such errors within the whole procedure.
* src/web/server/fiberized.scm (client-loop): Wrap the whole procedure inside
"with-ignored-disconnects" instead of wrapping individually "sendfile" and
"put-bytevector" calls.
Handle requests such as "/build/?nr" by ignoring parameters without any
associated value.
* src/cuirass/http.scm (request-parameters): Ignore silently parameters
without an associated value.
* tests/http.scm: Add a corresponding test case.
The request "/build/2465370mdb%22fx'ggx/details" causes a select query on the
whole Builds table, because failed "string->number" conversion is not handled.
* src/cuirass/http.scm (url-handler): Handle erroneous id -> number conversion
on "/build/id/details" route.
The request "/build/1234)" causes a select query on the whole Builds table,
because failed "string->number" conversion is not handled.
* src/cuirass/http.scm (url-handler): Handle erroneous id -> number conversion
on "/build/id" route.
* src/cuirass/utils.scm (with-operation, get-message-with-timeout): New
procedures,
(call-with-worker-thread): add timeout and timeout-proc arguments.
* src/cuirass/database.scm (with-db-worker-thread): Pass a 5 seconds timeout
to call-with-worker-thread, and print a debug message on timeout expiration.
* src/web/server/fiberized.scm (with-ignored-disconnects): New macro
factorizing the catch clause ignoring client disconnection related errors. Use
it for both "sendfiles" and "put-bytevector" procedures.
Setting current-fiber to #f in %non-blocking will prevent put-message in the
new thread to try suspending itself, but will also cause the same behavior on
get-message, which is not desired.
* src/cuirass/utils.scm (%non-blocking): Reduce the scope of current-fiber
parameter to the newly created thread.
It happens that some left-over checkout entries refer to removed inputs. In
that case, input will be #f, causing commit-hyperlink to fail with the
following error message:
In cuirass/http.scm:
475:25 9 (url-handler _ _)
193:3 8 (evaluation-html-page ((#:id . 5103) (# . 0) (# . 39) ?) ?)
In cuirass/templates.scm:
612:14 7 (evaluation-build-table _ #:checkouts _ #:inputs _ # _ # ?)
In srfi/srfi-1.scm:
586:17 6 (map1 (((#:commit . "3a3e9f2bb586e79a7931163f0191d?") ?)))
In cuirass/templates.scm:
621:39 5 (_ _)
584:15 4 (commit-hyperlink #f "3a3e9f2bb586e79a7931163f0191df615?")
In web/uri.scm:
308:23 3 (string->uri _)
278:14 2 (string->uri-reference _)
In unknown file:
1 (regexp-exec #<regexp 7f3a76c05980> #f #<undefined> #<u?>)
In ice-9/boot-9.scm:
1669:16 0 (raise-exception _ #:continuable? _)
In procedure regexp-exec: Wrong type argument in position 2 (expecting string): #f
* src/cuirass/templates.scm (evaluation-build-table): Do not call
"commit-hyperlink" if the matching input could not be found.
When "spawn-builds" is called to restart builds, the spec is not known,
preventing build products from being created as reported here:
https://issues.guix.gnu.org/42523
Fix this issue by reading the specification in database in
"set-build-successful!" procedure.
* src/cuirass/database.scm (db-get-specification): New exported procedure,
(db-get-specifications): add an optional name argument.
* tests/database.scm (db-get-specification): Add a corresponding test-case.
* src/cuirass/base.scm (set-build-successful!): Remove spec argument and read
it directly from database instead,
(update-build-statuses!): also remove spec argument, adapt
set-build-successful! call accordingly,
(spawn-builds): remove spec argument and adapt handle-build-event and
update-build-statuses! calls accordingly,
(handle-build-event): remove spec argument, adapt
set-build-successful! call accordingly,
(build-packages): remove spec argument, adapt spawn-builds call accordingly,
(process-specs): adapt build-packages call.
Add a "status" dropdown menu in the navigation bar. For now this menu only
contains one item, a link to the "latest builds" page at "/status" location.
* src/cuirass/database.scm (db-get-builds): Add support for 'started status.
* src/cuirass/http.scm (url-handler): Add "/status" route.
* src/cuirass/templates.scm (running-builds-table): New procedure,
(html-page): add a dropdown menu containing one item, a link to "/status"
route defined above.
* src/static/css/cuirass.css: Add new class to allow bootstrap dropdown menus
to work without javascript plugin.
Make sure that build products are also created when a batch of derivations
finishes, and not only when single build success events are received.
Factorize build status update to success and build products creation into a
single procedure.
* src/cuirass/base.scm (set-build-successful!): New procedure,
(update-build-statuses!): call it here,
(handle-build-event): and here.
It happens that we receive build succeeded notifications for derivations that
are not stored in the Builds table. In that case, do not try to create build
products.
* src/cuirass/base.scm (handle-build-event): Check if the derivation is
registered in the Builds table.
This is a follow-up of 2280ae18eb. The gzipped
log files are supposed to be displayed directly in the browser and not
downloaded. Set "Content-Disposition" header to "inline" to ensure that.
* src/cuirass/http.scm (respond-gzipped-file): Set to "Content-Disposition" to
"inline".
When searching for successful builds in Cuirass, it is expected that the build
products are already available. Hence, make sure that build products are
computed before the build is marked as successful.
* src/cuirass/base.scm (spawn-builds): Add a spec argument,
(handle-build-event): ditto, and call create-build-outputs when the
build-succeeded event is received,
(create-build-outputs): take a single build argument instead of a builds list
and adapt accordingly,
(build-packages): pass spec argument to spawn-builds and remove
create-build-outputs call.
Remove the Builds_index from the schema, as it would have been removed in
error by the upgrade-6 upgrade. Add a specific index on the Builds status
field, as this helps with db-get-builds queries, and add an index on the
Outputs derivation field, as this helps with the db-get-outputs part of
db-get-builds.
* src/sql/upgrade-8.sql: New file.
* src/schema.sql: Update accordingly.
* src/cuirass/http.scm (respond-file): Send the file name as 'x-raw-file
header argument, instead of the raw file content,
(respond-gzipped-file): ditto. Also set 'content-disposition header.
* src/web/server/fiberized.scm (strip-headers, with-content-length): New procedures,
(client-loop): Check if 'x-raw-file is set. If it's the case, use sendfiles to
send the given file. Otherwise, keep the existing behaviour and send directly
the received bytevector.
* src/sql/upgrade-7.sql: New file.
* Makefile.am: Add it.
* src/cuirass/base.scm (create-build-outputs): New procedure,
(build-packages): call it,
(process-spec): add the new spec argument and pass it to create-build-outputs.
* src/cuirass/database.scm (db-add-build-product, db-get-build-product-path,
db-get-build-products): New exported procedures.
* src/cuirass/http.scm (respond-static-file): Move file sending to ...
(respond-file): ... this new procedure,
(url-handler): add a new "download/<id>" route, serving the requested file
with the new respond-file procedure. Also gather build products and pass them
to "build-details" for "build/<id>/details" route.
* src/cuirass/templates.scm (build-details): Honor the new "products" argument
to display all the build products associated to the given build.
* src/schema.sql (BuildProducts): New table,
(Specifications)[build_outputs]: new field.
* tests/database.scm: Add empty build-outputs spec.
* tests/http.scm: Ditto.
* examples/guix-jobs.scm: Ditto.
* examples/hello-git.scm: Ditto.
* examples/hello-singleton.scm: Ditto.
* examples/hello-subset.scm: Ditto.
* examples/random.scm: Ditto.
* doc/cuirass.texi (overview): Document it.
This commit does several things, the big change is to try and construct a
simpler query for SQLite. I'm not confident that SQLite's query planner can
look past handling the NULL parameters, so I think it could be helpful to try
and create a simpler query, both to avoid that problem if it exists, but also
move the complexity in to Guile code, which I think is a bit more manageable.
The way ordering is handled is also changed. Order is one of the filters,
although it's not a filter, and some of the other filters also influenced the
order. I think there are things still to fix/improve with the handling of
ordering, but at least this commit just has the ordering happen once in the
query.
* src/cuirass/database.scm (filters->order): Remove procedure, inline in to
db-get-builds.
(db-get-builds): Change query generation in an attempt to make it easier to
tune the queries for performance.
Add a "optimize" step that occurs when starting up the main Curiass
process. Currently this does two things, but could be extended to do more.
The "PRAGMA optimize;" command prompts SQLite to ANALYZE tables where that
might help. The "PRAGMA wal_checkpoint(TRUNCATE);" command has SQLite process
any unprocessed changes from the WAL file, then truncate it to 0 bytes. I've
got no data to suggest this helps with performance, but I'm hoping that going
from a large WAL file to a small one occasionally might be useful.
* src/cuirass/database.scm (db-optimize): New procedure.
* bin/cuirass.in (main): Run it.
* src/cuirass/templates.scm (%vcs-web-views): New variable.
(commit-hyperlink): New procedure.
(evaluation-build-table): Use it in the input/commit table.
* src/cuirass/database.scm (db-get-inputs, db-get-checkouts): Export.
* src/cuirass/http.scm (evaluation-html-page): Pass #:checkouts and
#:inputs to 'evaluation-build-table'.
* src/cuirass/templates.scm (evaluation-build-table): Add #:checkouts
and #:inputs. Emit a table with "Input" and "Commit" columns.
Previously, if an error occurred, the worker fiber simply never sends a
reply. In the case of HTTP requests to Cuirass, where an exception occurs when
performing a database query, the fiber handling the request blocks as it never
gets a response. I think that this has the potential to cause the process to
hit file descriptor limits, as the connections are never responded to.
This is fixed by responding with the details of the exception, and then
throwing it within the fiber that made the call.
* src/cuirass/utils.scm (make-worker-thread-channel): Catch exceptions when
calling proc.
(call-with-worker-thread): Handle receiving exceptions from the worker thread.
The number of threads is copied from bin/cuirass.in. When you have at least
two processors, this will allow database queries to be executed in parallel.
With some crude testing using the Apache HTTP server benchmarking tool (ab
from the httpd package), the max request latency does seem to drop when
multiple threads are used, especially when the database queries are slow (I
tested by adding usleep to the worker thread code).
* src/cuirass/database.scm (with-database): Pass #:parallelism to
make-worker-thread-channel.
Mathieu Othacehe
Mathieu Othacehe
Christopher Baines
Ludovic Courtès
Ricardo Wurmus