2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/guix-cuirass.git synced 2023-12-14 06:03:04 +01:00
Commit graph

1110 commits

Author SHA1 Message Date
Ludovic Courtès 317d8fc812
Update TODO. 2023-09-26 12:18:20 +02:00
Ludovic Courtès 0e2252176a
remote-server: Remove ‘%fetch-queue-size’ global variable.
* src/cuirass/scripts/remote-server.scm (%fetch-queue-size): Remove.
(fetch-worker): Define ‘queue-size’ and spawn a fiber to log its size.
(spawn-periodic-updates-fiber): Remove reference to ‘%fetch-queue-size’.
2023-09-26 12:10:58 +02:00
Ludovic Courtès 9d35eced9c
remote-server: Trim module imports.
* src/cuirass/scripts/remote-server.scm: Trim module imports.
2023-09-26 11:48:40 +02:00
Ludovic Courtès 15e9301a1f
parameters: Remove ‘%debug’ in favor of ‘current-logging-level’.
* src/cuirass/parameters.scm <top level>: Re-export ‘current-logging-level’.
(%debug): Remove.
* src/cuirass/scripts/remote-server.scm: Remove all (%debug) conditionals.
2023-09-26 11:35:25 +02:00
Ludovic Courtès e62012bae9
remote-server: Use a channel instead of a zmq inproc socket for downloads.
* src/cuirass/scripts/remote-server.scm (zmq-fetch-workers-endpoint)
(zmq-fetch-worker-socket, start-fetch-worker): Remove.
(fetch-worker, spawn-fetch-worker): New procedures.
(zmq-start-proxy): Add ‘fetch-worker’ parameter.  Remove
‘fetch-socket’.  Send message on ‘fetch-worker’ instead of
‘fetch-socket’.
(cuirass-remote-server): Call ‘spawn-fetch-worker’ only once.  Pass the
result to ‘zmq-start-proxy’.
2023-09-26 11:16:46 +02:00
Ludovic Courtès 9c25f943ed
base: ‘channel-update-service’ correctly reports system errors.
* src/cuirass/base.scm (channel-update-service): Add missing argument in
'system-error handling, and improve message.
2023-09-25 18:06:20 +02:00
Ludovic Courtès 06c33a8f9c
remote: Fix docstring of ‘zmq-socket->port’.
* src/cuirass/remote.scm (zmq-socket->port): Fix docstring.
2023-09-25 14:04:05 +02:00
Ludovic Courtès 7688e9888e
remote: ‘send-log’ ignores ‘close-port’ exceptions.
Previously, ‘close-port’ could throw, leading the calling fiber to
terminate prematurely.

* src/cuirass/remote.scm (send-log): Ignore errors when closing
COMPRESSED from the ‘catch’ handler.
2023-09-25 09:56:32 +02:00
Ludovic Courtès b97ea9d737
evaluate: Builds inherit their priority from the spec.
Previously, builds would all get the default priority, meaning that jobset
priority would effectively ignored when scheduling builds via
‘db-get-pending-build’.

* src/cuirass/scripts/evaluate.scm (user-alists->builds): Change
‘specification-name’ parameter to ‘spec’.  Set the ‘priority’ field of <build>.
(inferior-evaluation): Adjust call accordingly.
2023-09-22 11:21:29 +02:00
Ludovic Courtès f5c2bafc63
database: ‘db-get-pending-build’ returns older builds first.
* src/cuirass/database.scm (db-get-pending-build): Sort ‘timestamp’
in ascending order.  Clarify docstring.
* tests/database.scm (make-dummy-build): Add #:system, #:jobset, #:priority,
and #:timestamp.
("db-get-builds no-dependencies"): Remove.
("db-get-pending-build"): New test.
2023-09-22 09:17:05 +02:00
Ludovic Courtès c9914f90ea
remote-worker: Clarify ‘--systems’ argument.
* src/cuirass/scripts/remote-worker.scm (show-help): Clarify SYSTEMS.
2023-09-22 00:14:45 +02:00
Ludovic Courtès 8c6fe0be32
remote-server: Pop builds for any worker-supported system.
Previously, for workers supporting multiple systems, this would pick a
system at random and return #f, even if pending builds are available for
one of the other systems supported by the worker.  Thus, it would
practically divide throughput by N for a worker supporting N systems.

* src/cuirass/scripts/remote-server.scm (random-seed, shuffle): New
procedures.
(pop-build): Change to return a build for *any* of the systems supported
by WORKER.
2023-09-21 23:55:36 +02:00
Ludovic Courtès be79b1e406
templates: Show build machine and/or worker on build page.
Fixes <https://issues.guix.gnu.org/66059>.

* src/cuirass/templates.scm (build-details): Show build machine.

Reported-by: Maxim Cournoyer <maxim.cournoyer@gmail.com>
2023-09-20 11:31:27 +02:00
Ludovic Courtès 6db8d6c5a9
templates: Show spec name on build page.
* src/cuirass/templates.scm (build-details): Show the spec name next to
the evaluation number.
2023-09-20 11:19:38 +02:00
Ludovic Courtès ae7ff698e0
base: Add comment giving an overview of the actors.
* src/cuirass/base.scm: Add commentary.
2023-09-17 15:39:57 +02:00
Ludovic Courtès 56902e7bc8
base: Define new “builder” actor with two implementations.
* src/cuirass/base.scm (%build-remote?): Remove.
(restart-builds): Add BUILDER argument; sent it the build request.
(build-packages): Remove.
(local-builder, spawn-local-builder)
(remote-builder, spawn-remote-builder): New procedures.
(jobset-evaluator): Add BUILDER argument.  Inline some of what was in
‘build-packages’.  Send build request to BUILDER.
(spawn-jobset-evaluator): Add #:builder and honor it.
* src/cuirass/scripts/register.scm (cuirass-register): Remove reference
to ‘%build-remote?’.  Cap THREADS at 8.  Define ‘builder’ and pass it to
‘spawn-jobset-evaluator’ and ‘restart-builds’.
2023-09-13 19:00:30 +02:00
Ludovic Courtès b0f93551bc
Move store and GC helpers from (cuirass base) to (cuirass store).
* src/cuirass/base.scm (default-gc-root-directory, %gc-root-directory)
(%gc-root-ttl, gc-roots, gc-root-expiration-time)
(register-gc-root, register-gc-roots)
(non-blocking-port, ensure-non-blocking-store-connection)
(with-store/non-blocking, process-build-log, build-derivations&): Move
to…
* src/cuirass/store.scm: … here.  New file.
* src/cuirass/scripts/remote-server.scm: Adjust accordingly.
* src/cuirass/scripts/remote-worker.scm: Likewise.
* src/cuirass/scripts/register.scm: Likewise.
* Makefile.am (dist_pkgmodule_DATA): Add ‘src/cuirass/store.scm’.
2023-09-13 19:00:12 +02:00
Ludovic Courtès b9f7f6e4a1
base: Add pages for structure.
* src/cuirass/base.scm: Add pages and comments.
(evaluate, evaluation-log-file): Move where they belong.
2023-09-13 19:00:12 +02:00
Ludovic Courtès f6645164ba
http: /admin/specification/edit notifies jobset via the bridge.
* src/cuirass/http.scm (url-handler): Send ‘update-jobset’ message to
BRIDGE.
* src/cuirass/scripts/register.scm (bridge): Handle ‘update-jobset’
messages.
2023-09-13 19:00:12 +02:00
Ludovic Courtès 0ddd43fd30
base: Implement ‘update-jobset’.
* src/cuirass/base.scm (jobset-monitor): Thread SPEC through the loop.
Introduce ‘perform-update’.  Implement ‘update-spec’ message handling.
(jobset-registry): Handle ‘update’ messages.
(update-jobset): New procedure.
2023-09-13 19:00:12 +02:00
Ludovic Courtès 7fdd780623
base: Move evaluations to a separate actor.
* src/cuirass/base.scm (start-evaluation, jobset-evaluator)
(spawn-jobset-evaluator): New procedures.
(jobset-monitor): Add #:evaluator.  Replace inline evaluation with a
message to EVALUATOR.
(spawn-jobset-monitor): Add #:evaluator and pass honor it.
(jobset-registry, spawn-jobset-registry): Likewise.
* src/cuirass/scripts/register.scm (cuirass-register): Call
‘spawn-jobset-evaluator’ and pass it to ‘spawn-jobset-registry’.
2023-09-13 19:00:11 +02:00
Ludovic Courtès c2ac72da03
http: Add /jobset/SPEC/hook/evaluate route to trigger a jobset.
* src/cuirass/base.scm (%jobset-trigger-rate-window)
(%jobset-trigger-maximum-rate): New variables.
(jobset-monitor): Replace 'sleep' call with 'get-message*'.  Define
'recent?' and 'rate'; use it to check whether the trigger rate is
exceeded.
* src/cuirass/http.scm (url-handler): Add /jobset/SPEC/hook/evaluate route.
* src/cuirass/scripts/register.scm (bridge): Handle 'trigger-jobset'
requests.
(show-help): Update ‘--interval’ description.
(cuirass-register): Increase default interval.
* doc/cuirass.texi (Invocation): Update ‘--interval’ documentation; add
a note documenting the push hook.
2023-09-13 15:44:16 +02:00
Ludovic Courtès 84e2a638ab
utils: Add 'get-message*'.
Taken from the Shepherd.

* src/cuirass/utils.scm (get-message*): New procedure.
2023-09-12 11:51:19 +02:00
Ludovic Courtès 7d86c5123c
http: Send jobset registration requests to the bridge.
* src/cuirass/http.scm (url-handler): Add 'bridge' parameter.
In "/admin/specification/add" route, write to BRIDGE.
(run-cuirass-server): Add #:bridge-socket-file-name.  When true, open
connection to the bridge.  Pass it to 'url-handler'.
* tests/http.scm ("cuirass-run"): Pass #:bridge-socket-file-name to
'run-cuirass-server'.
2023-09-12 11:51:19 +02:00
Ludovic Courtès d2794943d9
register: Listen for commands on a "bridge" socket.
* src/cuirass/base.scm (%bridge-socket-file-name): New variable.
* src/cuirass/scripts/register.scm (open-bridge-socket, bridge)
(spawn-bridge): New procedures.
(cuirass-register): Call 'spawn-bridge'.
2023-09-12 11:51:19 +02:00
Ludovic Courtès e4233a94ea
config: Define '%runstatedir'.
* configure.ac: Require 2.71.
* Makefile.am (do_subst): Substitute $(runstatedir).
* src/cuirass/config.scm.in (%runstatedir): New variable.
* src/cuirass/base.scm (%cuirass-run-state-directory): New variable.
* guix.scm <native-inputs>: Use AUTOCONF-2.71.
2023-09-12 11:51:19 +02:00
Ludovic Courtès 8fa6bcf1b9
base: Split jobset evaluation into several actors.
* src/cuirass/base.scm (process-specs): Remove.
(channel-update-service, spawn-channel-update-service)
(jobset-monitor, spawn-jobset-monitor)
(jobset-registry, spawn-jobset-registry)
(lookup-jobset, register-jobset): New procedures.
* src/cuirass/http.scm (url-handler): Add FIXMEs.
* src/cuirass/scripts/register.scm (cuirass-register): Remove 'build'
fiber and call 'spawn-jobset-registry' instead.  Error out on
'one-shot?'.
2023-09-12 11:51:19 +02:00
Ludovic Courtès 9b227abd29
http: Correctly render build products as JSON.
Fixes a bug introduced in c445d2d642 where
<build-product> records would end up being passed to ‘scm->json’.

* src/cuirass/http.scm (build->hydra-build): Convert <build-product>
records to alists.
2023-09-12 11:50:29 +02:00
Ludovic Courtès c20ca601c2
templates: Fix possible division by zero when rendering spec table.
Division by zero could happen in the unlikely case of a successful
evaluation containing zero jobs.

* src/cuirass/templates.scm (specifications-table)[summary->percentage]:
Check whether TOTAL is zero before dividing.
2023-09-12 10:40:13 +02:00
Ludovic Courtès 613188072c
templates: Avoid type error when displaying a jobset without evaluations.
Previously we’d pass #f to ‘eval-summary’ for jobsets that do not have
any successful evaluation.

* src/cuirass/templates.scm (specifications-table): Correctly handle
‘spec->latest-eval-ok’ returning #f.
2023-09-07 16:45:49 +02:00
Ludovic Courtès 647f13f0a9
http: /api/evaluations returns the full list of checkouts of each eval.
Previously it would only return checkouts changed compared to the
previous evaluation of that jobset.

* src/cuirass/http.scm (evaluation->json-object): Use ‘latest-checkouts’
instead of ‘evaluation-checkouts’.
* tests/http.scm (evaluations-query-result): Adjust accordingly.
2023-09-06 23:45:37 +02:00
Ludovic Courtès a057228492
database: Document the pitfalls of ‘db-get-checkouts’.
* src/cuirass/base.scm (latest-checkouts): Improve docstring.
* src/cuirass/database.scm (db-get-checkouts): Add docstring.
2023-09-06 23:39:25 +02:00
Ludovic Courtès 7a9194b15e
http: Really fix /jobsets for non-trivial build types.
* src/cuirass/http.scm (specification->json-object): Fix the non-trivial
cases for ‘build’.
2023-09-06 14:46:30 +02:00
Ludovic Courtès d4d0368cb8
http: /jobsets properly renders non-trivial build types.
Previously we’d get 500 for anything other than a symbol.

* src/cuirass/http.scm (specification->json-object): Handle build types
other than symbols describing a subset.
2023-09-06 12:33:35 +02:00
Ludovic Courtès 82c785d4f3
templates: Tolerate missing checkout info.
* src/cuirass/templates.scm (checkout-table): Tolerate CHECKOUT as #f.
2023-09-05 22:44:07 +02:00
Ludovic Courtès 64d7ad1109
database: ‘db-get-evaluations-absolute-summary’ returns a full summary.
* src/cuirass/database.scm (db-get-evaluations-absolute-summary): Extend
SQL query and fill out all the <evaluation-summary> fields.
* tests/database.scm ("db-get-evaluation-absolute-summary"): Check ‘status’.
("db-get-evaluations-absolute-summary"): Expect three results.
2023-09-05 16:04:03 +02:00
Ludovic Courtès 0930669024
templates: Fix broken evaluation display on the spec page.
Fixes a bug introduced in c445d2d642
whereby we’d call ‘evaluation-badges’ with #f as its first argument, due
to the last of a summary for LAST-EVAL.

* src/cuirass/templates.scm (specifications-table): When
LAST-EVAL-STATUS-OK? is false, call ‘broken-evaluation-badge’.
2023-09-05 15:08:49 +02:00
Ludovic Courtès c74a2a1332
templates: Factorize badges for broken evaluations.
* src/cuirass/templates.scm (broken-evaluation-badge): New procedure…
(evaluation-badges): … extracted from here.  Use it.
2023-09-05 15:08:43 +02:00
Ludovic Courtès 2a63a01519
http: Update to use symbols, not keywords, as keys on search results.
Fixes a regression introduced in
77bf78ecf7.

* src/cuirass/http.scm (url-handler): In “/search/latest” and
“/search/latest/PRODUCT-TYPE”, use symbols instead of keywords as the
second argument to ‘assoc-ref’.
2023-09-03 16:54:51 +02:00
Ludovic Courtès 88b63e0f64
templates: Use record accessors in build detail page.
Fixes a regression introduced in
c445d2d642.

* src/cuirass/templates.scm (build-details): Use record accessors.
2023-09-03 16:46:39 +02:00
Ludovic Courtès 042efcdf7c
examples: random: Create jobs with dependencies.
* examples/random-manifest.scm (random-computed-file): Add ‘dependency’
parameter and honor it.
<top level>: Replace ‘unfold’ call with a loop; pass ‘dependency’
argument to ‘random-computed-file’.
2023-09-03 16:05:31 +02:00
Ludovic Courtès 951ce6c0c0
examples: random: Provide correct output in manifest.
* examples/random-manifest.scm (make-job): Add ‘output’ parameter.
<top level>: Pass third argument to ‘make-job’.
2023-09-03 15:54:09 +02:00
Ludovic Courtès c91971a063
templates: Properly type the case where the last evaluation failed.
Fixes a bug introduced in c445d2d642.

* src/cuirass/templates.scm (specifications-table)[eval-summary]:
Use the right record accessors.
Call ‘eval-summary’ on ‘last-eval’.
2023-09-03 15:29:31 +02:00
Ludovic Courtès 596b364f84
base: 'build-derivations&' UTF-8-encodes build logs.
* src/cuirass/base.scm (build-derivations&): Set the encoding and
conversion strategy of OUTPUT.
2023-09-01 18:04:38 +02:00
Ludovic Courtès fda885b513
examples: random: Add non-ASCII UTF-8 string to the build log.
* examples/random-manifest.scm (random-computed-file): Print a lambda.
2023-09-01 17:47:40 +02:00
Ludovic Courtès 2ee6dc6947
base: 'create-build-outputs' catches 'register-gc-root' errors.
Commit 9e897d84ed would lead the
"set-build-successful!" test to fail when running in build environments:

  In procedure symlink: No such file or directory: "/var/guix/gcroots/profiles/per-user/nixbld/cuirass/source"

This fixes this regression.

* src/cuirass/base.scm (create-build-outputs): Wrap 'register-gc-root'
in 'catch'.
2023-09-01 17:03:48 +02:00
Ludovic Courtès ea233c2ffd
tests: Reify exceptions instead of hanging.
Previously, exceptions raised in a test would lead Fibers to print
"Uncaught exception in task" and the test would then hang.  With this
change, tests actually fail.

* tests/database.scm (with-fibers): Wrap EXP... in
'with-exception-handler'.  When RESULT is an exception, rethrow it.
2023-09-01 16:06:18 +02:00
Ludovic Courtès 0571f954cb
Remove 'sexp->channel*'.
This procedure was introduced in
9e3e9b691c on the grounds that
'sexp->channel' in (guix channels) did not support the 'branch' field.
This was fixed in Guix commit
8f999e1ac26d939e1dfc8fbdfa9316fb412df21a (March 2021).

Furthermore, removing this procedure allows us to store channel
introductions in the database, which was not the case until now.

* src/cuirass/specification.scm (channel->sexp): Add comment.
(sexp->channel*): Remove.
* src/cuirass/database.scm (db-get-specifications): Use 'sexp->channel'.
2023-08-31 21:12:01 +02:00
Ludovic Courtès 9e897d84ed
'create-build-outputs' registers a GC root on build products.
Fixes <https://issues.guix.gnu.org/64317>.

* src/cuirass/base.scm (create-build-outputs): Add call to
'register-gc-root'.
2023-08-24 15:04:21 +02:00
Ludovic Courtès 103a6ec27b
http: Gracefully handle missing files in "/download".
Partly fixes <https://issues.guix.gnu.org/64317>.

* src/cuirass/http.scm (url-handler): Catch 'system-error around
'respond-file'; return 500 upon ENOENT.  Pass #:ttl to 'respond-file'.
2023-08-24 14:54:55 +02:00