Previously, when an exception was raised from
'handle-build-event' (e.g., a "database is locked" error), we'd throw,
thereby leaving PORT open and we'd never read from it again. Thus, the
corresponding 'guix-daemon' process would eventually get stuck in a
'write' call to that socket, and its build processes would stall.
* src/cuirass/base.scm (exception-reporter): New procedure.
(spawn-builds): Use it.
Fixes <https://bugs.gnu.org/30618>.
Reported by Andreas Enge <andreas@enge.fr>.
* bin/evaluate.in (fill-job): Remove.
(main): Remove 'database' command-line argument. Remove DB and its
uses. Write an (evaluation EVAL JOBS) sexp.
* src/cuirass/base.scm (evaluate)[augment-job]: New procedure.
Use it. Adjust to read (evaluation EVAL JOBS) sexp. Call
'db-add-evaluation' and 'db-add-derivation'.
Previously any derivation not directly built by Cuirass would be
considered as failed because 'handle-build-event' wouldn't see any build
event. Here we just make sure the build status recorded in the database
corresponds to reality.
* src/cuirass/base.scm (update-build-statuses!): New procedure.
(spawn-builds): Call it after 'build-derivations&'.
This is purely "defensive programming" since the default value for
'print-build-trace' is currently #t.
* src/cuirass/base.scm (with-store): Pass #:print-build-trace to
'set-build-options'.
* src/cuirass/database.scm (%sqlite-exec): Remove.
(sqlite-exec): Turn back into a procedure. Use 'sqlite-bind'. Add
'normalize' procedure and use it.
(db-add-specification, db-add-derivation, db-get-derivation)
(db-add-evaluation, db-add-build, db-update-build-status!)
(db-get-build, db-get-stamp, db-add-stamp): Use question marks in SQL
queries.
* src/cuirass/base.scm (build-packages)[register]: Make #:log
non-false.
* tests/database.scm (make-dummy-job): Add #:job-name, #:system,
#:nix-name, and #:eval-id. This is necessary because 'sqlite-bind'
would now translate #f to a real NULL (before it would translate to the
string "#f"...), and would thus report violations of the non-NULL
constraint.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
Works around <https://github.com/wingo/fibers/issues/19>.
The effect in practice would be that we'd usually not reach the
'close-pipe' call in 'evaluate', leaving zombie processes behind us,
never executing the continuation, and additionally spinning fast on a
sequence of epoll_wait/epoll_ctl calls.
* src/cuirass/base.scm <top level>: Monkey-patch (fibers internal).
We used to have 'build-derivations' write to the custom binary port
returned by 'build-event-output-port'. However, custom binary ports
constitute continuation barriers, thereby preventing fibers from being
suspended.
To make build log processing non-blocking, we therefore invert this
inversion of control and use a suspendable I/O procedure,
'read-line/non-blocking', when reading the build log.
* src/cuirass/base.scm (read-line/non-blocking, process-build-log)
(build-derivations&): New procedures.
(%newline, build-event-output-port): Remove.
(spawn-builds): Use 'build-derivations&' instead of 'build-derivations'
with 'build-event-output-port'.
Fixes a bug whereby 'build-event-output-port' would sometimes read more
than COUNT from BV, which would usually result in a 'decoding-error'
exception from 'utf8->string'.
* src/cuirass/base.scm (build-event-output-port)[write!]: Use
'bytevector-range' to honor COUNT and OFFSET.
Previously these calls would effectively suspend execution of all the
fibers until they had completed.
* src/cuirass/base.scm (process-specs): Wrap 'fetch-repository' and
'compile' calls in 'non-blocking'.
* src/cuirass/base.scm (handle-build-event)[valid?]: New procedure.
Use it when handling 'build-started', 'build-succeeded', and
'build-failed' events.
This works around a scalability issue in guix-daemon when passing a long
list of derivations to 'build-derivations'.
* src/cuirass/base.scm (spawn-builds): New procedure.
(restart-builds, build-packages): Use it.
Fixes a regression introduced in
d7306a4f48 whereby 'register' would pass
the empty list as #:outputs to 'db-add-build'.
* src/cuirass/base.scm (build-packages)[register]: Always include
outputs.
Previously STORE would be closed on the first context switch, before
'restart-builds' has completed, leading to an error (writing to a closed
port).
* src/cuirass/base.scm (restart-builds): Remove 'store' parameter. Wrap
body in 'with-store'.
* bin/cuirass.in (main): Adjust accordingly.
Now the database is updated as things are built, rather than after the
whole batch of derivation builds has completed.
* src/cuirass/base.scm (handle-build-event): Call
'db-update-build-status!'.
(build-packages)[register]: Set #:starttime and #:stoptime to 0.
Set #:status to 'scheduled'.
Define 'build-ids' and use it to determine the overall result.
* src/cuirass/utils.scm (define-enumeration): New macro.
* src/cuirass/database.scm (build-status): New macro.
* src/cuirass/base.scm (build-packages)[hydra-build-status]: Remove.
Use the 'build-status' macro instead.
* README: Mark Fibers as required.
* configure.ac: Check for Guile 2.2 only. Check for (fibers).
* bin/cuirass.in (main): Use (fibers). Run 'process-specs' and web
server in separate fibers.
* src/cuirass/base.scm (with-store): New macro.
(non-blocking-port): New procedure.
(evaluate): Use 'non-blocking-port'. Use 'read-string' followed by 'read'.
(process-specs): Move 'db-add-stamp' right after 'string=?' comparison.
Run evaluation and subsequent builds in a separate fiber.
* src/cuirass/http.scm (run-cuirass-server): Pass 'fibers as the second
argument to 'run-server'. Use 'log-message' instead of 'format'.
* src/cuirass/database.scm (with-database): Remove 'dynamic-wind'.
* src/cuirass/logging.scm: New file.
* Makefile.am (dist_pkgmodule_DATA): Add it.
* src/cuirass/base.scm (handle-build-event): Use 'log-message' instead
of 'log'.
* src/cuirass/base.scm (prepare-git): New procedure. Body moved from...
(process-specs): ... here. Remove it.
* bin/cuirass.in (main): Call 'prepare-git'.
* src/cuirass/base.scm (%newline): New variable.
(build-event-output-port, handle-build-event): New procedures.
(build-packages): Use 'handle-build-event'.
'GIT_SSL_CAINFO' designates a single-file certificate bundle, not a
directory; thus it must be passed as the second argument to
'set-tls-certificate-locations!'.
* src/cuirass/base.scm (process-specs): Pass $SSL_CERT_DIR as the first
argument to 'set-tls-certificate-locations!', and $GIT_SSL_CAINFO or
$SSL_CERT_FILE as the second argument.
integration.
* bin/evaluate.in (fill-job): New procedure.
(main): Use it to fill informations (nix-name, system) that will later be
added to database.
* doc/cuirass.texi (Database)[Derivation]: Add system and nix_name fields.
(Database)[Builds]: Add id, status, timestamp, starttime and stoptime
fields. Remove output field.
(Database)[Outputs]: New table describing the build outputs.
* src/cuirass/base.scm (build-packages): Add new fields to build object before
adding it to database.
* src/cuirass/database.scm (db-get-build, db-get-builds): New procedures to get
a build by id from database and a list of builds using filter parameters
respectively.
* src/schema.sql (Outputs) : New table.
(Derivations): Add system and nix_name columns.
(Builds): Remove output column and add id, status, timestamp, starttime and
stoptime columns.
When current-processor-count is used without (ice-9 threads) being used, Guile
complains with the following warning:
Import (ice-9 threads) to have access to `current-processor-count'.
* src/cuirass/base.scm: Use (ice-9 threads).
* src/cuirass/base.scm (report-git-error): New procedure.
(with-git-error-handling): New macro.
(process-specs): Use with-git-error-handling to catch and report git errors.
* build-aux/guix.scm (package)[inputs]: Add guile-git.
* configure.ac: Check for (git) module. Also check that (git) exports
git-error-message procedure.
* src/cuirass/base.scm (&evaluation-error): New condition type.
(evaluate): Report an &evaluation-error if eof-object? is true on
data read from port. Otherwise, suppose that data are correct and keep things
going.
(process-specs): Catch &evaluation-error, report the error in the log and keep
going.
* src/cuirass/base.scm (copy-repository-cache) : New procedure.
(fetch-repository): Use latest-repository-commit to fetch git
repository instead of raw git system commands.
(process-specs): Use fetch-repository to get a store directory
containing the repository described in SPEC, add copy it to cache with
"copy-repository-cache".
* configure.ac: Check (guix git) module presence.
* src/cuirass/utils.scm (mkdir-p, make-user-module,
call-with-temporary-directory, with-directory-excursion): Remove because
already defined in guix.
* tests/utils (with-directory-excursion): Remove associated test.
* src/cuirass/base.scm: Use (guix build utils) to provide procedure removed
from (cuirass utils).
* bin/evaluate.in: Ditto.
* bin/cuirass.in: Use "make-user-module" provided by (guix ui).
Fixes a regression introduced in
a42cf16fec whereby build logs would go the
Cuirass' stderr.
* src/cuirass/base.scm (build-packages): Parameterize
'current-build-output-port' to a void output port.
This improves parallelism when 'max-jobs' is large.
* src/cuirass/base.scm (build-packages): Rewrite to pass the complete
list of derivations to 'build-derivations' at once. Handle multiple outputs.
* src/schema.sql (Builds): Make "output" part of the primary key.
Co-authored-by: Mathieu Lirzin <mthl@gnu.org>
Fixes https://notabug.org/mthl/cuirass/issues/1.
* src/cuirass/base.scm (fetch-repository): Return #f when 'git clone' fails.
(process-specs): Test if commit is not #f before using its value.
Signed-off-by: Mathieu Lirzin <mthl@gnu.org>
* bin/cuirass.in (%options): Add "--load-path" and "-L" command line options.
(show-help): Adapt.
* src/cuirass/base.scm (%guix-package-path): New parameter.
(set-guix-package-path!): New procedure.
(evaluate): Call "evaluate" script with '%guix-package-path'.
* bin/evaluate.in (main): Match 'guix-package-path' command line argument and
handle it.
Signed-off-by: Mathieu Lirzin <mthl@gnu.org>
Ludovic Courtès
Danny Milosavljevic
Jan Nieuwenhuizen
Mathieu Othacehe
Ricardo Wurmus
Mathieu Othacehe
Mathieu Lirzin