This avoids a situation whereby, when evaluating from a Guix checkout,
we'd have already loaded slightly different and incompatible (guix …)
modules.
Hydra's 'hydra-eval-guile-jobs' implemented the same solution as in this
patch already.
* bin/evaluate.in: Remove use of (cuirass …) and (guix …) modules.
(ref): New procedure.
(with-directory-excursion): New macro.
(main): Use 'ref'. Remove uses of Guix or Cuirass modules.
The previous load path was potentially incorrect since commit
2fe7ff87e2.
* src/cuirass/base.scm (evaluate)[tokenize, load-path]: New variables.
Assume #:load-path is colon-separated. Pass LOAD-PATH as the second
argument to 'evaluate'.
* doc/cuirass.texi (Database): Adjust documentation.
Previously we would make a SQL query that would return many build jobs,
and then call 'delete-duplicates' on that. This was extremely wasteful
because the list of returned by the query was huge leading to a heap of
several tens of GiB on a big database, and 'delete-duplicates' would
lead to more GC and it would take ages.
Furthermore, since 'delete-duplicates' is written in C as of Guile
2.2.3, it is uninterruptible from Fiber's viewpoint. Consequently, the
kernel thread running the 'restart-builds' fiber would never schedule
other fibers, which could lead to deadlocks--e.g., since fibers are
scheduled on a circular shuffled list of kernel threads, once every N
times, a web server fiber would be sent to that kernel thread and not be
serviced.
* src/cuirass/base.scm (shuffle-jobs): Remove.
(shuffle-derivations): New procedure.
(spawn-builds): Take a list of derivations instead of a list of jobs.
(restart-builds): Remove 'builds' parameter. Remove 'delete-duplicates'
call. Remove done/remaining partitioning.
(build-packages): Adjust to pass 'spawn-builds' a list of derivations.
* bin/cuirass.in (main): Remove computation of PENDING. Remove second
parameter in call to 'restart-builds'.
* bin/evaluate.in (main): Always pass an alist as the arguments to PROC,
containing at least 'file-name' and 'revision'.
* examples/random-jobs.scm (make-random-jobs): Display 'file-name' and
'revision' from ARGUMENTS.
This avoids copying things back and forth.
* src/cuirass/base.scm (fetch-repository): Add #:writable-copy?
parameter. Call 'make-writable-copy' when it's true.
(copy-repository-cache): Remove.
(make-writable-copy): New procedure.
(evaluate): Add 'source' parameter and pass it to the 'evaluate' program.
(process-specs): Define 'compile?'. Pass #:writable-copy? to
'fetch-repository'. Remove call to 'copy-repository-cache'. Remove
computation of the checkout directory name. Pass CHECKOUT to 'evaluate'.
* bin/evaluate.in (main): Replace 'cachedir' with 'source'. Remove
computation of the checkout directory name.
Fixes a regression introduced in
1bab5c4e56 whereby the
'status+submission-time' order would no longer be honored.
As a result, /api/queue would return the queue ordered by build IDs,
making it largely useless.
* src/cuirass/database.scm (db-get-builds): Remove 'order' and rename
'order-column-name' to 'order'. Add case for 'status+submission-time'.
* tests/database.scm ("database")["db-get-builds"]: Move below
"db-update-build-status!" test. Add case for the
'status+submission-time' order.
This reinstates c47dfdf82b and fixes the
issues that led to the revert in
b71f0cdca5.
Before that, 'run-server' would force sequential processing of client
requests one after another.
* src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop
instead of 'run-server'. Spawn a database fiber.
(with-database-access): New macro.
(handle-build-request): Expect 'db-channel' and use 'with-database-access'.
(handle-builds-request): Likewise.
(url-handler): Likewise.
Fixes a bug whereby some fibers would get a SQLITE_BUSY exception while
accessing the database: see
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30644#26>.
Suggested by Danny Milosavljevic <dannym@scratchpost.org>.
* src/cuirass/database.scm (wal-mode): Rename to...
(set-db-options): ... this. Add call to 'sqlite-exec' for
'busy_timeout'.
Fixes a regression introduced in
f083282fd3, whereby we'd return 0 as the
timestamp for everything in /api/queue.
* src/cuirass/http.scm (build->hydra-build): Make 'finished?' a
Boolean. Move 'bool->int' call in #:finished definition.
Previously, when an exception was raised from
'handle-build-event' (e.g., a "database is locked" error), we'd throw,
thereby leaving PORT open and we'd never read from it again. Thus, the
corresponding 'guix-daemon' process would eventually get stuck in a
'write' call to that socket, and its build processes would stall.
* src/cuirass/base.scm (exception-reporter): New procedure.
(spawn-builds): Use it.
Fixes <https://bugs.gnu.org/30618>.
Reported by Andreas Enge <andreas@enge.fr>.
* bin/evaluate.in (fill-job): Remove.
(main): Remove 'database' command-line argument. Remove DB and its
uses. Write an (evaluation EVAL JOBS) sexp.
* src/cuirass/base.scm (evaluate)[augment-job]: New procedure.
Use it. Adjust to read (evaluation EVAL JOBS) sexp. Call
'db-add-evaluation' and 'db-add-derivation'.
Fixes a regression introduced in
593cb7be10.
* src/cuirass/http.scm (request-parameters): Fix fallback case in
'match' form. Previously it would return a procedure in this case, as
returned by (const param), leading to a failure down the road in
'sqlite-bind-arguments' as could be seen by running tests/http.scm.
I've seen 'scandir' report #f once, even though that's theoretically
impossible.
* src/cuirass/logging.scm (log-monitoring-stats): Return '() if
'scandir' returns #f.
Previously it would be installed under cuirass/.
* Makefile.am (webmoduledir, webobjectdir): New variables.
(dist_pkgmodule_DATA): Remove src/web/server/fiberized.scm.
(dist_webmodule_DATA, nodist_webobject_DATA): New variables.
(CLEANFILES): Add $(nodist_webobject_DATA).
* examples/random-jobs.scm (random-derivation): Add 'suffix' parameter
and honor it.
(make-random-jobs): Pass a non-empty suffix to 'random-derivation'.
* src/cuirass/database.scm (db-add-derivation): Catch 'sqlite-error and
handle SQLITE_CONSTRAINT_PRIMARYKEY.
(SQLITE_CONSTRAINT_UNIQUE): New variable.
* tests/database.scm ("database")["db-add-derivation"]: Add extra call to
'db-add-derivation'.
Before that, 'run-server' would force sequential processing of client
requests one after another.
* src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop
instead of 'run-server'.
That makes it safe from SQL injection.
* src/cuirass/database.scm (db-get-builds): Rewrite to use question
marks in SQL queries and binding through '%sqlite-exec'.
* tests/database.scm ("database")["db-get-builds"]: Exercise 'WHERE'
clauses.
The new macro automatically takes care of inserting question marks in
the SQL queries, which in turn guarantees that there are always as many
question marks and arguments.
* src/cuirass/database.scm (sqlite-exec): Rename to...
(%sqlite-exec): ... this.
(sqlite-exec/bind, sqlite-exec): New macros.
(assq-refs): Remove.
(db-add-specification): Use the new 'sqlite-exec' form.
(db-get-specifications): Correctly deal with REV or TAG being #f.
(db-add-derivation, db-get-derivation, db-add-evaluation)
(db-add-build, db-update-build-status!, db-get-outputs)
(db-get-build, db-get-stamp, db-add-stamp): Adjust to the new
'sqlite-exec' form.
Previously any derivation not directly built by Cuirass would be
considered as failed because 'handle-build-event' wouldn't see any build
event. Here we just make sure the build status recorded in the database
corresponds to reality.
* src/cuirass/base.scm (update-build-statuses!): New procedure.
(spawn-builds): Call it after 'build-derivations&'.
Ludovic Courtès
Ludovic Courtès
Danny Milosavljevic