* src/cuirass/remote.scm: New file.
* src/cuirass/remote-server.scm: New file.
* src/cuirass/remote-worker.scm: New file.
* bin/remote-server.in: New file.
* bin/remote-worker.in: New file.
* Makefile.am (bin_SCRIPTS): Add new binaries,
(dist_pkgmodule_DATA): add new files,
(EXTRA_DIST): add new binaries,
(bin/remote-server, bin/remote-worker): new targets.
* .gitignore: Add new binaries.
* bin/cuirass.in (%options): Add "--build-remote" option,
(show-help): document it,
(main): honor it.
* src/cuirass/base.scm (with-build-offload-thread): New macro,
(%build-remote?, %build-offload-channel): new parameters,
(make-build-offload-thread): new procedure,
(build-derivations/offload): new procedure,
(restart-builds): use it to offload builds when %build-remote? is set,
(build-packages): ditto.
* src/sql/upgrade-17.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds): Add "machine" field.
* src/cuirass/database.scm (db-update-build-machine!): New procedure.
* src/cuirass/base.scm (handle-build-event): Use it to record the machine
performing the build.
This is a work-around: https://issues.guix.gnu.org/44742.
* src/cuirass/base.scm (fetch-inputs): Set "current-error-port" parameter to a
void port to prevent the fetch progression bar to appear.
Set the store socket as non-blocking so that fibers communicating with the
store don't get blocked as described here: https://issues.guix.gnu.org/43565.
* src/cuirass/base.scm (with-store): Set the store socket as non-blocking.
(build-derivations&): Unset current-read-waiter and current-write-waiter.
This is a follow-up of 761443bca6.
* src/cuirass/base.scm (fetch-inputs): Use map to replace the previous par-map
call.
(compile-checkouts): Also replace par-map call by map.
Fixes: <https://issues.guix.gnu.org/43757>.
* src/cuirass/base.scm (fetch-inputs): Call par-map from a separate thread as
this call is not suspendable.
* src/cuirass/database.scm (db-register-builds): Remove store argument and
assume that 'log and 'outputs properties are provided by the evaluation.
* src/cuirass/base.scm (build-packages): Adapt accordingly.
Make sure that all registration queries are done from within a single database
worker. Otherwise, when builds from multiple evaluations are registered at
the same time, some contention occurs communicating with workers.
* src/cuirass/base.scm (new-outputs?, build-packages): Move build registration
to ...
* src/cuirass/database.scm (db-register-builds): ... this new procedure.
(with-db-worker-thread-no-timeout): New procedure. Use it in
"db-register-builds" to avoid timeout messages.
* src/cuirass/base.scm (evaluate): Expect 'nix-name and 'system properties to
be provided by the evaluation.
(build-packages): Use the eval-id provided as argument.
Once the evaluation is over, the new builds are registered. This registration
tries to insert a new build for each derivation returned by the evaluation
phase. If the new build does not add a new output, the insertion query is then
rollbacked. This means that there are at least as many insertion queries as
new derivations.
SQlite allows at most one writer at a time, and even though we are using WAL
mode, performing a lot of insertions will reduce the reading perforances.
When multiple evaluations are performed in parallel, the large number of
concurrent insertion queries also causes contention.
To avoid those issues, check first in the "Outputs" table which derivations
are already registered. This means that most of the insertion queries will be
replaced by reading queries, that are much less expensive and more suitable
for Cuirass concurrent implementation.
* src/cuirass/base.scm (new-outputs?): New procedure.
(build-packages): Use it to insert only builds registering new outputs.
If a derivation is GC'd just before calling "spawn-builds",
"build-derivations: will throw an exception that will be catched. Then,
"update-build-statuses!" will call "derivation-path->output-paths" that will
throw another exception because the derivation does not exit. This exception
is not handled, causing Cuirass to crash.
2020-09-18T10:41:18 batch of builds (partially) failed: build of `/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv' failed (status: 1)
Backtrace:
In ice-9/boot-9.scm:
1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In unknown file:
10 (apply-smob/0 #<thunk 7ffff5cbd4a0>)
In ice-9/boot-9.scm:
718:2 9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
619:8 8 (_ #(#(#<directory (guile-user) 7ffff58f8f00>)))
In ice-9/boot-9.scm:
2806:4 7 (save-module-excursion _)
4351:12 6 (_)
In cuirass/base.scm:
562:10 5 (spawn-builds #<store-connection 256.99 7fffec03f910> _ #:max-batch-size _)
In srfi/srfi-1.scm:
634:9 4 (for-each #<procedure update! (drv)> ("/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv"))
In cuirass/base.scm:
474:4 3 (update! "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")
In guix/derivations.scm:
552:17 2 (derivation-path->output-paths "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")
In ice-9/ports.scm:
440:11 1 (call-with-input-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" #<procedure read-derivation (drv-port #:optional read-derivation-from-file)> #:binary _ …)
In unknown file:
0 (open-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" "r" #:encoding #f #:guess-encoding #f)
* src/cuirass/base.scm (update-build-statuses!): Catch
"derivation-path->output-paths" exceptions and set the build status to
"failed".
The Evaluation table currently has an 'in_progress' field. Distinction between
succeeded and failed evaluations are based on the presence of Builds records
for the evaluation. It it also not possible to distinguish aborted evaluations
from failed evaluations.
Rename 'in_progress' field to 'status'. The 'status' field can be equal to
'started', 'succeeded', 'failed' or 'aborted'.
* src/cuirass/database.scm (evaluation-status): New exported enumeration.
(db-set-evaluations-done, db-set-evaluation-done): Remove them.
(db-abort-pending-evaluations, db-set-evaluation-status): New exported procedures.
(db-add-evaluation, db-get-builds, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluation-summary): Adapt to use
'status' field instead of 'in_progress' field.
* src/cuirass/templates.scm (evaluation-badges): Ditto.
* src/schema.sql (Evaluations): Rename 'in_progress' field to 'status'.
* src/sql/upgrade-10.sql: New file.
* bin/cuirass.in (main): Use "db-abort-pending-evaluations" instead of
"db-set-evaluations-done".
* src/cuirass/base.scm (evaluate): Use "db-set-evaluation-status" instead of
"db-set-evaluations-done".
(build-packages): Use "db-set-evaluation-status" instead of
"db-set-evaluation-done".
* tests/database.scm (sqlite-exec): Adapt accordingly.
* tests/http.scm (evaluations-query-result): Ditto.
Commit time can be far away from the actual push time. Thus, use the checkout
time instead of the commit time to fill Checkouts 'timestamp' field.
* src/cuirass/base.scm (fetch-input): Return the checkout time instead of the
commit time.
src/cuirass/base.scm (fetch-input): Add the commit timestamp to the returned
association list,
(process-specs): Pass a timestamp taken at procedure start and another one
taken after inputs are fetched to "db-add-evaluation" procedure. Once the
evaluation is over, call "db-set-evaluation-time" to save the evaluation
completion time.
src/cuirass/database.scm (db-set-evaluation-time): New procedure,
(db-add-checkout): Handle the "timestamp" field,
(db-add-evaluation): add "checkouttime" and "evaltime" arguments. Modify the
associated SQL query accordingly.
(db-get-builds): Use "Builds.timestamp" instead of "timestamp" as this field
is also part of the Evaluations table.
src/schema.sql (Checkouts): Add "timestamp" field,
(Evaluations): add "timestamp", "checkouttime" and "evaltime" fields.
src/sql/upgrade-9.sql: New file.
tests/database.scm (sqlite-exec): Adapt Evaluations table insertions to include
"timestamp", "checkouttime" and "evaltime" required fields.
When "spawn-builds" is called to restart builds, the spec is not known,
preventing build products from being created as reported here:
https://issues.guix.gnu.org/42523
Fix this issue by reading the specification in database in
"set-build-successful!" procedure.
* src/cuirass/database.scm (db-get-specification): New exported procedure,
(db-get-specifications): add an optional name argument.
* tests/database.scm (db-get-specification): Add a corresponding test-case.
* src/cuirass/base.scm (set-build-successful!): Remove spec argument and read
it directly from database instead,
(update-build-statuses!): also remove spec argument, adapt
set-build-successful! call accordingly,
(spawn-builds): remove spec argument and adapt handle-build-event and
update-build-statuses! calls accordingly,
(handle-build-event): remove spec argument, adapt
set-build-successful! call accordingly,
(build-packages): remove spec argument, adapt spawn-builds call accordingly,
(process-specs): adapt build-packages call.
Make sure that build products are also created when a batch of derivations
finishes, and not only when single build success events are received.
Factorize build status update to success and build products creation into a
single procedure.
* src/cuirass/base.scm (set-build-successful!): New procedure,
(update-build-statuses!): call it here,
(handle-build-event): and here.
It happens that we receive build succeeded notifications for derivations that
are not stored in the Builds table. In that case, do not try to create build
products.
* src/cuirass/base.scm (handle-build-event): Check if the derivation is
registered in the Builds table.
When searching for successful builds in Cuirass, it is expected that the build
products are already available. Hence, make sure that build products are
computed before the build is marked as successful.
* src/cuirass/base.scm (spawn-builds): Add a spec argument,
(handle-build-event): ditto, and call create-build-outputs when the
build-succeeded event is received,
(create-build-outputs): take a single build argument instead of a builds list
and adapt accordingly,
(build-packages): pass spec argument to spawn-builds and remove
create-build-outputs call.
* src/sql/upgrade-7.sql: New file.
* Makefile.am: Add it.
* src/cuirass/base.scm (create-build-outputs): New procedure,
(build-packages): call it,
(process-spec): add the new spec argument and pass it to create-build-outputs.
* src/cuirass/database.scm (db-add-build-product, db-get-build-product-path,
db-get-build-products): New exported procedures.
* src/cuirass/http.scm (respond-static-file): Move file sending to ...
(respond-file): ... this new procedure,
(url-handler): add a new "download/<id>" route, serving the requested file
with the new respond-file procedure. Also gather build products and pass them
to "build-details" for "build/<id>/details" route.
* src/cuirass/templates.scm (build-details): Honor the new "products" argument
to display all the build products associated to the given build.
* src/schema.sql (BuildProducts): New table,
(Specifications)[build_outputs]: new field.
* tests/database.scm: Add empty build-outputs spec.
* tests/http.scm: Ditto.
* examples/guix-jobs.scm: Ditto.
* examples/hello-git.scm: Ditto.
* examples/hello-singleton.scm: Ditto.
* examples/hello-subset.scm: Ditto.
* examples/random.scm: Ditto.
* doc/cuirass.texi (overview): Document it.
As far as I'm aware, it's necessary to use a separate thread for interacting
with SQLite as one of the threads used for fibers will be blocked while the
SQLite query is running.
This doesn't mean all queries have to be executed one at a time though,
providing the queries are executed outside the threads used by fibers, and a
single connection isn't used in multiple threads.
These changes start to move in this direction, first by just changing the
terminology.
* src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change
with-db-critical-section to with-db-worker-thread.
* src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to
with-db-worker-thread.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-get-inputs, db-get-specification, db-add-evaluation,
db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output,
db-add-build, db-update-build-status!, db-get-output, db-get-outputs,
db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id,
db-add-event, db-get-events, db-delete-events-with-ids-<=-to,
db-get-pending-derivations, db-get-checkouts, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluations-id-max,
db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change
from using with-db-critical-section to
with-db-worker-thread.
(with-database): Change syntax rule to use make-worker-thread-channel,
renaming from make-critical-section.
* src/cuirass/utils.scm (%critical-section-args): Rename parameter to
%worker-thread-args.
(make-critical-section): Rename to make-worker-thread-channel, and adjust
parameter and docstring.
(call-with-critical-section): Rename to call-with-worker-thread and adjust
parameter.
(with-critical-section): Rename to with-worker-thread, and adjust to call
call-with-worker-thread.
* tests/database.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* tests/http.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* src/cuirass/base.scm (&evaluation-error)[id]: New field.
(evaluate): Specify the 'id' field.
(process-specs): Show the log file name upon evaluation failure.
This fixes a longstanding issue where evalution output would splatter
over the /var/log/cuirass.log and be inscrutable.
* src/cuirass/base.scm (%cuirass-state-directory): New variable.
(evaluation-log-file): New procedure.
(evaluate)[log-file, log-pipe]: New variables.
Call 'spawn-fiber' with a logging fiber. Wrap 'open-pipe*' call into
'with-error-to-port'. Close 'log-pipe'.
It is used to know when a new evaluation must be triggered and to display
input changes.
* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-3.sql'.
* bin/cuirass.in (main): Call DB-SET-EVALUATION-DONE at startup to clear
'in-progress' evaluations.
* bin/evaluate.in (input-checkout, format-checkouts): Rename '#:name' to
'#:input'.
* doc/cuirass.texi (Stamps): Remove section.
(Checkouts): New section.
* src/cuirass/base.scm (fetch-input, fetch-inputs, compile-checkouts): Rename
'#:name' to '#:input'.
(evaluate): Remove the COMMITS argument. Add an EVAL-ID argument. Don't call
DB-ADD-EVALUATION because it was called sooner. Remove the EVAL-ID argument
to AUGMENT-JOB because it's a closure.
(build-packages): Add an EVAL-ID argument. Call DB-SET-EVALUATION-DONE once
all the derivations are registered.
(process-specs): Replace the stamping mechanism by the primary key constraint
of the Checkouts table: call "evaluate" only when DB-ADD-EVALUATION is true,
which means that at least one checkout was added. Change the EVALUATE and
BUILD-PACKAGES arguments accordingly.
* src/cuirass/database.scm (db-add-stamp, db-get-stamp): Remove procedures.
(db-set-evaluations-done, db-set-evaluation-done): New exported procedure.
(db-add-checkout): New procedure that returns #f if a checkout with the same
revision already exists.
(db-add-evaluation): Replace the EVAL argument with a SPEC-NAME and a
CHECKOUTS arguments. Insert the evaluation only if at least one checkout was
inserted. Return #f otherwise.
(db-get-checkouts): New procedure.
(db-get-evaluations, db-get-evaluations-build-summary): Handle the
'in_progress' column, remove the 'commits' column. Return the result of
DB-GET-CHECKOUTS as part of the evaluation.
* src/cuirass/templates.scm (input-changes, evaluation-badges): New
procedures.
(evaluation-info-table): Rename "Commits" to "Input changes". Use
INPUT-CHANGES to display the input changes that triggered the evaluation. Use
EVALUATION-BADGES to display a message indicating that the evaluation is in
progress.
* src/schema.sql (Stamps): Remove table.
(Checkouts): New table.
(Evaluations): Remove the 'commits' column. Add an 'in_progress' column.
* src/sql/upgrade-3.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-eval): Remove procedure.
(make-dummy-checkouts): New procedure.
("sqlite-exec"): Remove the 'commits' column. Add the 'in_progress' column.
("db-update-build-status!", "db-get-builds", "db-get-pending-derivations"):
Update the arguments of DB-ADD-EVALUATION accordingly.
* tests/http.scm (hash-table=?): Add support for lists of hash tables.
(evaluations-query-result): Replace '#:commits' with '#:checkouts'. Return a
list instead of returning one element, for symmetry.
("fill-db"): Add a new input so that the second checkout can refer to it.
Replace EVALUATION1 and EVALUATION2 with CHECKOUTS1 and CHECKOUTS2. Update
the arguments of DB-ADD-EVALUATION accordingly.
("/api/queue?nr=100"): Take the CAR of the EVALUATIONS-QUERY-RESULT list to
make it symmetrical with the other argument of HASH-TABLE=?.
Fixes <https://bugs.gnu.org/32234>.
* bin/cuirass.in (main): Keep only one WITH-DATABASE call around all fibers.
Remove all DB arguments.
* src/cuirass/base.scm (evaluate, update-build-statuses!, spawn-builds,
handle-build-event, build-packages): Remove all DB arguments.
(clear-build-queue, cancel-old-builds): Wrap in WITH-DB-CRITICAL-SECTION,
remove all DB arguments.
(restart-builds): Remove the NON-BLOCKING call, remove all DB arguments.
(process-specs): Remove all DB arguments, remove the WITH-DATABASE call.
* src/cuirass/database.scm (%db-channel): New parameter.
(with-db-critical-section): New macro.
(db-add-input, db-add-specification, db-get-inputs, db-get-specifications,
db-add-evaluation, db-add-build, db-update-build-status!, db-get-outputs,
db-get-builds, db-get-build, db-get-pending-derivations, db-get-stamp,
db-add-stamp, db-get-evaluations, db-get-evaluations-build-summary,
db-get-evaluations-id-min, db-get-evaluations-id-max, db-get-builds-min,
db-get-builds-max): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB arguments.
(with-database): Wrap BODY in PARAMETERIZE form that sets %DB-CHANNEL to the
channel returned by MAKE-CRITICAL-SECTION.
* src/cuirass/http.scm (handle-build-request, handle-builds-request): Remove
all DB arguments.
(url-handler): Remove all DB arguments, remove the DB-CHANNEL state, remove
the WITH-CRITICAL-SECTION calls.
(run-cuirass-server): Remove the DB arguments, remove the
MAKE-CRITICAL-SECTION call.
* src/cuirass/utils.scm (make-critical-section): Replace SPAWN-FIBER with
CALL-WITH-NEW-THREAD. Wrap body in PARAMETERIZE form that clears
CURRENT-FIBER.
* tests/database.scm (with-temporary-database, "db-add-specification",
"db-add-build", "db-update-build-status!", "db-get-builds",
"db-get-pending-derivations"): Remove the DB arguments.
("db-init"): Set the %DB-CHANNEL parameter to the channel returned by
MAKE-CRITICAL-SECTION, and return #t.
("database"): Set %DB-CHANNEL to #f during cleanup.
* tests/http.scm ("db-init"): Set the %DB-CHANNEL parameter to the channel
returned by MAKE-CRITICAL-SECTION, and return #t.
("cuirass-run", "fill-db"): Remove the DB arguments.
("http"): Set %DB-CHANNEL to #f during cleanup.
Fixes <https://bugs.gnu.org/32190>.
* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-2.sql'.
* doc/cuirass.texi (Derivations): Remove section.
(Builds): Update accordingly. Add columns from the Derivations table.
(Outputs): Replace 'id' with 'derivation'.
* src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.
(build-packages): Add columns that were in the Derivations table before. Only
build the derivations that were successfully registered, that is, those that
didn't exist in the Builds table. Give a derivation instead of a build id to
DB-GET-BUILD. Compute the number of failed jobs based on the derivations that
were added to the table, instead of the jobs.
* src/cuirass/database.scm (db-add-derivation, db-get-derivation): Remove
exported procedures.
(db-add-build): Catch SQLITE_CONSTRAINT_PRIMARYKEY error, which means that two
jobs produced the same derivation, and return #f in that case. Add columns
that were in the Derivations table before. Use 'derivation' as primary key
for the Outputs table.
(db-get-outputs): Use 'derivation' as identifier, instead of 'build-id'.
(filters->order): Replace 'id' with 'rowid'.
(db-get-builds): Add a 'derivation' filter. Replace 'id' with 'rowid'.
Remove the 'INNER JOIN Derivations'. Replace Derivations with Builds. Return
'derivation' in first position to make it clear that it's the primary key.
Pass DERIVATION instead of ID to DB-GET-OUTPUTS.
(db-get-build): Allow to take a derivation as argument. Use NUMBER? to
differentiate between derivations and ids.
(db-get-pending-derivations): Remove the 'SELECT DISTINCT' clause now that
derivations are unique. Remove the 'INNER JOIN Builds'.
(db-get-evaluations-build-summary, db-get-builds-min, db-get-builds-max):
Replace 'id' with 'rowid'.
* src/schema.sql (Derivations): Remove table.
(Outputs): Replace Builds.id with Builds.derivation.
(Builds): Use 'derivation' as primary key. Remove the 'id' column. Add
'job_name', 'system', 'nix_name' columns that were in the Derivations table
before.
(Builds_Derivations_index): Rename to Builds_index. Update accordingly.
(Derivations_index): Remove index.
* src/sql/upgrade-2.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-job, make-dummy-derivation): Remove
procedures.
(make-dummy-build): Add columns that were in MAKE-DUMMY-DERIVATION. Get the
DRV parameter to be mandatory because it's a primary key.
(%id): Remove parameter.
("db-add-derivation", "db-get-derivation"): Remove tests.
("db-add-build"): Expect #f, because it adds twice the same derivation. Pass
the derivation argument to MAKE-DUMMY-BUILD.
("db-update-build-status!"): Rename 'id' to 'derivation'. Pass the derivation
argument to MAKE-DUMMY-BUILD. Remove the DB-ADD-DERIVATION call.
("db-get-builds", "db-get-pending-derivations"): Pass the derivation argument
to MAKE-DUMMY-BUILD. Remove the DB-ADD-DERIVATION calls.
* tests/http.scm ("fill-db"): Remove DERIVATION1 and DERIVATION2, and put
their content in BUILD1 and BUILD2. Remove the DB-ADD-DERIVATION calls.
* Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql.
* bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that
was used afterwards as %GUIX-PACKAGE-PATH.
* bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path,
spec-package-path, format-checkouts): New procedures.
(%not-colon): Remove variable.
(main): Take the load path, package path and PROC from the checkouts that
result from the inputs. Format the checkouts before sending them to the
procedure. Remove the LOAD-PATH argument.
* doc/cuirass.texi (Overview, Database schema): Document the changes.
* examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm,
hello-subset.scm, random.scm}: Adapt to the new specification format.
* examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT.
(package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle
the new format of its return value.
* examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT.
Rename the checkout from 'random (which is a specification) to 'cuirass (which
is a checkout resulting from an input).
* src/cuirass/base.scm (fetch-repository): Rename to fetch-input. Rename SPEC
to INPUT. Return a checkout object instead of returning two values.
(evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of
SOURCE. Remove TOKENIZE and LOAD-PATH. Pass the CHECKOUTS instead of the
SOURCE to "evaluate". Remove %GUIX-PACKAGE-PATH. Build the EVAL object
instead of getting it from "evaluate".
(compile?, fetch-inputs, compile-checkouts): New procedures.
(process-specs): Fetch all inputs instead of only fetching one repository.
The result of that fetching operation is a list of CHECKOUTS whose COMMITS are
used as a STAMP.
(%guix-package-path, set-guix-package-path): Remove them.
* src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures.
(db-add-specification, db-get-specifications): Adapt to the new specification
format. Add/get all inputs as well.
(db-add-evaluation): Rename REVISION to COMMITS. Store COMMITS as space
separated commit hashes.
(db-get-builds): Rename REPO_NAME to NAME.
(db-get-stamp): Rename COMMIT to STAMP. Return #f when there is no STAMP.
(db-add-stamp): Rename COMMIT to STAMP. Deal with DB-GET-STAMP's new return
value.
(db-get-evaluations): Rename REVISION to COMMITS. Tokenize COMMITS.
* src/cuirass/utils.scm (%non-blocking): Export it.
* src/schema.sql (Inputs): New table that refers to the Specifications table.
(Specifications): Move input related fields to the Inputs table. Rename
REPO_NAME to NAME. Rename ARGUMENTS to PROC_ARGS. Rename FILE to PROC_FILE.
Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to
the Inputs table.
(Stamps): Rename REPO_NAME to NAME.
(Evaluations): Rename REPO_NAME to NAME. Rename REVISION to COMMITS.
(Specifications_index): Replace with Inputs_index.
* src/sql/upgrade-1.sql: New file.
* tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to
the new specifications format. Rename REVISION to COMMITS.
* tests/http.scm (evaluations-query-result, fill-db): Idem.
Fixes a regression introduced in
074b9d02f1 whereby we'd attempt to build
all of DRV at once.
* src/cuirass/base.scm (spawn-builds): Pass BATCH, not DRV, to
'build-derivations&' and to 'update-build-statuses!'.
The previous load path was potentially incorrect since commit
2fe7ff87e2.
* src/cuirass/base.scm (evaluate)[tokenize, load-path]: New variables.
Assume #:load-path is colon-separated. Pass LOAD-PATH as the second
argument to 'evaluate'.
* doc/cuirass.texi (Database): Adjust documentation.
Previously we would make a SQL query that would return many build jobs,
and then call 'delete-duplicates' on that. This was extremely wasteful
because the list of returned by the query was huge leading to a heap of
several tens of GiB on a big database, and 'delete-duplicates' would
lead to more GC and it would take ages.
Furthermore, since 'delete-duplicates' is written in C as of Guile
2.2.3, it is uninterruptible from Fiber's viewpoint. Consequently, the
kernel thread running the 'restart-builds' fiber would never schedule
other fibers, which could lead to deadlocks--e.g., since fibers are
scheduled on a circular shuffled list of kernel threads, once every N
times, a web server fiber would be sent to that kernel thread and not be
serviced.
* src/cuirass/base.scm (shuffle-jobs): Remove.
(shuffle-derivations): New procedure.
(spawn-builds): Take a list of derivations instead of a list of jobs.
(restart-builds): Remove 'builds' parameter. Remove 'delete-duplicates'
call. Remove done/remaining partitioning.
(build-packages): Adjust to pass 'spawn-builds' a list of derivations.
* bin/cuirass.in (main): Remove computation of PENDING. Remove second
parameter in call to 'restart-builds'.
This avoids copying things back and forth.
* src/cuirass/base.scm (fetch-repository): Add #:writable-copy?
parameter. Call 'make-writable-copy' when it's true.
(copy-repository-cache): Remove.
(make-writable-copy): New procedure.
(evaluate): Add 'source' parameter and pass it to the 'evaluate' program.
(process-specs): Define 'compile?'. Pass #:writable-copy? to
'fetch-repository'. Remove call to 'copy-repository-cache'. Remove
computation of the checkout directory name. Pass CHECKOUT to 'evaluate'.
* bin/evaluate.in (main): Replace 'cachedir' with 'source'. Remove
computation of the checkout directory name.
Mathieu Othacehe
Ludovic Courtès
Mathieu Othacehe
Christopher Baines
Clément Lassieur
Ludovic Courtès