2
0
Fork 0
mirror of git://git.savannah.gnu.org/guix/guix-cuirass.git synced 2023-12-14 06:03:04 +01:00
Commit graph

198 commits

Author SHA1 Message Date
Christopher Baines 46f73b6b7c Support returning build information by output.
Being able to take a derivation and query the build information is useful, but
in cases where there are multiple derivations that produce the same outputs,
the probability of getting the data back from Cuirass is reduced.

This is because Cuirass might not have build the exact derivation you have,
but a different derivation that produces the same outputs (this can commonly
happen when a related fixed output derivation changes).

Cuirass doesn't store derivations if they produce the same outputs as a
derivation it already knows about, so it can't determine if this is the
case. Therefore, provide a way of querying build results by output, rather
than derivation.

The motivation behind this is to make it easier to import build information in
to the Guix Data Service.

* src/cuirass/database.scm (db-get-output): New procedure.
* src/cuirass/http.scm (respond-output-not-found): New procedure.
(request-path-components): Handle /output/… requests.
* doc/cuirass.texi (Build information): Mention that you can get build
information by output.
2020-01-16 08:33:03 +00:00
Christopher Baines 267649c9f0 Support publishing evaluation events
* src/cuirass/database.scm (db-add-evaluation): Record the creation of new
evaluations as events.
(db-set-evaluation-done): Record when evaluations finish as an event.
* src/cuirass/http.scm (url-handler): Add a new /api/evaluation-events page.
2020-01-16 08:33:03 +00:00
Christopher Baines 12def48b3b Support publishing build events
Add a table to store events, which have a type and a JSON blob. These can be
used to record changes, this commit inserts events when new builds are
created, and when the status of builds change.

The EventsOutbox table is then used to track when events have been sent
out. This is done through the new cuirass-send-events script.

* Makefile.am (bin_SCRIPTS): Add bin/cuirass-send-events.
.gitignore: Add bin/cuirass-send-events.
(dist_pkgmodule_DATA): Add src/cuirass/send-events.scm.
(dist_sql_DATA): Add src/sql/upgrade-5.sql.
(EXTRA_DIST): bin/cuirass-send-events.in.
(bin/cuirass-send-events): New rule.
* bin/cuirass-send-events.in: New file.
* src/cuirass/send-events.scm: New file.
* src/sql/upgrade-5.sql: New file.
* src/cuirass/database.scm (changes-count): New procedure.
(db-update-build-status!): Call db-add-event after updating the build status.
(db-add-event): New procedure.
(db-add-build): Insert an event when a new build is inserted.
(db-delete-events-with-ids-<=-to): New procedure.
* src/schema.sql (Events): New table.
2020-01-16 08:32:52 +00:00
Ricardo Wurmus 8a618322b1
database: Add db-remove-specification.
* src/cuirass/database.scm (db-remove-specification): New procedure.
2019-10-30 09:19:28 +01:00
Ricardo Wurmus fed15b83b0
database: query->bind-arguments: Process ^ and $ characters.
* src/cuirass/database.scm (query->bind-arguments): Handle ^ and $ characters
in query strings.
2019-06-20 23:31:34 +02:00
Ricardo Wurmus c3ff064e97
database: query->bind-arguments: Ignore invalid query terms.
* src/cuirass/database.scm (query->bind-arguments): Ignore invalid query
terms.
2019-06-19 15:37:05 +02:00
Ricardo Wurmus efe7d36e64
http: Show number of builds.
* src/cuirass/database.scm (db-get-evaluation-summary): New procedure.
* src/cuirass/http.scm (url-handler): Display number of builds in tabs.
2019-06-19 10:22:51 +02:00
Ricardo Wurmus 767f34bd68
database: db-get-specifications: Order specs by name.
* src/cuirass/database.scm (db-get-specifications): Order rows by "name".
2019-06-18 23:36:37 +02:00
Ricardo Wurmus 9017b9fbc1
database: Support advanced query strings.
* src/cuirass/database.scm (query->bind-arguments): New procedure.
(db-get-builds-by-search): Support filtering by spec and system.
(db-get-builds-query-min, db-get-builds-query-max): Adjust.
2019-05-21 11:40:28 +02:00
Ricardo Wurmus ce114d8446
cuirass: Add search for builds.
* src/cuirass/database.scm (db-get-builds-by-search, db-get-builds-query-min,
db-get-builds-query-max): New procedures.
* src/cuirass/http.scm (handle-builds-search-request): New procedure.
(url-handler): Handle "search" route.
* src/cuirass/templates.scm (search-form): New variable.
(html-page): Use it.
(build-search-results-table): New procedure.
2019-05-17 22:40:12 +02:00
Ludovic Courtès d0ed3341d7 database: Re-throw the message of 'sqlite-error' exceptions.
Fixes <https://bugs.gnu.org/33407>.
Reported by Björn Höfling <bjoern.hoefling@bjoernhoefling.de>.

* src/cuirass/database.scm (catch-sqlite-error): Add missing MESSAGE
argument to 'throw'.
2018-11-16 21:49:02 +01:00
Ludovic Courtès 09afb02528 database: Factorize 'sqlite-error' handling.
* src/cuirass/database.scm (catch-sqlite-error): New macro.
(db-add-checkout, db-add-output, db-add-build): Use it instead of custom
'catch' block'.
2018-11-16 21:48:28 +01:00
Ludovic Courtès c76edac15d database: Remove duplicate #:in-progress entry.
* src/cuirass/database.scm (db-get-evaluations-build-summary): Remove
duplicate #:in-progress entry.
2018-11-07 15:28:05 +01:00
Clément Lassieur fe2b73c235
database: Return #f when one row is expected and there is none.
* src/cuirass/database.scm (expect-one-row): New procedure.
(db-get-build, db-get-evaluations-id-min, db-get-evaluations-id-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Use
it.
2018-09-30 17:26:12 +02:00
TSholokhova cbf8e13835
templates: Add a navigation bar.
* src/cuirass/database.scm (db-get-evaluation-specification): New exported
procedure.
* src/cuirass/http.scm (respond-html): Allow to pass CODE as argument.
(respond-html-eval-not-found): New procedure.
(url-handler): Fill navigation arguments.  Handle the case where the
evaluation doesn't exist.
* src/cuirass/templates.scm (navigation-items): New procedure.
(html-page): Add navigation bar.

Co-authored-by: Clément Lassieur <clement@lassieur.org>
2018-09-30 17:08:15 +02:00
TSholokhova f51bd3f554
templates: Allow to filter builds by status.
* src/cuirass/database.scm (db-get-builds): Add 'succeeded' and 'failed'
status filters.
(db-get-builds-min, db-get-builds-max): Add support for a new 'status'
argument.
* src/cuirass/http.scm (url-handler): Add a status argument for the '/eval/id'
endpoint.
* src/cuirass/templates.scm (evaluation-badges): Add links to the buttons.
(build-eval-table): Add a 'status' argument.

Co-authored-by: Clément Lassieur <clement@lassieur.org>
2018-09-30 13:10:56 +02:00
Clément Lassieur 4e661552c3
database: Add builds only if one of their outputs is new.
* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'.
* src/cuirass/database.scm (db-add-output): New procedure.
(db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and return #f if
DB-ADD-OUTPUT returned an empty list.
* src/schema.sql (Outputs): Set 'path' as primary key, instead of 'derivation,
name'.
* src/sql/upgrade-4.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-build): Use the #:OUTPUTS key.  Get default
OUTPUTS to depend on DRV.
("db-add-build-with-fixed-output"): New test.
2018-09-29 22:29:06 +02:00
Clément Lassieur 8d40c49170
database: Add a Checkouts table.
It is used to know when a new evaluation must be triggered and to display
input changes.

* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-3.sql'.
* bin/cuirass.in (main): Call DB-SET-EVALUATION-DONE at startup to clear
'in-progress' evaluations.
* bin/evaluate.in (input-checkout, format-checkouts): Rename '#:name' to
'#:input'.
* doc/cuirass.texi (Stamps): Remove section.
(Checkouts): New section.
* src/cuirass/base.scm (fetch-input, fetch-inputs, compile-checkouts): Rename
'#:name' to '#:input'.
(evaluate): Remove the COMMITS argument.  Add an EVAL-ID argument.  Don't call
DB-ADD-EVALUATION because it was called sooner.  Remove the EVAL-ID argument
to AUGMENT-JOB because it's a closure.
(build-packages): Add an EVAL-ID argument.  Call DB-SET-EVALUATION-DONE once
all the derivations are registered.
(process-specs): Replace the stamping mechanism by the primary key constraint
of the Checkouts table: call "evaluate" only when DB-ADD-EVALUATION is true,
which means that at least one checkout was added.  Change the EVALUATE and
BUILD-PACKAGES arguments accordingly.
* src/cuirass/database.scm (db-add-stamp, db-get-stamp): Remove procedures.
(db-set-evaluations-done, db-set-evaluation-done): New exported procedure.
(db-add-checkout): New procedure that returns #f if a checkout with the same
revision already exists.
(db-add-evaluation): Replace the EVAL argument with a SPEC-NAME and a
CHECKOUTS arguments.  Insert the evaluation only if at least one checkout was
inserted.  Return #f otherwise.
(db-get-checkouts): New procedure.
(db-get-evaluations, db-get-evaluations-build-summary): Handle the
'in_progress' column, remove the 'commits' column.  Return the result of
DB-GET-CHECKOUTS as part of the evaluation.
* src/cuirass/templates.scm (input-changes, evaluation-badges): New
procedures.
(evaluation-info-table): Rename "Commits" to "Input changes".  Use
INPUT-CHANGES to display the input changes that triggered the evaluation.  Use
EVALUATION-BADGES to display a message indicating that the evaluation is in
progress.
* src/schema.sql (Stamps): Remove table.
(Checkouts): New table.
(Evaluations): Remove the 'commits' column.  Add an 'in_progress' column.
* src/sql/upgrade-3.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-eval): Remove procedure.
(make-dummy-checkouts): New procedure.
("sqlite-exec"): Remove the 'commits' column.  Add the 'in_progress' column.
("db-update-build-status!", "db-get-builds", "db-get-pending-derivations"):
Update the arguments of DB-ADD-EVALUATION accordingly.
* tests/http.scm (hash-table=?): Add support for lists of hash tables.
(evaluations-query-result): Replace '#:commits' with '#:checkouts'.  Return a
list instead of returning one element, for symmetry.
("fill-db"): Add a new input so that the second checkout can refer to it.
Replace EVALUATION1 and EVALUATION2 with CHECKOUTS1 and CHECKOUTS2.  Update
the arguments of DB-ADD-EVALUATION accordingly.
("/api/queue?nr=100"): Take the CAR of the EVALUATIONS-QUERY-RESULT list to
make it symmetrical with the other argument of HASH-TABLE=?.
2018-08-27 15:38:44 +02:00
Clément Lassieur 4612a3a70f
database: Serialize all database accesses in a thread.
Fixes <https://bugs.gnu.org/32234>.

* bin/cuirass.in (main): Keep only one WITH-DATABASE call around all fibers.
Remove all DB arguments.
* src/cuirass/base.scm (evaluate, update-build-statuses!, spawn-builds,
handle-build-event, build-packages): Remove all DB arguments.
(clear-build-queue, cancel-old-builds): Wrap in WITH-DB-CRITICAL-SECTION,
remove all DB arguments.
(restart-builds): Remove the NON-BLOCKING call, remove all DB arguments.
(process-specs): Remove all DB arguments, remove the WITH-DATABASE call.
* src/cuirass/database.scm (%db-channel): New parameter.
(with-db-critical-section): New macro.
(db-add-input, db-add-specification, db-get-inputs, db-get-specifications,
db-add-evaluation, db-add-build, db-update-build-status!, db-get-outputs,
db-get-builds, db-get-build, db-get-pending-derivations, db-get-stamp,
db-add-stamp, db-get-evaluations, db-get-evaluations-build-summary,
db-get-evaluations-id-min, db-get-evaluations-id-max, db-get-builds-min,
db-get-builds-max): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB arguments.
(with-database): Wrap BODY in PARAMETERIZE form that sets %DB-CHANNEL to the
channel returned by MAKE-CRITICAL-SECTION.
* src/cuirass/http.scm (handle-build-request, handle-builds-request): Remove
all DB arguments.
(url-handler): Remove all DB arguments, remove the DB-CHANNEL state, remove
the WITH-CRITICAL-SECTION calls.
(run-cuirass-server): Remove the DB arguments, remove the
MAKE-CRITICAL-SECTION call.
* src/cuirass/utils.scm (make-critical-section): Replace SPAWN-FIBER with
CALL-WITH-NEW-THREAD.  Wrap body in PARAMETERIZE form that clears
CURRENT-FIBER.
* tests/database.scm (with-temporary-database, "db-add-specification",
"db-add-build", "db-update-build-status!", "db-get-builds",
"db-get-pending-derivations"): Remove the DB arguments.
("db-init"): Set the %DB-CHANNEL parameter to the channel returned by
MAKE-CRITICAL-SECTION, and return #t.
("database"): Set %DB-CHANNEL to #f during cleanup.
* tests/http.scm ("db-init"): Set the %DB-CHANNEL parameter to the channel
returned by MAKE-CRITICAL-SECTION, and return #t.
("cuirass-run", "fill-db"): Remove the DB arguments.
("http"): Set %DB-CHANNEL to #f during cleanup.
2018-08-16 19:19:23 +02:00
Clément Lassieur 4db99f647b
database: Merge Derivations into Builds table.
Fixes <https://bugs.gnu.org/32190>.

* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-2.sql'.
* doc/cuirass.texi (Derivations): Remove section.
(Builds): Update accordingly.  Add columns from the Derivations table.
(Outputs): Replace 'id' with 'derivation'.
* src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.
(build-packages): Add columns that were in the Derivations table before.  Only
build the derivations that were successfully registered, that is, those that
didn't exist in the Builds table.  Give a derivation instead of a build id to
DB-GET-BUILD.  Compute the number of failed jobs based on the derivations that
were added to the table, instead of the jobs.
* src/cuirass/database.scm (db-add-derivation, db-get-derivation): Remove
exported procedures.
(db-add-build): Catch SQLITE_CONSTRAINT_PRIMARYKEY error, which means that two
jobs produced the same derivation, and return #f in that case.  Add columns
that were in the Derivations table before.  Use 'derivation' as primary key
for the Outputs table.
(db-get-outputs): Use 'derivation' as identifier, instead of 'build-id'.
(filters->order): Replace 'id' with 'rowid'.
(db-get-builds): Add a 'derivation' filter.  Replace 'id' with 'rowid'.
Remove the 'INNER JOIN Derivations'.  Replace Derivations with Builds.  Return
'derivation' in first position to make it clear that it's the primary key.
Pass DERIVATION instead of ID to DB-GET-OUTPUTS.
(db-get-build): Allow to take a derivation as argument.  Use NUMBER? to
differentiate between derivations and ids.
(db-get-pending-derivations): Remove the 'SELECT DISTINCT' clause now that
derivations are unique.  Remove the 'INNER JOIN Builds'.
(db-get-evaluations-build-summary, db-get-builds-min, db-get-builds-max):
Replace 'id' with 'rowid'.
* src/schema.sql (Derivations): Remove table.
(Outputs): Replace Builds.id with Builds.derivation.
(Builds): Use 'derivation' as primary key.  Remove the 'id' column.  Add
'job_name', 'system', 'nix_name' columns that were in the Derivations table
before.
(Builds_Derivations_index): Rename to Builds_index.  Update accordingly.
(Derivations_index): Remove index.
* src/sql/upgrade-2.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-job, make-dummy-derivation): Remove
procedures.
(make-dummy-build): Add columns that were in MAKE-DUMMY-DERIVATION.  Get the
DRV parameter to be mandatory because it's a primary key.
(%id): Remove parameter.
("db-add-derivation", "db-get-derivation"): Remove tests.
("db-add-build"): Expect #f, because it adds twice the same derivation.  Pass
the derivation argument to MAKE-DUMMY-BUILD.
("db-update-build-status!"): Rename 'id' to 'derivation'.  Pass the derivation
argument to MAKE-DUMMY-BUILD.  Remove the DB-ADD-DERIVATION call.
("db-get-builds", "db-get-pending-derivations"): Pass the derivation argument
to MAKE-DUMMY-BUILD.  Remove the DB-ADD-DERIVATION calls.
* tests/http.scm ("fill-db"): Remove DERIVATION1 and DERIVATION2, and put
their content in BUILD1 and BUILD2.  Remove the DB-ADD-DERIVATION calls.
2018-08-16 19:19:23 +02:00
Clément Lassieur b4d058fc8d
database: Fix the builds limit issue.
Fixes <https://bugs.gnu.org/32300>.

* src/cuirass/database.scm (filters->order): New procedure.
(db-get-builds): Remove FORMAT-OUTPUT, CONS-OUTPUT, COLLECT-OUTPUTS,
FINISH-GROUP, SAME-GROUP?, GROUP-OUTPUTS procedures.  Remove the 'LEFT JOIN
Outputs' clause.  Use DB-GET-OUTPUTS for each build that was fetched.
2018-08-16 19:19:23 +02:00
Clément Lassieur 31a398f8a3
database: Put database in /var/lib to avoid removal at each boot.
Fixes <https://bugs.gnu.org/32330>.

* src/cuirass/database.scm (%package-database): Change default file name from
/var/run/cuirass/cuirass.db to /var/lib/cuirass/cuirass.db.
2018-08-16 19:19:14 +02:00
Clément Lassieur fb11e79c2e
database: Fix the evaluations limit issue.
* src/cuirass/database.scm (db-get-evaluations-build-summary): Use a 'LEFT
JOIN' instead of a 'JOIN', with 'Evaluations' on the left, so that evaluations
are displayed even when they don't have associated builds.
2018-08-04 16:34:27 +02:00
Clément Lassieur dc27cb369c
database: Remove unused DB-FORMAT-BUILD procedure.
* src/cuirass/database.scm (db-format-build): Remove it.
2018-08-04 16:33:59 +02:00
Clément Lassieur 4c044c658a
Rename 'repo-name' to 'specification'.
* src/cuirass/database.scm (db-get-builds): Rename 'repo-name' to
'specification'.
* src/cuirass/http.scm (build->hydra-build): Idem.
2018-08-04 16:32:22 +02:00
Clément Lassieur d36ec7a385
database: Limit SQL queries to 80 columns. 2018-08-04 16:31:34 +02:00
Clément Lassieur 99241ef1af
http: Change the paramater format from two-elements lists to pairs.
* src/cuirass/database.scm (assqx-ref): Remove exported procedure.
(db-get-builds, db-get-build): Adapt to new format.
* src/cuirass/http.scm (request-parameters): Use (cons key param) instead
of (list key param).
(url-handler): Adapt to new format.
* tests/database.scm ("db-get-builds"): Idem.
2018-07-30 00:26:31 +02:00
TSholokhova 675cd04a85
Add a web interface.
* Makefile.am (dist_sql_DATA): Add static files.
* src/cuirass/database.scm (assqx-ref): Export it.
(db-get-builds): Add 'evaluation' filter and filters for pagination.
(db-get-evaluations-build-summary, db-get-evaluations-id-min,
db-get-evaluations-id-max, db-get-builds-min, db-get-builds-max): New exported
procedures.
* src/cuirass/http.scm (%static-directory): New parameter.
(%page-size, %file-mime-types, %file-white-list): New variables.
(handle-build-request, handle-builds-request): Move the WITH-CRITICAL-SECTION
call out.
(url-handler): Add RESPOND-HTML, RESPOND-STATIC-FILE and RESPOND-NOT-FOUND
procedures.  Call WITH-CRITICAL-SECTION sooner for the '/build',
'/api/latestbuilds' and '/api/queue' routes.  Add '/', '/jobset/<name>',
'/eval/<id>', '/static/<path>' routes.  Use RESPOND-NOT-FOUND when the route
isn't found.
* src/cuirass/templates.scm: New file.
(html-page, specifications-table, evaluation-info-table,
build-eval-table): New exported procedures.
(pagination): New procedure.

Signed-off-by: Clément Lassieur <clement@lassieur.org>
2018-07-29 23:47:00 +02:00
Clément Lassieur 6f8dc0b616
database: Use SQLite in Multi-thread mode.
This disables mutexing on database connection and prepared statement objects,
thus making us responsible for serializing access to database connections and
prepared statements.  It may result in a performance improvement.

* src/cuirass/database.scm (db-init, db-open): Pass the SQLITE_OPEN_NOMUTEX
flag to SQLITE-OPEN.
2018-07-20 15:19:23 +02:00
Clément Lassieur 7b2f9e0de1
Add support for multiple inputs.
* Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql.
* bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that
was used afterwards as %GUIX-PACKAGE-PATH.
* bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path,
spec-package-path, format-checkouts): New procedures.
(%not-colon): Remove variable.
(main): Take the load path, package path and PROC from the checkouts that
result from the inputs.  Format the checkouts before sending them to the
procedure.  Remove the LOAD-PATH argument.
* doc/cuirass.texi (Overview, Database schema): Document the changes.
* examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm,
hello-subset.scm, random.scm}: Adapt to the new specification format.
* examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT.
(package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle
the new format of its return value.
* examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT.
Rename the checkout from 'random (which is a specification) to 'cuirass (which
is a checkout resulting from an input).
* src/cuirass/base.scm (fetch-repository): Rename to fetch-input.  Rename SPEC
to INPUT.  Return a checkout object instead of returning two values.
(evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of
SOURCE.  Remove TOKENIZE and LOAD-PATH.  Pass the CHECKOUTS instead of the
SOURCE to "evaluate".  Remove %GUIX-PACKAGE-PATH.  Build the EVAL object
instead of getting it from "evaluate".
(compile?, fetch-inputs, compile-checkouts): New procedures.
(process-specs): Fetch all inputs instead of only fetching one repository.
The result of that fetching operation is a list of CHECKOUTS whose COMMITS are
used as a STAMP.
(%guix-package-path, set-guix-package-path): Remove them.
* src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures.
(db-add-specification, db-get-specifications): Adapt to the new specification
format.  Add/get all inputs as well.
(db-add-evaluation): Rename REVISION to COMMITS.  Store COMMITS as space
separated commit hashes.
(db-get-builds): Rename REPO_NAME to NAME.
(db-get-stamp): Rename COMMIT to STAMP.  Return #f when there is no STAMP.
(db-add-stamp): Rename COMMIT to STAMP.  Deal with DB-GET-STAMP's new return
value.
(db-get-evaluations): Rename REVISION to COMMITS.  Tokenize COMMITS.
* src/cuirass/utils.scm (%non-blocking): Export it.
* src/schema.sql (Inputs): New table that refers to the Specifications table.
(Specifications): Move input related fields to the Inputs table.  Rename
REPO_NAME to NAME.  Rename ARGUMENTS to PROC_ARGS.  Rename FILE to PROC_FILE.
Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to
the Inputs table.
(Stamps): Rename REPO_NAME to NAME.
(Evaluations): Rename REPO_NAME to NAME.  Rename REVISION to COMMITS.
(Specifications_index): Replace with Inputs_index.
* src/sql/upgrade-1.sql: New file.
* tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to
the new specifications format.  Rename REVISION to COMMITS.
* tests/http.scm (evaluations-query-result, fill-db): Idem.
2018-07-16 21:33:14 +02:00
Clément Lassieur be713f8a30
database: Call a specification 'jobset' instead of 'project'.
This removes the possibility to filter specifications by branch, because
branches were previously called 'jobset'.  But it doesn't matter because later
on, specifications will have as many branches as inputs.  And people should
filter by specification name instead.

* doc/cuirass.texi (Build Information, Latest builds): Remove 'jobset',
replace 'project' with 'jobset'.
* src/cuirass/http.scm (build->hydra-build): Idem.
* tests/database.scm (db-get-builds): Idem.
* tests/http.scm (build-query-result, /api/latestbuilds?nr=1&jobset=guix,
/api/latestbuilds?nr=1&jobset=gnu): Idem.
* src/cuirass/database.scm (db-format-build, db-get-builds): Don't associate
builds with branches (which were called 'jobset' afterwards).
(db-get-builds): Remove the #:project filter.
2018-07-14 21:35:14 +02:00
Clément Lassieur 03c4095f0a
database: Add support for database upgrades.
* src/cuirass/database.scm (%package-sql-dir): New parameter.
(db-load, db-schema-version, db-set-schema-version, latest-db-schema-version,
schema-upgrade-file, db-upgrade): New procedures.
(db-init): Set version corresponding to the existing upgrade-n.sql files.
(db-open): If database exists, upgrade it.
2018-07-14 21:35:13 +02:00
Mathieu Othacehe 85a373f639 http: Reverse evaluations list.
* src/cuirass/database.scm (db-get-evaluations): Reverse returned evaluations
list.
2018-04-23 11:28:37 +02:00
Mathieu Othacehe c40e61bd3e
http: Add /api/evaluations route.
* src/cuirass/database.scm (db-get-evaluations): New exported procedure.
* src/cuirass/http.scm (url-handler): Add /api/evaluations route.
* tests/http.scm ("http"): Add /api/evaluations test route.
2018-04-19 11:17:42 +02:00
Ludovic Courtès fc24ca2eac database: Add 'db-get-pending-derivations'.
* src/cuirass/database.scm (db-get-pending-derivations): New procedure.
* tests/database.scm ("database")["db-get-pending-derivations"]: New test.
2018-04-05 22:15:20 +02:00
Ludovic Courtès 8eefd24672 database: 'db-get-builds' honors 'status+submission-time' ordering again.
Fixes a regression introduced in
1bab5c4e56 whereby the
'status+submission-time' order would no longer be honored.

As a result, /api/queue would return the queue ordered by build IDs,
making it largely useless.

* src/cuirass/database.scm (db-get-builds): Remove 'order' and rename
'order-column-name' to 'order'.  Add case for 'status+submission-time'.
* tests/database.scm ("database")["db-get-builds"]: Move below
"db-update-build-status!" test.  Add case for the
'status+submission-time' order.
2018-03-29 15:30:57 +02:00
Ludovic Courtès 326264c8e9 database: Set a 'busy_timeout' to handle concurrent accesses.
Fixes a bug whereby some fibers would get a SQLITE_BUSY exception while
accessing the database: see
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30644#26>.

Suggested by Danny Milosavljevic <dannym@scratchpost.org>.

* src/cuirass/database.scm (wal-mode): Rename to...
(set-db-options): ... this.  Add call to 'sqlite-exec' for
'busy_timeout'.
2018-03-25 00:09:33 +01:00
Ludovic Courtès 99d88929a6 'with-store' and 'with-database' and written in terms of 'unwind-protect'.
* src/cuirass/base.scm (with-store): Rewrite using 'unwind-protect'.
* src/cuirass/database.scm (with-database): Likewise.
2018-03-19 22:13:54 +01:00
Ludovic Courtès be3650d965 database: Indent 'db-get-builds'.
* src/cuirass/database.scm (db-get-builds): Reindent.
2018-03-01 15:49:30 +01:00
Ludovic Courtès 36d7c93119 database: Adjust style of 'assqx-ref'.
* src/cuirass/database.scm (db-get-builds)[assqx-ref]: Rewrite with
'match'.
2018-03-01 15:44:40 +01:00
Danny Milosavljevic f5a15cab51
database: Fix grouping in db-get-builds.
* src/cuirass/database.scm (db-get-builds): Fix grouping.
2018-02-19 23:01:47 +01:00
Danny Milosavljevic 7be12b9b0f
database: db-get-builds: Remove debugging output.
* src/cuirass/database.scm (db-get-builds): Remove debugging output.
(db-get-build): Remove debugging output.
2018-02-19 22:28:21 +01:00
Danny Milosavljevic 8277233f15
database: db-get-builds: Inline output selection.
* src/cuirass/database.scm (db-get-builds): Inline output selection.
2018-02-19 22:20:23 +01:00
Danny Milosavljevic 1bab5c4e56
database: Simplify 'db-get-builds'.
* src/cuirass/database.scm (db-get-builds): Modify.
(db-get-build): Modify.
2018-02-19 22:13:43 +01:00
Ludovic Courtès 1f31134d33 database: Make 'db-add-derivation' idempotent.
* src/cuirass/database.scm (db-add-derivation): Catch 'sqlite-error and
handle SQLITE_CONSTRAINT_PRIMARYKEY.
(SQLITE_CONSTRAINT_UNIQUE): New variable.
* tests/database.scm ("database")["db-add-derivation"]: Add extra call to
'db-add-derivation'.
2018-02-14 16:41:50 +01:00
Ludovic Courtès 8c7c93922b database: Use argument binding in 'db-get-builds' queries.
That makes it safe from SQL injection.

* src/cuirass/database.scm (db-get-builds): Rewrite to use question
marks in SQL queries and binding through '%sqlite-exec'.
* tests/database.scm ("database")["db-get-builds"]: Exercise 'WHERE'
clauses.
2018-02-08 18:45:03 +01:00
Ludovic Courtès b0c39b31f6 database: Handle binding directly in 'sqlite-exec'.
The new macro automatically takes care of inserting question marks in
the SQL queries, which in turn guarantees that there are always as many
question marks and arguments.

* src/cuirass/database.scm (sqlite-exec): Rename to...
(%sqlite-exec): ... this.
(sqlite-exec/bind, sqlite-exec): New macros.
(assq-refs): Remove.
(db-add-specification): Use the new 'sqlite-exec' form.
(db-get-specifications): Correctly deal with REV or TAG being #f.
(db-add-derivation, db-get-derivation, db-add-evaluation)
(db-add-build, db-update-build-status!, db-get-outputs)
(db-get-build, db-get-stamp, db-add-stamp): Adjust to the new
'sqlite-exec' form.
2018-02-08 17:35:32 +01:00
Ludovic Courtès 2887b1dfa9 database: 'db-update-build-status!' keeps stoptime unchanged when nothing new.
* src/cuirass/database.scm (db-update-build-status!): And "AND status != ?"
in SQL queries.
* tests/database.scm ("database")["db-update-build-status!"]: Add call
to 'db-update-build-status!'.
2018-02-08 14:19:19 +01:00
Ludovic Courtès 53fcecd25f database: Cache prepared statements.
Suggested by Danny Milosavljevic.

* src/cuirass/database.scm (sqlite-exec): Pass #:cache? to
'sqlite-prepare'.
2018-02-08 14:19:19 +01:00
Danny Milosavljevic eb01f46987 database: Use 'sqlite-bind' to avoid SQL injection.
* src/cuirass/database.scm (%sqlite-exec): Remove.
(sqlite-exec): Turn back into a procedure.  Use 'sqlite-bind'.  Add
'normalize' procedure and use it.
(db-add-specification, db-add-derivation, db-get-derivation)
(db-add-evaluation, db-add-build, db-update-build-status!)
(db-get-build, db-get-stamp, db-add-stamp): Use question marks in SQL
queries.
* src/cuirass/base.scm (build-packages)[register]: Make #:log
non-false.
* tests/database.scm (make-dummy-job): Add #:job-name, #:system,
 #:nix-name, and #:eval-id.  This is necessary because 'sqlite-bind'
would now translate #f to a real NULL (before it would translate to the
string "#f"...), and would thus report violations of the non-NULL
constraint.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
2018-02-08 14:18:27 +01:00
Ludovic Courtès fcd1bc13bc http: /api/queue returns builds sorted by status.
* src/cuirass/database.scm (db-get-builds)[format-order-clause]: Add
'status+submission-time'.
* src/cuirass/http.scm (url-handler) <"queue">: Use it.
2018-01-29 18:06:47 +01:00
Ludovic Courtès f3335e880c database: Open the database in "write-ahead log" mode.
Suggested by Ricardo Wurmus.

* src/cuirass/database.scm (wal-mode): New procedure.
(db-open): Use it.
2018-01-26 18:21:56 +01:00
Ludovic Courtès a97665672a http: Sort /api/latestbuilds and /api/queue by time.
* src/cuirass/database.scm (db-get-builds)[format-order-clause]: Add
'order' clauses.
* src/cuirass/http.scm (url-handler): Default to (order finish-time) for
/latestbuilds and (order submission-time) for /queue.
2018-01-25 10:39:18 +01:00
Ludovic Courtès c4963deca1 database: Turn 'sqlite-exec' into a macro.
* src/cuirass/database.scm (%sqlite-exec): New procedure.
(sqlite-exec): New macro.
2018-01-25 10:38:53 +01:00
Ludovic Courtès 8675d6309b database: 'db-update-build-status!' takes a #:log-file parameter.
* src/cuirass/database.scm (sqlite-exec): Use (ice-9 format).
(db-update-build-status!): Add #:log-file parameter and honor it.
* tests/database.scm ("database")["db-update-build-status!"]: Test it.
2018-01-24 23:41:24 +01:00
Ludovic Courtès dd30a1a25c base: Restart pending builds upfront.
* src/cuirass/database.scm (db-get-builds)[format-where-clause]:
Honor (status pending) filter.
* src/cuirass/base.scm (restart-builds): New procedure.
* bin/cuirass.in (main): Fetch pending builds.  Start fiber that invokes
'restart-builds' on them.
2018-01-23 23:40:07 +01:00
Ludovic Courtès 06b8af00fb http: /api/latestbuilds only returns finished builds.
* src/cuirass/database.scm (db-get-builds)[format-where-clause]:
Honor (status done) filter.
* src/cuirass/http.scm (url-handler): Always add (status done)
to the filters passed to 'handle-builds-request'.
2018-01-23 23:39:13 +01:00
Ludovic Courtès 6fb7118839 database: Use US spelling for "canceled".
* src/cuirass/database.scm (build-status): Use US spelling for
"canceled".
2018-01-23 23:39:13 +01:00
Ludovic Courtès fe0a98315d database: Add 'db-update-build-status!'.
* src/cuirass/database.scm (build-status): Add 'scheduled' and
'started'.
(db-add-build): Make sure #:timestamp, #:starttime, #:stoptime, and
 #:status are integers.
(db-update-build-status!): New procedure.
* tests/database.scm ("database")["db-update-build-status!"]: New test.
2018-01-23 18:17:31 +01:00
Ludovic Courtès b0f1a438ff Add a 'build-status' enumeration.
* src/cuirass/utils.scm (define-enumeration): New macro.
* src/cuirass/database.scm (build-status): New macro.
* src/cuirass/base.scm (build-packages)[hydra-build-status]: Remove.
Use the 'build-status' macro instead.
2018-01-23 17:37:08 +01:00
Ludovic Courtès ee11ba1d93 Introduce concurrency with Fibers.
* README: Mark Fibers as required.
* configure.ac: Check for Guile 2.2 only.  Check for (fibers).
* bin/cuirass.in (main): Use (fibers).  Run 'process-specs' and web
server in separate fibers.
* src/cuirass/base.scm (with-store): New macro.
(non-blocking-port): New procedure.
(evaluate): Use 'non-blocking-port'.  Use 'read-string' followed by 'read'.
(process-specs): Move 'db-add-stamp' right after 'string=?' comparison.
Run evaluation and subsequent builds in a separate fiber.
* src/cuirass/http.scm (run-cuirass-server): Pass 'fibers as the second
argument to 'run-server'.  Use 'log-message' instead of 'format'.
* src/cuirass/database.scm (with-database): Remove 'dynamic-wind'.
2018-01-22 23:39:51 +01:00
Ludovic Courtès f9481e2222 database: Change default file name to /var/run/cuirass/cuirass.db.
* src/cuirass/database.scm (%package-database): Change default file name
from /var/cuirass.db (or similar) to /var/run/cuirass/cuirass.db.
2018-01-22 14:51:14 +01:00
Ludovic Courtès 60b6c6fcc5 database: Extend and test 'db-get-builds'.
* src/cuirass/database.scm (db-get-builds): Make 'order' a separate
filter.  Add 'format-limit-clause'.  Reverse OUTPUTS.
* tests/database.scm (make-dummy-eval, make-dummy-derivation)
(make-dummy-build): New procedures.
(with-temporary-database): New macro.
("database"): Use 'make-dummy-build'.
("db-get-builds"): New test.
2018-01-07 23:59:30 +01:00
Mathieu Othacehe 8a7483a3bb
cuirass: add Hydra compatible HTTP API.
* doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API.
* src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it
  object->json-string.
(object->json-scm): Move it utils.scm.
(handle-*-request): New helpers procedures.
(request-parameters): New procedure to parse a request query.
(url-handler): Add new API's.
* src/cuirass/utils.scm (object->json-scm, object->json-string): Exported
procedures moved from http.scm.
* tests/http.scm: Add various tests on new HTTP API.
2017-09-08 21:15:32 +02:00
Mathieu Othacehe e550cb6a9a
cuirass: Store new information in database to prepare new HTTP API
integration.

* bin/evaluate.in (fill-job): New procedure.
(main): Use it to fill informations (nix-name, system) that will later be
added to database.
* doc/cuirass.texi (Database)[Derivation]: Add system and nix_name fields.
(Database)[Builds]: Add id, status, timestamp, starttime and stoptime
fields. Remove output field.
(Database)[Outputs]: New table describing the build outputs.
* src/cuirass/base.scm (build-packages): Add new fields to build object before
adding it to database.
* src/cuirass/database.scm (db-get-build, db-get-builds): New procedures to get
a build by id from database and a list of builds using filter parameters
respectively.
* src/schema.sql (Outputs) : New table.
(Derivations): Add system and nix_name columns.
(Builds): Remove output column and add id, status, timestamp, starttime and
stoptime columns.
2017-09-08 21:00:57 +02:00
Ludovic Courtès 72f2b6b77c database: 'db-add-build' is now idempotent.
Fixes <https://bugs.gnu.org/28094>.
Reported by Ricardo Wurmus <rekado@elephly.net>.

* src/cuirass/database.scm (SQLITE_CONSTRAINT)
(SQLITE_CONSTRAINT_PRIMARYKEY): New variables.
(db-add-build): Catch 'sqlite-error, and swallow
SQLITE_CONSTRAINT_PRIMARYKEY errors.
* tests/database.scm ("db-add-build"): New test.
2017-08-26 10:45:43 +02:00
Mathieu Othacehe 707b053024
database: Fix typo.
* src/cuirass/database.scm (db-get-stamp, db-add-stamp): Replace #:id
  by #:name. There is no #:id in specifications. Plus, the primary key
  of Specifications is #:name.
2017-07-12 16:28:41 +02:00
Mathieu Othacehe d7dc7cb92c
cuirass: Stop λ/lambda mixing.
* bin/cuirass.in: Replace λ by lambda.
* bin/evaluate.in: Ditto.
* build-aux/guix.scm: Ditto.
* examples/gnu-system.scm: Ditto.
* examples/guix-track-git.scm: Ditto.
* src/cuirass.scm: Ditto.
* src/cuirass/base.scm: Ditto.
* src/cuirass/database.scm: Ditto.
* src/cuirass/ui.scm: Ditto.
* src/cuirass/utils.scm: Remove λ* macro.
2017-07-06 10:03:16 +02:00
Mathieu Lirzin d0a5801e39
database: db-add-derivation: Don't try to add a derivation twice.
This fixes the issue of having multiple identical derivations associated with
an evaluation as reported by Ludovic Courtès <ludo@gnu.org> at
<https://lists.gnu.org/archive/html/guix-devel/2017-01/msg00109.html>.

* src/cuirass/database.scm (db-add-derivation): Ignore if JOB is already
present in DB.
2017-01-09 10:55:58 +01:00
Mathieu Lirzin bfd395c09f
db: Forbid inserting the same specification twice.
* src/schema.sql (Specifications): Make 'repo_name' the primary key.
(Stamps, Evaluations): Adapt.
* src/cuirass/database.scm (db-get-specifications): Likewise.
(db-add-specification): Ignore if SPEC has already been added.
* tests/database.scm (example-spec): Adapt.
2016-11-17 01:19:05 +01:00
Jan Nieuwenhuizen 5ef0701f54
base: Support tracking of a Guix package's git.
* src/schema.sql (Specifications): Add no_compile_p column.
* src/cuirass/database.scm (db-add-specification)
(db-get-specifications): Handle #:no-compile? property.
* tests/database.scm (example-spec): Adapt.
* src/cuirass/base.scm (process-specs): Skip compilation if #:no-compile?.

Signed-off-by: Mathieu Lirzin <mthl@gnu.org>
2016-09-23 16:50:36 +02:00
Mathieu Lirzin ac4512897c
schema: Evaluations: Add 'revision' column.
* src/schema.sql (Evaluations): Add 'revision' column.
* src/cuirass/database.scm (db-add-evaluation): Adapt.
All callers changed.
2016-07-30 23:07:19 +02:00
Mathieu Lirzin 24bfb61e8e
schema: Builds: Add 'evaluation' column.
* src/schema.sql (Builds): Add 'evaluation' column.
* src/cuirass/database.scm (db-add-build): Adapt.
* src/cuirass/base.scm (build-packages): Likewise.
2016-07-27 14:17:10 +02:00
Mathieu Lirzin d493a58823
schema: Separate "Derivations" from "Evaluations".
* src/schema.sql (Derivations): New table.
(Evaluations): Remove 'derivation' and 'job_name' columns.  Add 'id'
column.
* src/cuirass/database.scm (db-add-evaluation): Adapt.
(db-get-derivation, db-add-derivation): New procedures.
(evaluation-exists?, db-get-evaluation): Delete.
* bin/evaluate.in (main): Adapt.
* tests/database.scm ("sqlite-exec"): Likewise.
("db-add-derivation", "db-get-derivation"): New tests.
("db-add-evaluation", "db-get-evaluation"): Delete.
2016-07-27 14:17:09 +02:00
Mathieu Lirzin 719c126863
database: Remove 'db-delete-evaluation' procedure.
* src/cuirass/database.scm (db-delete-evaluation): Delete.
2016-07-27 14:17:09 +02:00
Mathieu Lirzin b8d8c99bd3
database: with-database: Keep the data.
* src/cuirass/database.scm (with-database): Use 'db-open'.
(db-open): Fallback to 'db-init' if DB do not exist.
2016-07-26 11:40:51 +02:00
Mathieu Lirzin f40ac8a535
database: db-open: Add DB optional argument.
* src/cuirass/database.scm (db-open): Add DB optional argument.
2016-07-26 11:21:27 +02:00
Mathieu Lirzin c83d5d4da8
Use stamps to poll repositories. 2016-07-26 00:36:12 +02:00
Mathieu Lirzin e51a755f10
database: Add 'db-add-build' procedure. 2016-07-25 20:32:01 +02:00
Mathieu Lirzin 7292bd5019
database: read-quoted-string: add default value to optional argument. 2016-07-25 20:32:01 +02:00
Mathieu Lirzin a063a2277e
cuirass: Use database to store specifications. 2016-07-25 02:37:00 +02:00
Mathieu Lirzin 5db6894ae2
database: db-init: Add arguments for debugging. 2016-07-25 02:12:42 +02:00
Mathieu Lirzin c7c9e91876
schema: Separate 'Evaluations' from 'Builds'.
Adapt src/cuirass/database.scm and its tests.
2016-07-25 02:12:41 +02:00
Mathieu Lirzin cf7e290dc2
maint: Update file headers. 2016-07-25 02:12:41 +02:00
Mathieu Lirzin a4c8bf0fd4
database: db-get-evaluation: Use 'sqlite-exec'. 2016-07-25 02:12:40 +02:00
Mathieu Lirzin 8ad3ce782f
database: Add 'last-insert-rowid' procedure. 2016-07-25 02:12:40 +02:00
Mathieu Lirzin 447d9f7ba8
database: Add 'assq-refs' procedure. 2016-07-25 02:12:40 +02:00
Mathieu Lirzin b1dd669196
database: sqlite-exec: Return every rows.
Export it.  Add a test.
2016-07-25 02:12:40 +02:00
Mathieu Lirzin 18e42be302
database: Use 'simple-format' in 'sqlite-exec'. 2016-07-25 02:12:40 +02:00
Mathieu Lirzin f9938a3caa
database: Don't catch exceptions from (sqlite3) module. 2016-07-25 02:12:39 +02:00
Mathieu Lirzin 46c9d432ea
database: Use an external SQL schema. 2016-07-25 02:12:39 +02:00
Mathieu Lirzin 0808ab3ecd Remove unnecessary (cuirass job) module. 2016-07-13 14:16:36 +02:00
Mathieu Lirzin 53c12be409 Evaluate derivations in a separate process.
This fixes a bug where different Guix branches gave the same
derivations.
2016-07-13 14:07:24 +02:00
Mathieu Lirzin 13db5aa618 λ all the things! 2016-07-02 15:49:34 +02:00
Mathieu Lirzin 9a9b3fbd45 database: Remove commented code. 2016-06-29 16:32:55 +02:00
Mathieu Lirzin 990c902fcc Store build logs in the database. 2016-06-29 16:16:48 +02:00
Mathieu Lirzin 5ff38984e8 Make %package-database a parameter object.
Move it to (cuirass database).
2016-06-26 22:54:49 +02:00
Mathieu Lirzin b24541e604 Add (cuirass database) module. 2016-06-26 16:59:26 +02:00