2016-05-11 19:59:47 +02:00
|
|
|
|
#!/bin/sh
|
|
|
|
|
# -*- scheme -*-
|
2016-11-12 17:36:18 +01:00
|
|
|
|
# @configure_input@
|
2016-12-24 01:08:04 +01:00
|
|
|
|
#GUILE_LOAD_PATH="@PACKAGE_LOAD_PATH@${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
|
|
|
|
|
#GUILE_LOAD_COMPILED_PATH="@PACKAGE_LOAD_COMPILED_PATH@${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_LOAD_COMPILED_PATH"
|
2016-05-11 19:59:47 +02:00
|
|
|
|
exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@"
|
|
|
|
|
!#
|
2016-11-12 17:36:18 +01:00
|
|
|
|
;;;; cuirass -- continuous integration tool
|
2016-05-11 19:59:47 +02:00
|
|
|
|
;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
|
2017-01-25 09:54:54 +01:00
|
|
|
|
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
|
2018-02-05 14:59:29 +01:00
|
|
|
|
;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
|
Add support for multiple inputs.
* Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql.
* bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that
was used afterwards as %GUIX-PACKAGE-PATH.
* bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path,
spec-package-path, format-checkouts): New procedures.
(%not-colon): Remove variable.
(main): Take the load path, package path and PROC from the checkouts that
result from the inputs. Format the checkouts before sending them to the
procedure. Remove the LOAD-PATH argument.
* doc/cuirass.texi (Overview, Database schema): Document the changes.
* examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm,
hello-subset.scm, random.scm}: Adapt to the new specification format.
* examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT.
(package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle
the new format of its return value.
* examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT.
Rename the checkout from 'random (which is a specification) to 'cuirass (which
is a checkout resulting from an input).
* src/cuirass/base.scm (fetch-repository): Rename to fetch-input. Rename SPEC
to INPUT. Return a checkout object instead of returning two values.
(evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of
SOURCE. Remove TOKENIZE and LOAD-PATH. Pass the CHECKOUTS instead of the
SOURCE to "evaluate". Remove %GUIX-PACKAGE-PATH. Build the EVAL object
instead of getting it from "evaluate".
(compile?, fetch-inputs, compile-checkouts): New procedures.
(process-specs): Fetch all inputs instead of only fetching one repository.
The result of that fetching operation is a list of CHECKOUTS whose COMMITS are
used as a STAMP.
(%guix-package-path, set-guix-package-path): Remove them.
* src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures.
(db-add-specification, db-get-specifications): Adapt to the new specification
format. Add/get all inputs as well.
(db-add-evaluation): Rename REVISION to COMMITS. Store COMMITS as space
separated commit hashes.
(db-get-builds): Rename REPO_NAME to NAME.
(db-get-stamp): Rename COMMIT to STAMP. Return #f when there is no STAMP.
(db-add-stamp): Rename COMMIT to STAMP. Deal with DB-GET-STAMP's new return
value.
(db-get-evaluations): Rename REVISION to COMMITS. Tokenize COMMITS.
* src/cuirass/utils.scm (%non-blocking): Export it.
* src/schema.sql (Inputs): New table that refers to the Specifications table.
(Specifications): Move input related fields to the Inputs table. Rename
REPO_NAME to NAME. Rename ARGUMENTS to PROC_ARGS. Rename FILE to PROC_FILE.
Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to
the Inputs table.
(Stamps): Rename REPO_NAME to NAME.
(Evaluations): Rename REPO_NAME to NAME. Rename REVISION to COMMITS.
(Specifications_index): Replace with Inputs_index.
* src/sql/upgrade-1.sql: New file.
* tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to
the new specifications format. Rename REVISION to COMMITS.
* tests/http.scm (evaluations-query-result, fill-db): Idem.
2018-06-26 11:18:23 +02:00
|
|
|
|
;;; Copyright © 2018 Clément Lassieur <clement@lassieur.org>
|
2016-05-11 19:59:47 +02:00
|
|
|
|
;;;
|
|
|
|
|
;;; This file is part of Cuirass.
|
|
|
|
|
;;;
|
2016-07-23 12:10:04 +02:00
|
|
|
|
;;; Cuirass is free software: you can redistribute it and/or modify
|
|
|
|
|
;;; it under the terms of the GNU General Public License as published by
|
|
|
|
|
;;; the Free Software Foundation, either version 3 of the License, or
|
|
|
|
|
;;; (at your option) any later version.
|
2016-05-11 19:59:47 +02:00
|
|
|
|
;;;
|
2016-07-23 12:10:04 +02:00
|
|
|
|
;;; Cuirass is distributed in the hope that it will be useful,
|
|
|
|
|
;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
|
2016-05-11 19:59:47 +02:00
|
|
|
|
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
;;; GNU General Public License for more details.
|
|
|
|
|
;;;
|
|
|
|
|
;;; You should have received a copy of the GNU General Public License
|
|
|
|
|
;;; along with Cuirass. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
|
|
2016-08-02 17:04:00 +02:00
|
|
|
|
(use-modules (cuirass)
|
2016-06-10 22:27:41 +02:00
|
|
|
|
(cuirass ui)
|
2018-01-22 23:07:10 +01:00
|
|
|
|
(cuirass logging)
|
2018-01-27 16:04:31 +01:00
|
|
|
|
(cuirass utils)
|
2017-07-01 12:23:23 +02:00
|
|
|
|
(guix ui)
|
2018-01-22 23:07:10 +01:00
|
|
|
|
(fibers)
|
2018-01-26 14:27:46 +01:00
|
|
|
|
(fibers channels)
|
2018-02-05 14:59:29 +01:00
|
|
|
|
(ice-9 threads) ;for 'current-processor-count'
|
2016-07-26 17:16:44 +02:00
|
|
|
|
(ice-9 getopt-long))
|
2016-05-11 19:59:47 +02:00
|
|
|
|
|
2016-07-13 18:34:08 +02:00
|
|
|
|
(define (show-help)
|
2016-07-26 11:58:28 +02:00
|
|
|
|
(format #t "Usage: ~a [OPTIONS]~%" (%program-name))
|
|
|
|
|
(display "Run build jobs from internal database.
|
2016-06-10 22:27:41 +02:00
|
|
|
|
|
2016-07-02 23:08:52 +02:00
|
|
|
|
--one-shot Evaluate and build jobs only once
|
2016-07-02 15:34:40 +02:00
|
|
|
|
--cache-directory=DIR Use DIR for storing repository data
|
2017-07-11 11:38:18 +02:00
|
|
|
|
--fallback Fall back to building when the substituter fails.
|
2016-07-26 11:58:28 +02:00
|
|
|
|
-S --specifications=SPECFILE
|
|
|
|
|
Add specifications from SPECFILE to database.
|
2016-06-26 17:02:23 +02:00
|
|
|
|
-D --database=DB Use DB to store build results.
|
2016-12-24 02:15:17 +01:00
|
|
|
|
-p --port=NUM Port of the HTTP server.
|
2017-09-18 21:57:05 +02:00
|
|
|
|
--listen=HOST Listen on the network interface for HOST
|
2016-07-26 16:17:54 +02:00
|
|
|
|
-I, --interval=N Wait N seconds between each poll
|
2016-09-15 22:50:42 +02:00
|
|
|
|
--use-substitutes Allow usage of pre-built substitutes
|
2018-02-05 14:59:29 +01:00
|
|
|
|
--threads=N Use up to N kernel threads
|
2016-06-10 22:27:41 +02:00
|
|
|
|
-V, --version Display version
|
|
|
|
|
-h, --help Display this help message")
|
|
|
|
|
(newline)
|
|
|
|
|
(show-package-information))
|
|
|
|
|
|
|
|
|
|
(define %options
|
2016-07-26 11:58:28 +02:00
|
|
|
|
'((one-shot (value #f))
|
|
|
|
|
(cache-directory (value #t))
|
|
|
|
|
(specifications (single-char #\S) (value #t))
|
|
|
|
|
(database (single-char #\D) (value #t))
|
2016-12-24 02:15:17 +01:00
|
|
|
|
(port (single-char #\p) (value #t))
|
2017-09-18 21:57:05 +02:00
|
|
|
|
(listen (value #t))
|
2016-07-26 11:58:28 +02:00
|
|
|
|
(interval (single-char #\I) (value #t))
|
2016-09-15 22:50:42 +02:00
|
|
|
|
(use-substitutes (value #f))
|
2018-02-05 14:59:29 +01:00
|
|
|
|
(threads (value #t))
|
2017-07-11 11:38:18 +02:00
|
|
|
|
(fallback (value #f))
|
2016-07-26 11:58:28 +02:00
|
|
|
|
(version (single-char #\V) (value #f))
|
|
|
|
|
(help (single-char #\h) (value #f))))
|
2016-06-10 22:27:41 +02:00
|
|
|
|
|
2016-05-11 19:59:47 +02:00
|
|
|
|
|
|
|
|
|
;;;
|
|
|
|
|
;;; Entry point.
|
|
|
|
|
;;;
|
|
|
|
|
|
|
|
|
|
(define* (main #:optional (args (command-line)))
|
2018-03-24 23:52:44 +01:00
|
|
|
|
|
|
|
|
|
;; Always have stdout/stderr line-buffered.
|
2018-04-09 11:08:06 +02:00
|
|
|
|
(setvbuf (current-output-port) 'line)
|
|
|
|
|
(setvbuf (current-error-port) 'line)
|
2018-03-24 23:52:44 +01:00
|
|
|
|
|
2016-08-02 18:13:48 +02:00
|
|
|
|
(let ((opts (getopt-long args %options)))
|
2016-07-01 14:39:21 +02:00
|
|
|
|
(parameterize
|
|
|
|
|
((%program-name (car args))
|
2016-07-02 15:34:40 +02:00
|
|
|
|
(%package-database (option-ref opts 'database (%package-database)))
|
|
|
|
|
(%package-cachedir
|
2016-09-15 22:50:42 +02:00
|
|
|
|
(option-ref opts 'cache-directory (%package-cachedir)))
|
2017-07-11 11:38:18 +02:00
|
|
|
|
(%use-substitutes? (option-ref opts 'use-substitutes #f))
|
|
|
|
|
(%fallback? (option-ref opts 'fallback #f)))
|
2016-07-01 14:39:21 +02:00
|
|
|
|
(cond
|
|
|
|
|
((option-ref opts 'help #f)
|
|
|
|
|
(show-help)
|
|
|
|
|
(exit 0))
|
|
|
|
|
((option-ref opts 'version #f)
|
|
|
|
|
(show-version)
|
|
|
|
|
(exit 0))
|
|
|
|
|
(else
|
2016-07-02 23:08:52 +02:00
|
|
|
|
(let ((one-shot? (option-ref opts 'one-shot #f))
|
2016-12-24 02:15:17 +01:00
|
|
|
|
(port (string->number (option-ref opts 'port "8080")))
|
2017-09-18 21:57:05 +02:00
|
|
|
|
(host (option-ref opts 'listen "localhost"))
|
2018-01-22 23:07:10 +01:00
|
|
|
|
(interval (string->number (option-ref opts 'interval "300")))
|
2018-02-05 14:59:29 +01:00
|
|
|
|
(specfile (option-ref opts 'specifications #f))
|
|
|
|
|
|
|
|
|
|
;; Since our work is mostly I/O-bound, default to a maximum of 4
|
|
|
|
|
;; kernel threads. Going beyond that can increase overhead (GC
|
|
|
|
|
;; may not scale well, work-stealing may become detrimental,
|
|
|
|
|
;; etc.) for little in return.
|
|
|
|
|
(threads (or (and=> (option-ref opts 'threads #f)
|
|
|
|
|
string->number)
|
|
|
|
|
(min (current-processor-count) 4))))
|
2018-01-22 13:04:20 +01:00
|
|
|
|
(prepare-git)
|
2018-02-05 14:59:29 +01:00
|
|
|
|
|
|
|
|
|
(log-message "running Fibers on ~a kernel threads" threads)
|
2018-01-22 23:07:10 +01:00
|
|
|
|
(run-fibers
|
|
|
|
|
(lambda ()
|
|
|
|
|
(with-database db
|
|
|
|
|
(and specfile
|
|
|
|
|
(let ((new-specs (save-module-excursion
|
|
|
|
|
(lambda ()
|
|
|
|
|
(set-current-module (make-user-module '()))
|
|
|
|
|
(primitive-load specfile)))))
|
|
|
|
|
(for-each (lambda (spec) (db-add-specification db spec))
|
|
|
|
|
new-specs)))
|
|
|
|
|
(if one-shot?
|
|
|
|
|
(process-specs db (db-get-specifications db))
|
2018-04-05 22:17:45 +02:00
|
|
|
|
(let ((exit-channel (make-channel)))
|
|
|
|
|
|
|
|
|
|
(clear-build-queue db)
|
2018-01-26 14:09:47 +01:00
|
|
|
|
|
2018-01-23 23:15:10 +01:00
|
|
|
|
;; First off, restart builds that had not completed or
|
|
|
|
|
;; were not even started on a previous run.
|
|
|
|
|
(spawn-fiber
|
2018-01-27 16:04:31 +01:00
|
|
|
|
(essential-task
|
|
|
|
|
'restart-builds exit-channel
|
|
|
|
|
(lambda ()
|
|
|
|
|
(with-database db
|
2018-04-05 22:17:45 +02:00
|
|
|
|
(restart-builds db)))))
|
2018-01-23 23:15:10 +01:00
|
|
|
|
|
2018-01-26 14:27:46 +01:00
|
|
|
|
(spawn-fiber
|
2018-01-27 16:04:31 +01:00
|
|
|
|
(essential-task
|
|
|
|
|
'build exit-channel
|
|
|
|
|
(lambda ()
|
|
|
|
|
(with-database db
|
|
|
|
|
(while #t
|
|
|
|
|
(process-specs db (db-get-specifications db))
|
|
|
|
|
(log-message "next evaluation in ~a seconds" interval)
|
|
|
|
|
(sleep interval))))))
|
2018-01-26 14:27:46 +01:00
|
|
|
|
|
2018-01-22 23:07:10 +01:00
|
|
|
|
(spawn-fiber
|
2018-01-27 16:04:31 +01:00
|
|
|
|
(essential-task
|
|
|
|
|
'web-server exit-channel
|
|
|
|
|
(lambda ()
|
|
|
|
|
(with-database db
|
2018-01-27 19:31:06 +01:00
|
|
|
|
(run-cuirass-server db #:host host #:port port))))
|
|
|
|
|
#:parallel? #t)
|
2018-01-23 23:15:10 +01:00
|
|
|
|
|
2018-01-29 12:17:20 +01:00
|
|
|
|
(spawn-fiber
|
|
|
|
|
(essential-task
|
|
|
|
|
'monitor exit-channel
|
|
|
|
|
(lambda ()
|
|
|
|
|
(while #t
|
|
|
|
|
(log-monitoring-stats)
|
|
|
|
|
(sleep 600)))))
|
|
|
|
|
|
2018-01-26 14:27:46 +01:00
|
|
|
|
(primitive-exit (get-message exit-channel))))))
|
2018-01-22 23:07:10 +01:00
|
|
|
|
|
2018-01-26 14:37:22 +01:00
|
|
|
|
;; Most of our code is I/O so preemption doesn't matter much (it
|
|
|
|
|
;; could help while we're doing SQL requests, for instance, but it
|
|
|
|
|
;; doesn't actually help since these are non-resumable
|
|
|
|
|
;; continuations.) Thus, reduce the tick rate.
|
|
|
|
|
#:hz 10
|
|
|
|
|
|
2018-02-05 14:59:29 +01:00
|
|
|
|
#:parallelism threads
|
2018-01-22 23:07:10 +01:00
|
|
|
|
#:drain? #t)))))))
|