There's no need to have a large store on bayfront, so change the gc
configuration to reduce it in size. Also reduce the frequency so that
the gc blocks other operations less often.
* hydra/bayfront.scm <services>: Change gc configuration.
To try and reduce the chance of the MDC blocking requests from
bayfront.
* hydra/bayfront.scm (%bordeaux.guix.gnu.org-nginx-servers): Enable
keeping connections to hydra-guix-129 alive.
<services>: Keep up to 2 connections from each NGinx worker process to
hydra-guix-129 alive.
The signing key on 'dover' was apparently changed on April 18th, 2023.
* hydra/keys/guix/berlin/dover.guix.info:9023.pub: Rename to...
* hydra/keys/guix/berlin/dover.guix.gnu.org.pub: ... this. Update key.
Rather than reverse proxying requests to bishan, which then would
reverse proxy to hydra-guix-129.
This will mean more requests get sent to hydra-guix-129 so we'll have
to monitor how well this works.
* hydra/bayfront.scm (%bordeaux.guix.gnu.org-nginx-servers): Reverse
proxy nar requests to hydra-guix-129.
This is a Hetzner VM I've had for a while. It's currently a key part
in the patch and branch testing since it runs data.qa.guix.gnu.org,
git.guix-patches.cbaines.net and patches.guix-patches.cbaines.net.
Publishing the configuration is long overdue.
* hydra/beid.scm: New file.
I'm moving this functionality to the build farm front-end, it's always
been a bit odd having it in the build coordinator codebase and now
that there's a proper frontend, I think that's the natural place for
it to sit.
* hydra/bayfront.scm <services>: Remove the
guix-build-coordinator-queue-builds-service-type service.
bishan has run out of space but now uses hydra-guix-129 as a source
for nars it doesn't have. This means that bayfront can now remove nars
as long as both hatysa and hydra-guix-129 have them.
Since I'd like to get rid of bishan, I was thinking of having bayfront
use hydra-guix-129 directly, but this isn't currently possible as the
MDC is blocking requests from bayfront.
* hydra/bayfront.scm (%bordeaux.guix.gnu.org-nginx-servers): Add Via
header to nar requests.
<services>: Change the nar-herder to remove nars if they're stored on
both hatysa and hydra-guix-129. Tweak the
guix-build-coordinator-agent configuration to allow parallel uploads.
* hydra/bishan.scm (%nginx-server-blocks): Add via header on requests,
and use hydra-guix-129 when nars aren't found locally.
<services>: Add IPv4 address, set a storage limit for the nar-herder,
increase the NGinx worker processes and change the nar-storage upstream.
Step in for bishan (which has run out of space) to store and serve all
the bordeaux nars.
* hydra/deploy-node-129.scm (%nginx-server-blocks): New variable.
<services>: Switch to mirroring bordeaux.guix.gnu.org directly, and
add certbot and nginx.
* hydra/modules/sysadmin/services.scm (guix-daemon-config)
(frontend-services): Add #:substitute-urls and honor it.
* hydra/berlin.scm: Pass #:substitute-urls to 'frontend-services'.
Dropping the cache size can (and did) lead to removing cached nars,
which has caused problems for Guix clients that expect the zstd
compression to be available.
Increasing the cache size will at least allow these zstd compressed
nars to be cached again.
This partially reverts commit
ce8d3000fd.
* hydra/bayfront.scm <services>[nar-herder]: Increase the zstd
directory-max-size.
As part of the plan to store and serve bordeaux nars from the attached
SSDs, this commit adds the nar-herder to start downloading the nars.
This is mirroring from bishan as that has most of the nars, and this
avoids downloading everything via bayfront.
The next step will be to setup DNS and deploy NGinx.
* hydra/deploy-node-129.scm (node-129-os) [packages]: Add nss-certs.
[file-systems]: Mount the @bordeaux-nars subvolume.
[services]: Add the nar-herder.
This is necessary to enable hydra-guix-129 to access bishan to sync
nars, since hydra-guix-129 is IPv4 only.
* hydra/modules/sysadmin/dns.scm (guix.gnu.org.zone): Add IPv4 address
for bishan.
(guix.gnu.org-zone): Bump 'serial'.
By dropping the size of some caches. This is hopefully a temporary
measure while the nar storage is sorted out.
* hydra/bayfront.scm (%bayfront-nginx-service-extra-config): Drop the
cache size.
<services>[nar-herder]: Drop the zstd directory-max-size.
Affecting both the build coordinator and nar herder:
https://issues.guix.gnu.org/63368
* hydra/bayfront.scm (%guix-build-coordinator-configuration): Add
GC_RETRY_SIGNALS=0 to extra-environment-variables.
<services>[nar-herder]: Likewise.
As announced on guix-sysadmin (06/05/23), in addition to the old
one. This will enable the FSF sysadmins to migrate ns1 to the new IP.
* hydra/bayfront.scm (gnu-ns1-ip4): Rename to gnu-ns1-ip4/old.
(gnu-ns1-ip4/new): New variable.
(operating-system)[services]: Change knot configuration to use both
new and old nameserver remotes.
* hydra/deploy-node-129.scm (node-129-os) [services] <openssh>:
Authorize the berlin.guixsd.org.pub SSH key for the 'root' user.
Allow berlin (141.80.181.40) to connect as root.
* hydra/deploy-node-129.scm (%btrfs-ssd-raid10-uuid): New variable.
(%btrfs-pool-san): Relocate mount point to /mnt/btrfs-pool-san.
(%btrfs-pool-ssd): New variable.
(btrfs-subvolume-mount): Support a #:device-uuid argument.
(node-129-os) [file-systems]: Add %btrfs-pool-ssd. Mount the
@publish-mirror subvolume to /srv/publish/substitutes.
* hydra/berlin.scm (btrfs-subvolume-mount): Move the DEVICE-UUID to a
#:device-uuid argument.
(operating-system) [file-systems]: Add a new mount point for the
@publish subvolume.
As part of using hydra-guix-129 to store and serve nars for the
bordeaux build farm.
* hydra/modules/sysadmin/build-machines.scm (berlin-new-build-machine-os)[sysadmins]:
Add myself.
The new ssl-ca certificate authority produced doesn't have a CRL file
for now. Remove it from the config to avoid crashing nginx.
* hydra/nginx/berlin.scm (%berlin-servers): Remove ssl_crl directive.
(%zabbix-nginx-server): Likewise.
Previously, the default route would be internal to the network,
causing replies to external requests to be filtered by the firewall.
* hydra/deploy-node-129.scm (node-129-os)
[static-networking-service-type]: Set default route to 141.80.181.1.
As there's now a riscv64-linux machine connected to the coordinator.
* hydra/bayfront.scm <services>: Add riscv64-linux to the systems list
in the queue builds service type configuration.
To help standardise access.
* hydra/modules/sysadmin/dns.scm (monokuma-ip6, dover-ip6, hatysa-ip6,
hamal-ip6): New variables.
(guix.gnu.org.zone): Use them and bump serial.
* hydra/machines.rec: Update accordingly.
The lakeside machine was replaced by bishan, and the fosshost machines
are no longer around.
* hydra/modules/sysadmin/dns.scm (guix.gnu.org.zone): Remove lakeside,
fosshost1 and fosshost2.
Arun Isaac
Christopher Baines
Giovanni Biscuolo
Ludovic Courtès
Lars-Dominik Braun
Maxim Cournoyer
Efraim Flashner
Simon Tournier