refactor: fetch modules from bling

This commit is contained in:
ER 2023-09-10 17:31:25 +03:00
parent 393d54ec8e
commit 4e0095bc96
22 changed files with 31 additions and 434 deletions

View file

@ -30,9 +30,14 @@ COPY --from=ghcr.io/ublue-os/bling:latest /files /tmp/bling/files
# Copy build scripts & configuration
COPY build.sh /tmp/build.sh
COPY modules /tmp/modules/
COPY config /tmp/config/
# Copy modules
# The default modules are inside ublue-os/bling
COPY --from=ghcr.io/ublue-os/bling:latest /modules /tmp/modules/
# Custom modules overwrite defaults
COPY modules /tmp/modules/
# `yq` is used for parsing the yaml configuration
# It is copied from the official container image since it's not available as an RPM.
COPY --from=docker.io/mikefarah/yq /usr/bin/yq /usr/bin/yq

View file

@ -18,9 +18,7 @@ At the top of the recipe, there are four *mandatory* configuration options.
The core of startingpoint's configuration is built around the idea of modules. Modules are scripts in the [`../modules`](../modules/) directory that you list out under `modules:` in the recipe. They are executed in order, and can run arbitrary shell commands and write any files.
This repository comes with some modules out of the box, like [`rpm-ostree`](../modules/rpm-ostree) for pseudo-declarative package management, [`bling`](../modules/bling) for pulling extra components from [`ublue-os/bling`](https://github.com/ublue-os/bling), and [`files`](../modules/files) for copying files from the `config/files/` directory into your image. For a comprehensive list of modules, check out [the modules directory](../modules/).
For more in-depth documentation on each module, check out the README.md files in each module folder.
This repository fetches some useful default modules from [`ublue-os/bling`](https://github.com/ublue-os/bling/), like [`rpm-ostree`](../modules/rpm-ostree) for pseudo-declarative package management, [`bling`](../modules/bling) for pulling extra components from [`ublue-os/bling`](https://github.com/ublue-os/bling), and [`files`](../modules/files) for copying files from the `config/files/` directory into your image. For a comprehensive list of modules and their in-depth documentation, check out [the modules page on the website](https://universal-blue.org/tinker/modules/).
### Including modules from other files and building multiple images
@ -30,30 +28,4 @@ Module configuration can be included from other files using the `from-file` synt
```yml
modules:
- from-file: common-packages.yml
```
### Making modules
If you want to extend Startingpoint with custom functionality that requires configuration, you should create a module. Modules are scripts in the subdirectories of the [`../modules`](../modules/) directory. The `type:` key in the recipe.yml should be used as both the name of the folder and script, with the script having an additional `.sh` suffix.
Each module intended for public usage should include a `README.md` file inside it's directory with a short description of the module and documentation for each configuration option.
Modules get only the configuration options given to them in the recipe.yml, not the configuration of other modules or any top-level keys. The configuration is given as the first argument as a single-line json string. You can check out the default modules for examples on how to parse such string using `yq` or `jq`.
Additionally, each module has access to four environment variables, `CONFIG_DIRECTORY` pointing to the Startingpoint directory in `/usr/share/ublue-os/`, `IMAGE_NAME` being the name of the image as declared in the recipe, `BASE_IMAGE` being the URL of the container image used as the base (FROM) in the image, and `OS_VERSION` being the `VERSION_ID` from `/usr/lib/os-release`.
A helper bash function called `get_yaml_array` is exported from the main build script.
```bash
# "$1" is the first cli argument, being the module configuration.
# If you need to read from some other JSON string, just replace "$1" with "$VARNAME".
get_yaml_array OUTPUT_VAR_NAME '.yq.key.to.array[]' "$1"
for THING in "${OUTPUT_VAR_NAME[@]}"; do
echo "$THING"
done
```
All bash-based modules should start with the following lines to ensure the image builds fail on errors, and that the correct shell is used to run them.
```bash
#!/usr/bin/env bash
set -oue pipefail
```

View file

@ -1,3 +1,25 @@
# Modules
# Making modules
This directory includes all the different modules for Startingpoint. Refer to the README files inside each module's corresponding directory for in-depth documentation, and the README inside `config/` for more general documentation.
If you want to extend Startingpoint with custom functionality that requires configuration, you should create a module. Modules are scripts in the subdirectories of this directory. The `type:` key in the recipe.yml should be used as both the name of the folder and script, with the script having an additional `.sh` suffix. Creating a custom module with the same name as a default module will override it.
Each module intended for public usage should include a `README.md` file inside it's directory with a short description of the module and documentation for each configuration option.
Modules get only the configuration options given to them in the recipe.yml, not the configuration of other modules or any top-level keys. The configuration is given as the first argument as a single-line json string. You can check out the default modules for examples on how to parse such string using `yq` or `jq`.
Additionally, each module has access to four environment variables, `CONFIG_DIRECTORY` pointing to the Startingpoint directory in `/usr/share/ublue-os/`, `IMAGE_NAME` being the name of the image as declared in the recipe, `BASE_IMAGE` being the URL of the container image used as the base (FROM) in the image, and `OS_VERSION` being the `VERSION_ID` from `/usr/lib/os-release`.
A helper bash function called `get_yaml_array` is exported from the main build script.
```bash
# "$1" is the first cli argument, being the module configuration.
# If you need to read from some other JSON string, just replace "$1" with "$VARNAME".
get_yaml_array OUTPUT_VAR_NAME '.yq.key.to.array[]' "$1"
for THING in "${OUTPUT_VAR_NAME[@]}"; do
echo "$THING"
done
```
All bash-based modules should start with the following lines to ensure the image builds fail on errors, and that the correct shell is used to run them.
```bash
#!/usr/bin/env bash
set -oue pipefail
```

View file

@ -1,20 +0,0 @@
# [`bling`](https://github.com/ublue-os/bling) Module for Startingpoint
The `bling` module allows you to easily declare which general parts of `ublue-os/bling` to pull in to your custom image. It requires the `rpms` and `files` directories from the `bling` container to already exist inside `/tmp/bling/` (pulled inside the Containerfile by default).
The blingbling to pull in is declared under `install:`, and the code for installing them is all in simple named scripts under the `installers/` directory. The basic code for the `bling` module is very similar to the code of the `script` module.
## Example configuration:
```yml
type: bling # configure what to pull in from ublue-os/bling
install:
- fonts # selection of common good free fonts
- justfiles # add "!include /usr/share/ublue-os/just/bling.just"
# in your custom.just (added by default) or local justfile
- nix-installer # these are the silverblue nix installer scripts from dnkmmr69420
- ublue-os-wallpapers
# - ublue-update # https://github.com/ublue-os/ublue-update
# - dconf-update-service # a service unit that updates the dconf db on boot
# - devpod # https://devpod.sh/ as an rpm
```

View file

@ -1,19 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
get_yaml_array INSTALL '.install[]' "$1"
export BLING_DIRECTORY="/tmp/bling"
cd "/tmp/modules/bling/installers"
# Make every bling installer executable
find "$PWD" -type f -exec chmod +x {} \;
for ITEM in "${INSTALL[@]}"; do
echo "Pulling from bling: $ITEM"
# The trainling newline from $ITEM is removed
eval "$PWD/${ITEM%$'\n'}.sh"
done

View file

@ -1,7 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
cp -r "$BLING_DIRECTORY/files/usr/etc/systemd/system/dconf-update.service" "/usr/etc/systemd/system/dconf-update.service"
systemctl enable dconf-update.services

View file

@ -1,6 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
rpm-ostree install "$BLING_DIRECTORY"/rpms/devpod*.rpm

View file

@ -1,6 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
cp -r "$BLING_DIRECTORY"/files/usr/share/fonts/* "/usr/share/fonts"

View file

@ -1,6 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
cp -r "$BLING_DIRECTORY"/files/usr/share/ublue-os/just/* "/usr/share/ublue-os/just"

View file

@ -1,7 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
cp "$BLING_DIRECTORY/files/usr/bin/ublue-nix-install" "/usr/bin/ublue-nix-install"
cp "$BLING_DIRECTORY/files/usr/bin/ublue-nix-uninstall" "/usr/bin/ublue-nix-uninstall"

View file

@ -1,6 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
rpm-ostree install "$BLING_DIRECTORY"/rpms/ublue-os-wallpapers*.rpm

View file

@ -1,20 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
# Check if ublue-os-update-services rpm is installed, these services conflict with ublue-update
if rpm -q ublue-os-update-services > /dev/null; then
rpm-ostree override remove ublue-os-update-services
fi
# Change the conflicting update policy for rpm-ostreed
RPM_OSTREE_CONFIG="/usr/etc/rpm-ostreed.conf"
if [[ -f $RPM_OSTREE_CONFIG ]]; then
if [[ "$(get_config_value AutomaticUpdatePolicy $RPM_OSTREE_CONFIG)" == "stage" ]]; then
set_config_value AutomaticUpdatePolicy none $RPM_OSTREE_CONFIG
fi
fi
rpm-ostree install "$BLING_DIRECTORY"/rpms/ublue-update*.rpm

View file

@ -1,16 +0,0 @@
# `files` Module for Startingpoint
The `files` module simplifies the process of copying files to the image during the build time. These files are sourced from the `config/files` directory, which is located at `/tmp/config/files` inside the image.
> **Warning**
> If you want to place anything in `/etc` of the final image, you MUST place them in `/usr/etc` in your repo, so that they're written to `/usr/etc` on the final system. That is the proper directory for "system" configuration templates on immutable Fedora distros, whereas the normal `/etc` is meant for manual overrides and editing by the machine's admin AFTER installation! See issue https://github.com/ublue-os/startingpoint/issues/28.
## Example Configuration:
```yaml
type: files
files:
usr: /usr
```
In the example above, `usr` represents the directory located inside the `config/files` in the repository, while `/usr` designates the corresponding destination within the image.

View file

@ -1,33 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
get_yaml_array FILES '.files[]' "$1"
cd "$CONFIG_DIRECTORY/files"
if [[ ${#FILES[@]} -gt 0 ]]; then
echo "Adding files to image"
for pair in "${FILES[@]}"; do
FILE="$PWD/$(echo $pair | yq 'to_entries | .[0].key')"
DEST=$(echo $pair | yq 'to_entries | .[0].value')
if [ -d "$FILE" ]; then
if [ ! -d "$DEST" ]; then
mkdir -p "$DEST"
fi
echo "Copying $FILE to $DEST"
cp -r "$FILE"/* $DEST
elif [ -f "$FILE" ]; then
DEST_DIR=$(dirname "$DEST")
if [ ! -d "$DEST_DIR" ]; then
mkdir -p "$DEST_DIR"
fi
echo "Copying $FILE to $DEST"
cp $FILE $DEST
else
echo "File or Directory $FILE Does Not Exist in $CONFIG_DIRECTORY/files"
exit 1
fi
done
fi

View file

@ -1,33 +0,0 @@
# [`rpm-ostree`](https://coreos.github.io/rpm-ostree/) Module for Startingpoint
The `rpm-ostree` module offers pseudo-declarative package and repository management using `rpm-ostree`.
The module first downloads the repository files from repositories declared under `repos:` into `/etc/yum.repos.d/`. The magic string `%OS_VERSION%` is substituted with the current VERSION_ID (major Fedora version), which can be used, for example, for pulling correct versions of repositories from [Fedora's Copr](https://copr.fedorainfracloud.org/).
Then the module installs the packages declared under `install:` using `rpm-ostree install`, it removes the packages declared under `remove:` using `rpm-ostree override remove`. If there are packages declared under both `install:` and `remove:` a hybrid command `rpm-ostree remove <packages> --install <packages>` is used, which should allow you to switch required packages for other ones.
Additionally, the `rpm-ostree` module supports a temporary (waiting for `rpm-ostree` issue [#233](https://github.com/coreos/rpm-ostree/issues/233)) fix for packages that install into `/opt/`. Installation for packages that install into folder names declared under `optfix:` are fixed using some symlinks.
## Example Configuration:
```yml
type: rpm-ostree
repos:
- https://copr.fedorainfracloud.org/coprs/atim/starship/repo/fedora-%OS_VERSION%/atim-starship-fedora-%OS_VERSION%.repo
install:
- python3-pip
- libadwaita
remove:
- firefox
- firefox-langpacks
```
## Known issues
When removing certain packages, some problem probably in upstream `rpm-ostree` causes a `depsolve` issue similar to below. [Removed packages are still present in the underlying ostree repository](https://coreos.github.io/rpm-ostree/administrator-handbook/#removing-a-base-package), what `remove` does is "hide" them from the system, it doesn't reclaim disk space.
```
Resolving dependencies...done
error: Could not depsolve transaction; 1 problem detected:
Problem: conflicting requests
```

View file

@ -1,58 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
# Pull in repos
get_yaml_array REPOS '.repos[]' "$1"
if [[ ${#REPOS[@]} -gt 0 ]]; then
echo "Adding repositories"
for REPO in "${REPOS[@]}"; do
REPO="${REPO//%OS_VERSION%/${OS_VERSION}}"
wget "${REPO//[$'\t\r\n ']}" -P "/etc/yum.repos.d/"
done
fi
# Create symlinks to fix packages that create directories in /opt
get_yaml_array OPTFIX '.optfix[]' "$1"
if [[ ${#OPTFIX[@]} -gt 0 ]]; then
echo "Creating symlinks to fix packages that install to /opt"
# Create symlink for /opt to /var/opt since it is not created in the image yet
mkdir -p "/var/opt"
ln -s "/var/opt" "/opt"
# Create symlinks for each directory specified in recipe.yml
for OPTPKG in "${OPTFIX[@]}"; do
OPTPKG="${OPTPKG%\"}"
OPTPKG="${OPTPKG#\"}"
OPTPKG=$(printf "$OPTPKG")
mkdir -p "/usr/lib/opt/${OPTPKG}"
ln -s "../../usr/lib/opt/${OPTPKG}" "/var/opt/${OPTPKG}"
echo "Created symlinks for ${OPTPKG}"
done
fi
get_yaml_array INSTALL '.install[]' "$1"
get_yaml_array REMOVE '.remove[]' "$1"
# The installation is done with some wordsplitting hacks
# because of errors when doing array destructuring at the installation step.
# This is different from other ublue projects and could be investigated further.
INSTALL_STR=$(echo "${INSTALL[*]}" | tr -d '\n')
REMOVE_STR=$(echo "${REMOVE[*]}" | tr -d '\n')
# Install and remove RPM packages
if [[ ${#INSTALL[@]} -gt 0 && ${#REMOVE[@]} -gt 0 ]]; then
echo "Installing & Removing RPMs"
echo "Installing: ${INSTALL_STR[*]}"
echo "Removing: ${REMOVE_STR[*]}"
# Doing both actions in one command allows for replacing required packages with alternatives
rpm-ostree override remove $REMOVE_STR $(printf -- "--install=%s " $INSTALL_STR)
elif [[ ${#INSTALL[@]} -gt 0 ]]; then
echo "Installing RPMs"
echo "Installing: ${INSTALL_STR[*]}"
rpm-ostree install $INSTALL_STR
elif [[ ${#INSTALL[@]} -gt 0 ]]; then
echo "Removing RPMs"
echo "Removing: ${REMOVE_STR[*]}"
rpm-ostree override remove $REMOVE_STR
fi

View file

@ -1,26 +0,0 @@
# `script` Module for Startingpoint
The `script` module can be used to run arbitrary scripts at image build time that take no or minimal external configuration (in the form of command line arguments).
The scripts, which are run from the `config/scripts` directory, are declared under `scripts:`.
## Example Configuration
```yml
type: script
scripts:
- signing.sh
```
## Creating a Script
Look at `example.sh` for an example shell script. You can rename and copy the file for your own purposes. In order for the script to be executed, declare it in the recipe
When creating a script, please make sure
- ...its filename ends with `.sh`.
- This follows convention for (especially bash) shell scripts.
- `autorun.sh` only executes files that match `*.sh`.
- ...it starts with a [shebang](<https://en.wikipedia.org/wiki/Shebang_(Unix)>) like `#!/usr/bin/env bash`.
- This ensures the script is ran with the correct interpreter / shell.
- ...it contains the command `set -oue pipefail` near the start.
- This will make the image build fail if your script fails. If you do not care if your script works or not, you can omit this line.

View file

@ -1,16 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
get_yaml_array SCRIPTS '.scripts[]' "$1"
cd "$CONFIG_DIRECTORY/scripts"
# Make every script executable
find "$PWD" -type f -exec chmod +x {} \;
for SCRIPT in "${SCRIPTS[@]}"; do
echo "Running script $SCRIPT"
eval "$PWD/$SCRIPT"
done

View file

@ -1,41 +0,0 @@
# `systemd` Module for Startingpoint
The `systemd` module streamlines the management of systemd units during image building. Units are divided into `system` and `user` categories, with `system` units managed directly using `systemctl` and `user` units using `systemctl --user`. You can specify which units to enable or disable under each category.
## Example Configuration:
```yaml
type: systemd
system:
enable:
- example.service
disable:
- example.target
user:
enable:
- example.timer
disable:
- example.service
```
In this example:
### System Units
- `example.service`: Enabled (runs on system boot)
- `example.target`: Disabled (does not run on system boot)
### User Units
- `example.timer`: Enabled (runs for the user)
- `example.service`: Disabled (does not run for the user)
This configuration achieves the same results as the following commands:
```sh
# System Units
systemctl enable example.service
systemctl disable example.target
# User Units
systemctl --user enable example.timer
systemctl --user disable example.service
```

View file

@ -1,35 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
get_yaml_array ENABLED '.system.enabled[]' "$1"
get_yaml_array DISABLED '.system.disabled[]' "$1"
get_yaml_array USER_ENABLED '.user.enabled[]' "$1"
get_yaml_array USER_DISABLED '.user.disabled[]' "$1"
if [[ ${#ENABLED[@]} -gt 0 ]]; then
for unit in "${ENABLED[@]}"; do
unit=$(printf "$unit")
systemctl enable $unit
done
fi
if [[ ${#DISABLED[@]} -gt 0 ]]; then
for unit in "${DISABLED[@]}"; do
unit=$(printf "$unit")
systemctl disable $unit
done
fi
if [[ ${#USER_ENABLED[@]} -gt 0 ]]; then
for unit in "${ENABLED[@]}"; do
unit=$(printf "$unit")
systemctl --user enable $unit
done
fi
if [[ ${#USER_DISABLED[@]} -gt 0 ]]; then
for unit in "${DISABLED[@]}"; do
unit=$(printf "$unit")
systemctl --user disable $unit
done
fi

View file

@ -1,16 +0,0 @@
# [`yafti`](https://github.com/ublue-os/yafti) Module for Startingpoint
If included, the `yafti` module will install `yafti` and set it up to run on first boot.
Optionally, a list of Flatpak names and IDs can be included under `custom-flatpaks:`. These will be enabled by default under their own section on the Flatpak installation screen of `yafti`.
The main `yafti` configuration file, `yafti.yml`, is in `/usr/share/ublue-os/firstboot/yafti.yml` and can be edited for a more custom first-boot experience.
## Example configuration:
```yml
type: yafti
custom-flatpaks:
- Celluloid: io.github.celluloid_player.Celluloid
- Krita: org.kde.krita
```

View file

@ -1,31 +0,0 @@
#!/usr/bin/env bash
# Tell build process to exit if there are any errors.
set -oue pipefail
FIRSTBOOT_DATA="/usr/share/ublue-os/firstboot"
FIRSTBOOT_LINK="/usr/etc/profile.d/ublue-firstboot.sh"
echo "Installing python3-pip and libadwaita"
rpm-ostree install python3-pip libadwaita
echo "Installing and enabling yafti"
pip install --prefix=/usr yafti
# Create symlink to our profile script, which creates the per-user "autorun yafti" links.
mkdir -p "$(dirname "${FIRSTBOOT_LINK}")"
ln -s "${FIRSTBOOT_DATA}/launcher/login-profile.sh" "${FIRSTBOOT_LINK}"
YAFTI_FILE="$FIRSTBOOT_DATA/yafti.yml"
get_yaml_array FLATPAKS '.custom-flatpaks[]' "$1"
if [[ ${#FLATPAKS[@]} -gt 0 ]]; then
echo "Adding Flatpaks to yafti.yml"
yq -i '.screens.applications.values.groups.Custom.description = "Flatpaks suggested by the image maintainer."' "${YAFTI_FILE}"
yq -i '.screens.applications.values.groups.Custom.default = true' "${YAFTI_FILE}"
for pkg in "${FLATPAKS[@]}"; do
echo "Adding to yafti: ${pkg}"
yq -i ".screens.applications.values.groups.Custom.packages += [$pkg]" "${YAFTI_FILE}"
done
fi