kubernetes-bootstrap: apply minimal bootstrap helm charts on switch (#5) #23

hermes-bot · 2026-10-14T09:58:05Z

hermes-bot commented

2026-10-14 09:58:05 +00:00

Implements the shape rpickler asked for in #5: charts rendered to a derivation, injected into the cluster by a systemd service during nixos-rebuild switch — and scoped to the absolute minimum.

What's in the slice

modules/kubernetes/bootstrap/default.nix — new module, option kubernetes.bootstrap:
- renders the btrfs-nfs-csi chart with helm template inside a stdenv derivation (pure, builds without a cluster). The chart is taken from the existing inputs.btrfs-nfs-csi flake pin, so chart version and the agent binary services.btrfs-nfs-csi ships to horse come from one pin and move together on nix flake update btrfs-nfs-csi — the version-coupling argument from the issue.
- systemd.services.kubernetes-bootstrap: oneshot reconciler, wantedBy = kubernetes.target, waits for the API server and retries via Restart=on-failure (unit start ≠ apiserver ready on a fresh cluster). Applies /etc/kubernetes/bootstrap-manifests with kubectl apply --server-side --prune -l app.kubernetes.io/managed-by=Helm — declarative, prunes removed resources, and never touches the runtime Secret (unlabelled).
- StorageClass set is parameterised; defaults keep parity with the current helmfile release (btrfs-nfs-nosync, btrfs-nfs-backup, kubeletDir = /var/lib/kubernetes, hostNetwork driver).
- Agent token never enters the Nix store: manifests reference a pre-existing Secret (existingSecret), created once at runtime from a sops-nix secret file. Verified against the driver source that the controller reads key agentToken from that Secret.
- snapshotClass = false by default: the chart's VolumeSnapshotClass needs the snapshot CRDs + controller, which stock services.kubernetes doesn't ship; enabling a snapshotClass would fail apply.
hosts/lurch.nix — the config block with enable = false (gated), sops key btrfs-csi-agent-token declared under mkIf enable. Flip-on procedure is in the code comment: sops key first, then enable = true.

What's deliberately NOT here

No nix-kube-generators/nixhelm input. The rendering mechanism is ~25 lines of helm template in a derivation (same thing nix-kube-generators' buildHelmChart does); for one bootstrap chart that dependency isn't worth pinning. Revisit when the ArgoCD slice arrives if we start vendoring many charts.
metallb / traefik / argocd: per the issue, those converge under ArgoCD; the argocd bootstrap itself is the natural follow-up child of this (its manifests drop into the same module + service).

Not enabled yet by design — enabling is a separate, visible switch after the sops key exists, and re-applies are idempotent once live.

No Nix runner available in this environment to nix eval the lurch config; changes were reviewed against the module sources of nixpkgs services.kubernetes (kubernetes.target, etcClusterAdminKubeconfig = kubernetes/cluster-admin.kubeconfig) and the chart at the pinned rev. Flagging honestly for review.

Implements the shape rpickler asked for in #5: charts rendered to a derivation, injected into the cluster by a systemd service during `nixos-rebuild switch` — and scoped to the absolute minimum. **What's in the slice** - `modules/kubernetes/bootstrap/default.nix` — new module, option `kubernetes.bootstrap`: - renders the **btrfs-nfs-csi** chart with `helm template` inside a stdenv derivation (pure, builds without a cluster). The chart is taken from the existing `inputs.btrfs-nfs-csi` flake pin, so chart version and the agent binary `services.btrfs-nfs-csi` ships to horse come from **one pin** and move together on `nix flake update btrfs-nfs-csi` — the version-coupling argument from the issue. - `systemd.services.kubernetes-bootstrap`: oneshot reconciler, `wantedBy = kubernetes.target`, waits for the API server and retries via `Restart=on-failure` (unit start ≠ apiserver ready on a fresh cluster). Applies `/etc/kubernetes/bootstrap-manifests` with `kubectl apply --server-side --prune -l app.kubernetes.io/managed-by=Helm` — declarative, prunes removed resources, and never touches the runtime Secret (unlabelled). - StorageClass set is parameterised; defaults keep parity with the current helmfile release (`btrfs-nfs-nosync`, `btrfs-nfs-backup`, `kubeletDir = /var/lib/kubernetes`, hostNetwork driver). - **Agent token never enters the Nix store**: manifests reference a pre-existing Secret (`existingSecret`), created once at runtime from a sops-nix secret file. Verified against the driver source that the controller reads key `agentToken` from that Secret. - `snapshotClass = false` by default: the chart's VolumeSnapshotClass needs the snapshot CRDs + controller, which stock `services.kubernetes` doesn't ship; enabling a snapshotClass would fail apply. - `hosts/lurch.nix` — the config block with `enable = false` (gated), sops key `btrfs-csi-agent-token` declared under `mkIf enable`. Flip-on procedure is in the code comment: sops key first, then `enable = true`. **What's deliberately NOT here** - No `nix-kube-generators`/`nixhelm` input. The rendering mechanism is ~25 lines of `helm template` in a derivation (same thing nix-kube-generators' `buildHelmChart` does); for one bootstrap chart that dependency isn't worth pinning. Revisit when the ArgoCD slice arrives if we start vendoring many charts. - metallb / traefik / argocd: per the issue, those converge under ArgoCD; the argocd bootstrap itself is the natural follow-up child of this (its manifests drop into the same module + service). Not enabled yet by design — enabling is a separate, visible switch after the sops key exists, and re-applies are idempotent once live. No Nix runner available in this environment to `nix eval` the lurch config; changes were reviewed against the module sources of nixpkgs `services.kubernetes` (kubernetes.target, `etcClusterAdminKubeconfig = kubernetes/cluster-admin.kubeconfig`) and the chart at the pinned rev. Flagging honestly for review.

hermes-bot added 1 commit

2026-10-14 09:58:05 +00:00

kubernetes-bootstrap: apply minimal bootstrap helm charts on switch (issue #5 ) ef272e569e

- modules/kubernetes/bootstrap: render the btrfs-nfs-csi chart from the
  pinned flake input via helm template (pure derivation), apply with a
  systemd oneshot reconciler on lurch during nixos-rebuild switch.
- Chart + agent binary share one pin (inputs.btrfs-nfs-csi), per
  rpickler: tied to the same version installed on horse.
- Agent token never enters the store: manifests link a pre-existing
  Secret created at runtime from a sops file.
- hosts/lurch.nix: config block, enable=false until the sops key lands.
- Nothing else changes; metallb/traefik/argocd converge under ArgoCD.

hermes-bot referenced this pull request

2026-10-14 10:00:20 +00:00

Start bootstrap helm charts #5

rpickler commented

2026-10-31 15:50:25 +00:00

I attempted to switch to this branch on lurch, and nothing happened. The expected service didn't install. Now that you have nix underpinnings, can you review your work and ensure that it works as expected?

hermes-bot added 1 commit

2026-10-31 18:06:32 +00:00

kubernetes-bootstrap: make the lurch switch actually work (issue #5 ) 7135e8a7c9

Root cause of 'nothing happened on switch': hosts/lurch.nix shipped
with enable = false, gated on a sops key (btrfs-csi-agent-token) that
does not exist in modules/secrets.yaml — flipping the gate would have
broken activation on the missing secret, and enabling the module at all
still failed with 'attribute inputs missing': lurch is the one host
whose specialArgs is the bare inputs attrset, so the module argument
`inputs` was never provided there.

- flake.nix: lurch specialArgs becomes inputs // { inherit inputs; }
  (additive; keeps the per-input named args).
- modules/kubernetes/csi-agent-token.nix: single source of truth for
  the agent tenant token; referenced by both the agent
  (btrfs-csi-driver.nix AGENT_TENANTS) and the cluster-side Secret, so
  both sides move together on rotation.
- bootstrap module: agentTokenSecretFile now defaults to a store file
  with that token (status-quo parity with the deployed agent; the
  token is already committed via the agent env file), overridable with
  a sops path — rotation path documented.
- hosts/lurch.nix: enable = true; sops gate removed.

Verified with nix: bootstrap manifests derivation builds (helm
renders 11 resources, all managed-by=Helm labelled, no token in the
store output, secret refs match the apply script), lurch/horse/moose
toplevels eval.

hermes-bot commented

2026-10-31 18:08:58 +00:00

Root cause found — nothing happened because it was still gated off, and two further bugs would have kept it from working even after unlocking the gate. Fixed and pushed in 7135e8a; lurch's full system config now evaluates and the manifest derivation builds.

Why the switch did nothing

hosts/lurch.nix shipped enable = false (the gate from my original PR description), so no unit was installed at all.
Flipping the gate would have failed activation: sops.secrets.btrfs-csi-agent-token points at a key that does not exist in modules/secrets.yaml.
Even with the key, the module would crash with attribute 'inputs' missing: lurch is the only host wired as specialArgs = inputs instead of the repo's { inherit inputs self; } convention, so the module arg inputs (used for inputs.btrfs-nfs-csi) was never provided. The gate masked this because with enable = false nothing forced the arg.

Fixes (7135e8a)

flake.nix: lurch specialArgs = inputs // { inherit inputs; } — strictly additive, keeps the per-input named args other lurch modules may use.
New modules/kubernetes/csi-agent-token.nix: single source of truth for the agent tenant token, read by both the cluster side (bootstrap Secret) and the agent side (btrfs-csi-driver.nix AGENT_TENANTS), so they can't drift and rotate together.
Bootstrap module: agentTokenSecretFile now defaults to a store file carrying that token, overridable with a sops path. Honest note: the token here is the placeholder already committed in btrfs-csi-driver.nix (default:DEADBEEF...) — the agent on horse runs with it today, so the bootstrap matches the deployed reality instead of being blocked on a secret nobody has. The sops hardening path is documented in the new file.
hosts/lurch.nix: enable = true, sops gate removed. option type on agentTokenSecretFile accepts paths or strings.

Verified with nix (this time for real)

nix build of the render derivation: helm template renders 11 resources, every object carries app.kubernetes.io/managed-by: Helm (the prune selector), both StorageClasses reference secret btrfs-nfs-csi-agent-token in ns btrfs-nfs-csi — exactly what the apply script creates — kubeletDir=/var/lib/kubernetes, hostNetwork on, no VolumeSnapshotClass, and no token value anywhere in the rendered output. (Built via a standalone x86 harness since lurch is aarch64 and this dev box can't build the Pi closure; YAML output is arch-independent.)
nixosConfigurations.{lurch,horse,moose}.config.system.build.toplevel.drvPath all eval.
kubernetes.target is defined by the pinned nixpkgs (wantedBy multi-user) and /etc/kubernetes/cluster-admin.kubeconfig is what the module actually generates (etcClusterAdminKubeconfig), so the unit wiring and kubeconfig path in the script are correct.

Remaining caveat: the apply against the live cluster itself (server-side adopting the helmfile-created resources via --force-conflicts) can only be exercised on lurch — switching should now install kubernetes-bootstrap.service, start it on the switch, wait for the API server and converge. The placeholder token is the one real security debt; happy to take a follow-up issue for the sops rotation.

Root cause found — `nothing happened` because it was still gated off, and two further bugs would have kept it from working even after unlocking the gate. Fixed and pushed in 7135e8a; lurch's full system config now evaluates and the manifest derivation builds. **Why the switch did nothing** 1. `hosts/lurch.nix` shipped `enable = false` (the gate from my original PR description), so no unit was installed at all. 2. Flipping the gate would have failed activation: `sops.secrets.btrfs-csi-agent-token` points at a key that does not exist in `modules/secrets.yaml`. 3. Even with the key, the module would crash with `attribute 'inputs' missing`: lurch is the only host wired as `specialArgs = inputs` instead of the repo's `{ inherit inputs self; }` convention, so the module arg `inputs` (used for `inputs.btrfs-nfs-csi`) was never provided. The gate masked this because with `enable = false` nothing forced the arg. **Fixes (7135e8a)** - `flake.nix`: lurch `specialArgs = inputs // { inherit inputs; }` — strictly additive, keeps the per-input named args other lurch modules may use. - New `modules/kubernetes/csi-agent-token.nix`: single source of truth for the agent tenant token, read by **both** the cluster side (bootstrap Secret) and the agent side (`btrfs-csi-driver.nix` `AGENT_TENANTS`), so they can't drift and rotate together. - Bootstrap module: `agentTokenSecretFile` now defaults to a store file carrying that token, overridable with a sops path. Honest note: the token here is the placeholder already committed in `btrfs-csi-driver.nix` (`default:DEADBEEF...`) — the agent on horse runs with it today, so the bootstrap matches the deployed reality instead of being blocked on a secret nobody has. The sops hardening path is documented in the new file. - `hosts/lurch.nix`: `enable = true`, sops gate removed. `option type` on `agentTokenSecretFile` accepts paths or strings. **Verified with nix (this time for real)** - `nix build` of the render derivation: `helm template` renders 11 resources, every object carries `app.kubernetes.io/managed-by: Helm` (the prune selector), both StorageClasses reference secret `btrfs-nfs-csi-agent-token` in ns `btrfs-nfs-csi` — exactly what the apply script creates — `kubeletDir=/var/lib/kubernetes`, `hostNetwork` on, no VolumeSnapshotClass, and no token value anywhere in the rendered output. (Built via a standalone x86 harness since lurch is aarch64 and this dev box can't build the Pi closure; YAML output is arch-independent.) - `nixosConfigurations.{lurch,horse,moose}.config.system.build.toplevel.drvPath` all eval. - `kubernetes.target` is defined by the pinned nixpkgs (wantedBy multi-user) and `/etc/kubernetes/cluster-admin.kubeconfig` is what the module actually generates (`etcClusterAdminKubeconfig`), so the unit wiring and kubeconfig path in the script are correct. Remaining caveat: the apply against the live cluster itself (server-side adopting the helmfile-created resources via `--force-conflicts`) can only be exercised on lurch — switching should now install `kubernetes-bootstrap.service`, start it on the switch, wait for the API server and converge. The placeholder token is the one real security debt; happy to take a follow-up issue for the sops rotation.

rpickler requested changes

2026-11-01 09:52:00 +00:00

rpickler left a comment

Break up the bootstrap/default.nix file to make room for other charts/modules in the future.

modules/kubernetes/bootstrap/default.nix Outdated

					
				@ -0,0 +171,4 @@

				      apply Nix-rendered bootstrap Helm charts to the local cluster on

				      nixos-rebuild switch'';

				    btrfs-nfs-csi = lib.mkOption {

It really seems like the btrfs-nfs-csi portion of this should be separated out into a separate submodule next to this bootstrap/default.nix file. This is to make room for other helm charts/modules we'll eventually add to the bootstrap mechanism. This default.nix should be left rendering and installing the charts provided by the other modules.

It really seems like the btrfs-nfs-csi portion of this should be separated out into a separate submodule next to this bootstrap/default.nix file. This is to make room for other helm charts/modules we'll eventually add to the bootstrap mechanism. This `default.nix` should be left rendering and installing the charts provided by the other modules.

hermes-bot added 1 commit

2026-11-02 23:47:41 +00:00

kubernetes-bootstrap: split btrfs-nfs-csi chart into its own submodule (PR #5 review) 77236daf14

default.nix now holds only the generic render+apply mechanism: charts
register via kubernetes.bootstrap.charts (enable/namespace/chart/values/
setupScript); enabled charts are rendered with helm template, collected
and applied by the single reconciler unit.  The btrfs-nfs-csi chart —
settings, values mapping, token/namespace setupScript and the
storageClasses assertion — lives next to it in btrfs-nfs-csi.nix, ready
as the template for further bootstrap charts.  Option surface for hosts
is unchanged; rendered output is byte-identical.

hermes-bot commented

2026-11-02 23:49:50 +00:00

Review points implemented in 77236da (branch issue-5-bootstrap-helm).

No hardcoded chart in the machinery. modules/kubernetes/bootstrap/default.nix now knows nothing about individual charts: chart submodules register entries in a generic kubernetes.bootstrap.charts.<name> option (namespace/chart/values/kubeVersion/setupScript). Everything registered + enabled is rendered with helm template into one manifest dir and applied by the single reconciler unit; per-chart setupScript hooks run between API-ready and apply. btrfs-nfs-csi lives in its own btrfs-nfs-csi.nix beside it.

Charts are optional. Each entry has enable; the btrfs chart participates by default when the bootstrap is on, and a host can turn it off per-chart via kubernetes.bootstrap.btrfs-nfs-csi.enable = false (or directly charts.btrfs-nfs-csi.enable = false). Assertion guards the empty-set case.

One NixOS module-system trap hit and fixed: gating the chart registration with config = { ... } // lib.optionalAttrs cfg.enable { ... } infinite-recurses (the // forces the config read at module-prep phase); the deferred lib.mkIf cfg.enable { ... } form is the working idiom — kept in a code comment so the next chart module doesn't rediscover it.

Verification (all re-run on 77236da):

nix eval of full-system toplevel drvPath for lurch, horse and moose — all green.
Equivalence: rendered the bootstrap manifests from the previous tip (7135e8a) and the new tree side-by-side via a standalone x86 evalModules harness — diff -ru byte-identical (only the single pinned input, btrfs-nfs-csi, feeds the chart; same helm 3.20.2, same values).
Apply-unit script old-vs-new diff: semantically identical (variable inlining + comment trim only); wait-loop, idempotent namespace/Secret setup and the server-side apply with helm-label prune selector unchanged.
Both assertions evaluate true for the lurch-style config.

Review points implemented in 77236da (branch `issue-5-bootstrap-helm`). **No hardcoded chart in the machinery.** `modules/kubernetes/bootstrap/default.nix` now knows nothing about individual charts: chart submodules register entries in a generic `kubernetes.bootstrap.charts.<name>` option (namespace/chart/values/kubeVersion/setupScript). Everything registered + enabled is rendered with `helm template` into one manifest dir and applied by the single reconciler unit; per-chart `setupScript` hooks run between API-ready and apply. btrfs-nfs-csi lives in its own `btrfs-nfs-csi.nix` beside it. **Charts are optional.** Each entry has `enable`; the btrfs chart participates by default when the bootstrap is on, and a host can turn it off per-chart via `kubernetes.bootstrap.btrfs-nfs-csi.enable = false` (or directly `charts.btrfs-nfs-csi.enable = false`). Assertion guards the empty-set case. One NixOS module-system trap hit and fixed: gating the chart registration with `config = { ... } // lib.optionalAttrs cfg.enable { ... }` infinite-recurses (the `//` forces the config read at module-prep phase); the deferred `lib.mkIf cfg.enable { ... }` form is the working idiom — kept in a code comment so the next chart module doesn't rediscover it. Verification (all re-run on 77236da): - `nix eval` of full-system toplevel drvPath for lurch, horse and moose — all green. - Equivalence: rendered the bootstrap manifests from the previous tip (7135e8a) and the new tree side-by-side via a standalone x86 `evalModules` harness — `diff -ru` byte-identical (only the single pinned input, `btrfs-nfs-csi`, feeds the chart; same helm 3.20.2, same values). - Apply-unit script old-vs-new diff: semantically identical (variable inlining + comment trim only); wait-loop, idempotent namespace/Secret setup and the server-side apply with helm-label prune selector unchanged. - Both assertions evaluate true for the lurch-style config.

rpickler added 1 commit

2026-11-07 07:27:17 +00:00

Merge branch 'main' into issue-5-bootstrap-helm 426bde10c3

rpickler commented

2026-11-07 07:34:35 +00:00

Have an error when the kubernetes-bootstrap service tries to start:

Nov 07 07:33:22 lurch systemd[1]: Starting Apply Nix-rendered bootstrap Helm charts to the cluster...
Nov 07 07:33:22 lurch kubernetes-bootstrap-start[457308]: error: --prune is in alpha and doesn't currently work on objects created by server-side apply
Nov 07 07:33:22 lurch systemd[1]: kubernetes-bootstrap.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 07:33:22 lurch systemd[1]: kubernetes-bootstrap.service: Failed with result 'exit-code'.
Nov 07 07:33:22 lurch systemd[1]: Failed to start Apply Nix-rendered bootstrap Helm charts to the cluster.

Have an error when the kubernetes-bootstrap service tries to start: ``` Nov 07 07:33:22 lurch systemd[1]: Starting Apply Nix-rendered bootstrap Helm charts to the cluster... Nov 07 07:33:22 lurch kubernetes-bootstrap-start[457308]: error: --prune is in alpha and doesn't currently work on objects created by server-side apply Nov 07 07:33:22 lurch systemd[1]: kubernetes-bootstrap.service: Main process exited, code=exited, status=1/FAILURE Nov 07 07:33:22 lurch systemd[1]: kubernetes-bootstrap.service: Failed with result 'exit-code'. Nov 07 07:33:22 lurch systemd[1]: Failed to start Apply Nix-rendered bootstrap Helm charts to the cluster. ```

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin issue-5-bootstrap-helm:issue-5-bootstrap-helm

git switch issue-5-bootstrap-helm

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff issue-5-bootstrap-helm

git switch issue-5-bootstrap-helm

git rebase main

git switch main

git merge --ff-only issue-5-bootstrap-helm

git switch issue-5-bootstrap-helm

git rebase main

git switch main

git merge --no-ff issue-5-bootstrap-helm

git switch main

git merge --squash issue-5-bootstrap-helm

git switch main

git merge --ff-only issue-5-bootstrap-helm

git switch main

git merge issue-5-bootstrap-helm

git push origin main

Sign in to join this conversation.

No reviewers

No labels

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

richardpickler.com/nixos!23

Rows
Columns