richardpickler.com/nixos
1
0
Fork 0
Code Issues 9 Pull requests 5 Projects Releases Packages Wiki Activity Actions

Start bootstrap helm charts #5

New issue
Open
opened 2026-10-06 05:25:57 +00:00 by rpickler · 3 comments
rpickler commented 2026-10-06 05:25:57 +00:00
Owner
Copy link

I'm curious if the base level helm charts needed for cluster operations can be started from the nixos flake for the master of the kubernetes cluster. These repositories show some promise.

https://github.com/nix-community/nixhelm
https://github.com/farcaller/nix-kube-generators

@hermes-bot do an evaluation and propose an implementation in a response here.

I'm curious if the base level helm charts needed for cluster operations can be started from the nixos flake for the master of the kubernetes cluster. These repositories show some promise. https://github.com/nix-community/nixhelm https://github.com/farcaller/nix-kube-generators @hermes-bot do an evaluation and propose an implementation in a response here.
hermes-bot commented 2026-10-09 10:13:44 +00:00
Member
Copy link

Evaluation: nixhelm + nix-kube-generators as the bootstrap path

I audited both projects against our actual setup (richardpickler.com/nixos modules/kubernetes/*, and the 28 chart repos + ~28 releases in richardpickler.com/helm/helmfile.yaml). Short answer: yes, this can work, and the master flake is a reasonable place to start — but as a hybrid, not a big-bang helmfile replacement.

What each project actually is

  • nixhelm (nix-community, active — nightly chart bumps, last commit today) is not a renderer. It is ~83 upstream chart repos vendored as charts/<repo>/<chart> with version+hash pinned in Nix, so nix flake update nixhelm moves charts as reviewable commits instead of at deploy time. Public Cachix (nixhelm.cachix.org) serves prebuilt chart derivations. It accepts PRs adding charts via helmupdater.
  • nix-kube-generators (same author, deliberately tiny) provides buildHelmChart (runs helm template in the Nix sandbox → a manifests derivation) and fromHelm (same, parsed to a Nix attrset). It renders; it never installs. Note the maintenance caveat: ~27 commits total, last activity Nov 2025 — stable but not lively, so we should pin it by rev.

Coverage vs our helmfile.yaml

Already vendored in nixhelm: authentik, traefik (+ traefik-crds), prometheus-community, argoproj (argo-cd/workflows — the ArgoCD TODO item), nextcloud, forgejo-helm, kubernetes-sigs (node-feature-discovery), coredns, nvidia, plus longhorn/metallb/external-secrets/oauth2-proxy/kyverno if we ever want them.

Not vendored (would need upstream PRs or the fallback below): woodpecker-ci, wg-easy (slybase OCI), zigbee2mqtt, homeassistant, node-red (schwarzit), dependency-track, helmforge, btrfs-nfs-csi (ours — it's a git-repo chart anyway), adnoctem/fmjstudios, bokysan smtp, cowboysysop whoami, CrunchyData pgo/postgres, local-path-provisioner.

Important for the bootstrap question specifically: the cluster plumbing (CNI, DNS, scheduler, RBAC bootstrap) is already done by nixpkgs services.kubernetes (flannel, addons.dns, addonManager.bootstrapAddons) on lurch. So "bootstrap helm charts" for us is really: traefik (+CRDs) → authentik → storage → monitoring — and the first two are exactly what nixhelm covers.

Proposed implementation (phased)

Phase 0 — inputs on the nixos flake (the master, lurch, is the apply point):

inputs.nixhelm.url = "github:nix-community/nixhelm";
inputs.nix-kube-generators.url = "github:farcaller/nix-kube-generators";
# both: inputs.nixpkgs.follows = "nixpkgs"; (kube-generators takes pkgs explicitly)

Phase 1 — a bootstrap/ library in the nixos repo. One Nix file per bootstrap chart that takes chart + values (values moved out of helmfile.yaml into bootstrap/values/*.nix or .yaml) and produces a rendered-manifests derivation:

kubelib = inputs.nix-kube-generators.lib { inherit pkgs; };
traefikCharts = inputs.nixhelm.charts { inherit pkgs; };
traefik-manifests = kubelib.buildHelmChart {
  name = "traefik";
  chart = traefikCharts.traefik.traefik;
  namespace = "kube-system";
  valuesFile = ./values/traefik.yaml;
  kubeVersion = "v1.33.0";   # pin to whatever nixpkgs 26.05 kubernetes module ships
};

Expose as flake packages.<system> outputs so nix build .#traefik-manifests is a pure, reproducible render, and review happens as a git diff of rendered output.

Phase 2 — apply mechanism, deliberately thin. A nix run .#apply-bootstrap script (or a systemd.onActivation service on lurch gated by a flag) doing kubectl apply -f ${result} with the master's kubeconfig. Charts not in nixhelm render via kubelib.downloadHelmChart { repo = ...; chart = ...; version = ...; chartHash = ...; } — one hash-bootstrap build per chart, then pinned forever. That keeps helmfile as the escape hatch for the long tail instead of forcing ~12 upstream PRs up front.

Phase 3 — GitOps convergence. The helmfile TODO already says "Install ArgoCD & continuous deployment": rendered manifests from Phase 1 are exactly ArgoCD's preferred input. ArgoCD itself rendered from nixhelm.argoproj.argo-cd, self-hosting its own app definitions. At that point helmfile retires except where the long tail still lives.

Risks / things I'd verify in the pilot

  1. buildHelmChart renders without cluster validation: it can't catch bad-vs-cluster API drift, so pinning kubeVersion/apiVersions per chart matters (our apiserver version comes from nixpkgs, not from the chart repo).
  2. Charts with OCI subchart dependencies render fine, but any chart needing helm repo add side effects (we have two git+https:// repos) won't — those stay on downloadHelmChart/helmfile.
  3. Secrets: our helm-secrets/sops flow doesn't disappear. Keep secret material out of rendered manifests — sops-nix templates on the target (we already use sops-nix on lurch) or external-secrets later; rendering with decrypted values baked into a derivation would leak into the Nix store.
  4. Determinism caveat: helm template output ordering is stable but not guaranteed across helm upgrades; diffs will occasionally be noisy.
  5. fromHelm's YAML parsing had an octal-permissions bug upstream (fixed via their fromYAML workaround) — pin by rev, not branch.

Suggested first concrete step

A pilot PR rendering traefik + traefik-crds + authentik three ways — helmfile template vs nix build of the nixhelm chart vs nix-kube-generators render — and a helm diff-equivalent comparison against what's live on the cluster. If the diff is clean, Phase 2's apply wrapper is ~30 lines of Nix and we've got the bootstrap pattern. Happy to build that pilot on request.

— hermes-bot (evaluation: nixhelm tree/API + nix-kube-generators README/lib checked 2026-10-09; coverage cross-checked against all repos in helmfile.yaml)

## Evaluation: nixhelm + nix-kube-generators as the bootstrap path I audited both projects against our actual setup (`richardpickler.com/nixos` modules/kubernetes/*, and the 28 chart repos + ~28 releases in `richardpickler.com/helm/helmfile.yaml`). Short answer: **yes, this can work, and the master flake is a reasonable place to start — but as a hybrid, not a big-bang helmfile replacement.** ### What each project actually is - **nixhelm** (nix-community, active — nightly chart bumps, last commit today) is *not* a renderer. It is ~83 upstream chart repos vendored as `charts/<repo>/<chart>` with version+hash pinned in Nix, so `nix flake update nixhelm` moves charts as reviewable commits instead of at deploy time. Public Cachix (`nixhelm.cachix.org`) serves prebuilt chart derivations. It accepts PRs adding charts via `helmupdater`. - **nix-kube-generators** (same author, deliberately tiny) provides `buildHelmChart` (runs `helm template` in the Nix sandbox → a manifests derivation) and `fromHelm` (same, parsed to a Nix attrset). It renders; it never installs. Note the maintenance caveat: ~27 commits total, last activity Nov 2025 — stable but not lively, so we should pin it by rev. ### Coverage vs our helmfile.yaml Already vendored in nixhelm: `authentik`, `traefik` (+ `traefik-crds`), `prometheus-community`, `argoproj` (argo-cd/workflows — the ArgoCD TODO item), `nextcloud`, `forgejo-helm`, `kubernetes-sigs` (node-feature-discovery), `coredns`, `nvidia`, plus `longhorn`/`metallb`/`external-secrets`/`oauth2-proxy`/`kyverno` if we ever want them. Not vendored (would need upstream PRs or the fallback below): `woodpecker-ci`, `wg-easy` (slybase OCI), `zigbee2mqtt`, `homeassistant`, `node-red` (schwarzit), `dependency-track`, `helmforge`, `btrfs-nfs-csi` (ours — it's a git-repo chart anyway), `adnoctem/fmjstudios`, `bokysan smtp`, `cowboysysop whoami`, CrunchyData `pgo`/`postgres`, `local-path-provisioner`. Important for the *bootstrap* question specifically: the cluster plumbing (CNI, DNS, scheduler, RBAC bootstrap) is already done by nixpkgs `services.kubernetes` (flannel, `addons.dns`, `addonManager.bootstrapAddons`) on `lurch`. So "bootstrap helm charts" for us is really: **traefik (+CRDs) → authentik → storage → monitoring** — and the first two are exactly what nixhelm covers. ### Proposed implementation (phased) **Phase 0 — inputs on the nixos flake** (the master, `lurch`, is the apply point): ```nix inputs.nixhelm.url = "github:nix-community/nixhelm"; inputs.nix-kube-generators.url = "github:farcaller/nix-kube-generators"; # both: inputs.nixpkgs.follows = "nixpkgs"; (kube-generators takes pkgs explicitly) ``` **Phase 1 — a `bootstrap/` library in the nixos repo.** One Nix file per bootstrap chart that takes chart + values (values moved out of helmfile.yaml into `bootstrap/values/*.nix` or `.yaml`) and produces a rendered-manifests derivation: ```nix kubelib = inputs.nix-kube-generators.lib { inherit pkgs; }; traefikCharts = inputs.nixhelm.charts { inherit pkgs; }; traefik-manifests = kubelib.buildHelmChart { name = "traefik"; chart = traefikCharts.traefik.traefik; namespace = "kube-system"; valuesFile = ./values/traefik.yaml; kubeVersion = "v1.33.0"; # pin to whatever nixpkgs 26.05 kubernetes module ships }; ``` Expose as flake `packages.<system>` outputs so `nix build .#traefik-manifests` is a pure, reproducible render, and review happens as a git diff of rendered output. **Phase 2 — apply mechanism, deliberately thin.** A `nix run .#apply-bootstrap` script (or a `systemd.onActivation` service on `lurch` gated by a flag) doing `kubectl apply -f ${result}` with the master's kubeconfig. Charts not in nixhelm render via `kubelib.downloadHelmChart { repo = ...; chart = ...; version = ...; chartHash = ...; }` — one hash-bootstrap build per chart, then pinned forever. That keeps helmfile as the escape hatch for the long tail instead of forcing ~12 upstream PRs up front. **Phase 3 — GitOps convergence.** The helmfile TODO already says "Install ArgoCD & continuous deployment": rendered manifests from Phase 1 are exactly ArgoCD's preferred input. ArgoCD itself rendered from `nixhelm.argoproj.argo-cd`, self-hosting its own app definitions. At that point helmfile retires except where the long tail still lives. ### Risks / things I'd verify in the pilot 1. `buildHelmChart` renders without cluster validation: it can't catch bad-vs-cluster API drift, so pinning `kubeVersion`/`apiVersions` per chart matters (our apiserver version comes from nixpkgs, not from the chart repo). 2. Charts with OCI subchart dependencies render fine, but any chart needing `helm repo add` side effects (we have two `git+https://` repos) won't — those stay on downloadHelmChart/helmfile. 3. Secrets: our helm-secrets/sops flow doesn't disappear. Keep secret material out of rendered manifests — sops-nix templates on the target (we already use sops-nix on `lurch`) or external-secrets later; rendering *with* decrypted values baked into a derivation would leak into the Nix store. 4. Determinism caveat: `helm template` output ordering is stable but not guaranteed across helm upgrades; diffs will occasionally be noisy. 5. `fromHelm`'s YAML parsing had an octal-permissions bug upstream (fixed via their fromYAML workaround) — pin by rev, not branch. ### Suggested first concrete step A pilot PR rendering **traefik + traefik-crds + authentik** three ways — helmfile template vs `nix build` of the nixhelm chart vs nix-kube-generators render — and a `helm diff`-equivalent comparison against what's live on the cluster. If the diff is clean, Phase 2's apply wrapper is ~30 lines of Nix and we've got the bootstrap pattern. Happy to build that pilot on request. *— hermes-bot (evaluation: nixhelm tree/API + nix-kube-generators README/lib checked 2026-10-09; coverage cross-checked against all repos in helmfile.yaml)*
rpickler commented 2026-10-09 17:41:01 +00:00
Author
Owner
Copy link

@hermes-bot

I would prefer it if the charts we embed in this way are directly injected into kubernetes as part of a nixos-rebuild switch. This could happen as a systemd service, which would simply mean we render the charts to a derivation, and have the service call helm apply from that path.

I also want what is controlled by this work to be as minimal as possible to stand up the cluster. The btrfs storage csi seems the most obvious as it's tied to the same version that's installed horse. It's also needed for anything else to store their data. Next would probably be metallb, traefik and argocd. The rest of the infrastructure should be managed by argocd, including forgejo.

@hermes-bot I would prefer it if the charts we embed in this way are directly injected into kubernetes as part of a nixos-rebuild switch. This could happen as a systemd service, which would simply mean we render the charts to a derivation, and have the service call helm apply from that path. I also want what is controlled by this work to be as minimal as possible to stand up the cluster. The btrfs storage csi seems the most obvious as it's tied to the same version that's installed horse. It's also needed for anything else to store their data. Next would probably be metallb, traefik and argocd. The rest of the infrastructure should be managed by argocd, including forgejo.
hermes-bot commented 2026-10-14 10:00:20 +00:00
Member
Copy link

Agreed on all three points — implemented as PR #23 (issue-5-bootstrap-helm), scoped to exactly the minimum you named.

Render-to-derivation + systemd apply, as requested. New modules/kubernetes/bootstrap/ module: the chart is rendered by helm template inside a derivation (pure — builds with no cluster), landed at /etc/kubernetes/bootstrap-manifests via environment.etc, and a kubernetes-bootstrap oneshot unit (wantedBy kubernetes.target) applies it during nixos-rebuild switch. The unit is a reconciler — it waits for the API server and retries (Restart=on-failure), so ordering against apiserver readiness on a fresh cluster is handled — and applies with --server-side --prune, so removed resources go away on the next switch.

Minimal = the btrfs storage CSI, and only that. The chart source is inputs.btrfs-nfs-csi + /charts/btrfs-nfs-csi — the same flake pin that builds the agent binary services.btrfs-nfs-csi ships on horse, so the "tied to the same version installed horse" property is structural: chart, appVersion, and agent move together on one nix flake update btrfs-nfs-csi. StorageClass names/values keep parity with the current helmfile release (btrfs-nfs-nosync, btrfs-nfs-backup, kubeletDir=/var/lib/kubernetes). metallb/traefik/argocd are deliberately not in this slice — per your line, they're the ArgoCD bootstrapper's job, which slots into this same module as a follow-up.

Details worth noting for review:

  • No new flake input. For one bootstrap chart I skipped nix-kube-generators; the render is the same mechanism (helm template in a derivation) in ~25 lines. Worth reconsidering only when we render many charts outside ArgoCD.
  • Agent token never enters the Nix store: manifests point at a pre-existing Secret (chart existingSecret); the unit creates it once from a sops-nix secret file at runtime. I verified in the driver source (integrations/kubernetes/controller/config.go) that the controller reads exactly key agentToken from that Secret.
  • snapshotClass defaults off — the chart's VolumeSnapshotClass needs the snapshot CRDs + controller, which stock services.kubernetes doesn't ship; enabling it would fail the apply. Snapshots belong to the ArgoCD layer.
  • hosts/lurch.nix carries the config block with enable = false: flip-on is (1) encrypt a btrfs-csi-agent-token sops key, (2) set enable = true. Once enabled on the live master the switch is an idempotent re-apply.

One honest caveat: this environment has no Nix runner, so the config isn't nix eval-verified yet — reviewed against the nixpkgs services.kubernetes module sources and the chart at the pinned rev. CI/woodpecker on the PR (or a nix eval .#nixosConfigurations.lurch) will close that out before merge.

Agreed on all three points — implemented as **PR #23** (`issue-5-bootstrap-helm`), scoped to exactly the minimum you named. **Render-to-derivation + systemd apply, as requested.** New `modules/kubernetes/bootstrap/` module: the chart is rendered by `helm template` inside a derivation (pure — builds with no cluster), landed at `/etc/kubernetes/bootstrap-manifests` via `environment.etc`, and a `kubernetes-bootstrap` oneshot unit (wantedBy `kubernetes.target`) applies it during `nixos-rebuild switch`. The unit is a reconciler — it waits for the API server and retries (`Restart=on-failure`), so ordering against apiserver readiness on a fresh cluster is handled — and applies with `--server-side --prune`, so removed resources go away on the next switch. **Minimal = the btrfs storage CSI, and only that.** The chart source is `inputs.btrfs-nfs-csi + /charts/btrfs-nfs-csi` — the *same flake pin* that builds the agent binary `services.btrfs-nfs-csi` ships on horse, so the "tied to the same version installed horse" property is structural: chart, appVersion, and agent move together on one `nix flake update btrfs-nfs-csi`. StorageClass names/values keep parity with the current helmfile release (`btrfs-nfs-nosync`, `btrfs-nfs-backup`, `kubeletDir=/var/lib/kubernetes`). metallb/traefik/argocd are deliberately **not** in this slice — per your line, they're the ArgoCD bootstrapper's job, which slots into this same module as a follow-up. Details worth noting for review: - **No new flake input.** For one bootstrap chart I skipped nix-kube-generators; the render is the same mechanism (`helm template` in a derivation) in ~25 lines. Worth reconsidering only when we render many charts outside ArgoCD. - **Agent token never enters the Nix store**: manifests point at a pre-existing Secret (chart `existingSecret`); the unit creates it once from a sops-nix secret file at runtime. I verified in the driver source (`integrations/kubernetes/controller/config.go`) that the controller reads exactly key `agentToken` from that Secret. - **`snapshotClass` defaults off** — the chart's VolumeSnapshotClass needs the snapshot CRDs + controller, which stock `services.kubernetes` doesn't ship; enabling it would fail the apply. Snapshots belong to the ArgoCD layer. - `hosts/lurch.nix` carries the config block with `enable = false`: flip-on is (1) encrypt a `btrfs-csi-agent-token` sops key, (2) set `enable = true`. Once enabled on the live master the switch is an idempotent re-apply. One honest caveat: this environment has no Nix runner, so the config isn't `nix eval`-verified yet — reviewed against the nixpkgs `services.kubernetes` module sources and the chart at the pinned rev. CI/woodpecker on the PR (or a `nix eval .#nixosConfigurations.lurch`) will close that out before merge.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-45-k8s-backup
issue-5-bootstrap-helm
postgres-chicken
issue-35-kube-client-user-module
external-home-rpickler-flake
issue-35-kubernetes-client-user-module
issue-30-tang-clevis
issue-27-image-inference
issue-24-codecompanion-session-id
fix/llama-swap-firewall
feature/llama-swap-replacement
26.05
embed_secrets
backup
woodpecker-pre-build
btrfs-csi
containerd-patch
import-home-manager
wireguard
pi4
games
25.11
clamav
kubernetes-certs.2
kubernetes-certs
routing_demo
rpi5-boot-default
rpi5-kube-master
sees-ai-laptop
unstable
25.05
_custom_cache
_laptop_battery
9-geezer
_sausage
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
richardpickler.com/nixos#5
No description provided.