kubernetes: client config as portable home-manager user module (#35) #37
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-35-kube-client-user-module"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #35.
Adds
users/modules/kubernetes.nix, a home-manager user module that re-implements thekubernetes.clientsystem module so the config is portable to the work system. The module uses only stock home-manager arguments (config, lib, pkgs) and no repo-internal imports, so it can be dropped into any home-manager config standalone:Design notes:
kubernetes.client.*) so host and user configs read the same way.kubeconfigandkubeconfigFileare mutually exclusive (module assertion).kubectl,kubernetes-helm,kubelogin-oidc. The cfssl/certmgr/openssl trio from the system module is left system-level (cluster-admin cert tooling, not client tooling); still overridable viapackages.pkgs.formats.yaml— notepkgs.formats.kubernetesno longer exists in nixpkgs (went away with the removed kubernetes service modules), and HM'sprograms.kubectlcompletion module was likewise removed upstream; shell completion iskubectl completion <shell>/ package-provided now.kubeconfigdefaults to null, so nothing writes~/.kube/configuntil you declare it — no conflict with existing hand-made kubeconfigs.kubernetes.client.enable(system) once the user layer covers them.Verified with the repo's locked nixpkgs (597283a) + home-manager release-26.05 (24f8f8e):
evalModulesharness: kubeconfig attrs render a valid YAML kubeconfig to~/.kube/config; enable/packages/symlink/mutual-exclusion assertion all behave;nix eval .#nixosConfigurations.kansas.config.home-manager.users.rpickler.kubernetes.client.enable→true, user profile carries kubectl, and the rest of rpickler's HM file set evaluates unchanged.Re-implements the kubernetes.client system module at the user layer so it is portable to systems outside this repo (e.g. the work machine): - users/modules/kubernetes.nix declares kubernetes.client.{enable, packages,kubeconfig,kubeconfigFile} using only stock home-manager arguments, so it can be imported standalone from any config - kubeconfig attrs render to ~/.kube/config via pkgs.formats.yaml; kubeconfigFile symlinks an externally-managed file (sops templates, work secret managers) instead; mutually exclusive via assertion - wired into rpickler's portable module list (enabled on every host) The system module stays as-is for headless hosts without home users. Co-Authored-By: Hermes Agent <hermes-bot@richardpickler.invalid>I'm going to need help with a few things on this review.
Let me know what you need from me, and I can slot in the changes you need from me (secret in particular)
All three points implemented in
896dd3d—~/.kube/configis now fully generated on kansas and sausage (no morenull/ fallback to the hand-made file).What the code does now
users/users/rpickler.nixgained akubernetes-home-kubeconfigmodule: declares the home cluster (local,https://lurch.lan:6443,cluster-admin— same shape as the commented template in the system module) askubernetes.client.kubeconfigattrs, with the three PEM fields pointing at user-space sops secretskube/certificate-authority,kube/client-certificate,kube/client-key. Only the secret paths render into~/.kube/config; the key material never touches the Nix store.@sausage/@kansasdeliberately: those are the only hosts whose user age key is a recipient ofmodules/secrets.yaml(rpicklerAtSausage/rpicklerAtKansasinmodules/keys.sops.yaml). Other hosts (pumpkin, vm) keep tooling only — verifiedkubeconfig = nullthere.home.file.".kube/config".force = truereplaces the existing hand-made config on the first switch. Back it up first (cp ~/.kube/config ~/.kube/config.bak); if it holds clusters beyondlocal, tell me and I'll add them to the attrs.Verified with the repo's locked nixpkgs:
nix eval .#nixosConfigurations.kansas.config.home-manager.users.rpickler.kubernetes.client.kubeconfig."current-context"→local, the generated kubeconfig derivation builds to valid YAML with sops paths only, pumpkin still evaluates clean withkubeconfig = null.What I need from you — the three PEMs in
modules/secrets.yaml(the same file the user-sops stub already points at). Do this on kansas, where your user SSH key is a recipient:Add three multi-line strings at top level — grab the values from your current kubeconfig (
kubectl config view --flattenshows them base64-embedded;base64 -ddecodes):Saving re-encrypts for the existing recipient list automatically; the new keys are inert for everything else until declared, which the module above does. Then commit + push
modules/secrets.yamland rebuild as usual — order matters only in that the firstswitchon kansas/sausage will fail until the keys are in the file.The generated file lands at
~/.kube/configwith the cert fields pointing at the sops-nix user secret paths (/run/user/$UID/secrets/kube/...), sokubectlworks unmodified.For the sops secrets, I meant separate home-manager based secrets. I see my home manager currently points to the same modules/secrets.yaml, but my intention is to separate the home-manager user setup from this repository. Therefore, this secret will need to live in a different secrets file. This means the user/modules/sops.nix file will need to change.
Separation done in
a1c48db— home-manager no longer readsmodules/secrets.yaml.What changed
users/modules/sops.nix:defaultSopsFilenow points at a newusers/secrets.yaml, inside theusers/tree, so the whole home layer (modules + secrets file) moves out of this repo as one unit when you split it. The module comment documentssops updatekeys users/secrets.yamlfor re-encrypting to the work recipients at that point.users/secrets.yamlis committed already sops-encrypted (3.13.3, same as the repo file) to the same 11 age recipients asmodules/secrets.yaml— therpicklerAtSausage/rpicklerAtKansasuser keys keep working unchanged. It containskube/certificate-authority,kube/client-certificate,kube/client-keyas placeholders so eval and activation build; nothing decrypts to real material until you fill them.backup.nix'srestic-passphraseis the one entry still living inmodules/secrets.yaml— its plaintext can't be moved without the decrypt key, so it keeps an explicitsopsFile = ../../modules/secrets.yamloverride with aTODO(home-split)describing how to move the value when you extract the home setup. That override is the only remaining coupling betweenusers/and the system secrets.Your one step — put the real PEMs in the new file (kansas, with your user key as before):
Replace the three
PLACEHOLDER-...blocks (values fromkubectl config view --flatten,base64 -dthecertificate-authority-data/client-certificate-data/client-key-datafields). Saving re-encrypts for the existing recipient list automatically. Nothing else to edit —rpickler.nixand the kubernetes user module pick the paths up through the new default.Verified with the repo's locked nixpkgs: kansas
kubernetes.client.kubeconfig."current-context"=local;sops.secrets."kube/client-key".sopsFileresolves tousers/secrets.yaml;restic-passphrase.sopsFilestill resolves tomodules/secrets.yaml; pumpkin stayskubeconfig = null(tooling only). Nokube/*keys were ever added tomodules/secrets.yaml, so there is nothing to migrate out of it.rpickler's kubernetes-home-kubeconfig now renders ~/.kube/config with a sops-nix template: OIDC issuer/client-id/client-secret are substituted at activation from users/secrets.yaml (kube/oidc-* keys), so credential values never land in the Nix store; the cluster CA stays a secret-path reference. kubeconfigFile points at the rendered file (0400) and ~/.kube/config links to it. interactiveMode IfAvailable so a fresh host can do the first browser login (Never would hard-fail it). Also reverts the hardcoded OIDC attrset that had slipped into kubernetes.client.kubeconfig's default on this branch: it referenced undefined identifiers at module scope (config, sops, kubectl), wired a nonexistent plugin binary ('oidc-login'), mismatched user vs context, and would have interpolated decrypted secret VALUES into the Nix store via pkgs.formats.yaml. Default is null again; a corrected, store-safe OIDC exec pattern lives in the option example with the template guidance.Implemented the sops-nix template approach you asked for — commit
1960eb9.How it works (
kubernetes-home-kubeconfiginusers/users/rpickler.nix): the whole kubeconfig is asops.templates."kube-config"whose content is your OIDC exec config with${config.sops.placeholder."kube/oidc-*"}tokens; sops-nix substitutes the real values fromusers/secrets.yamlat activation. The rendered file (~/.config/sops-nix/secrets/rendered/kube-config, mode 0400) has zero secret material in the Nix store, andkubernetes.client.kubeconfigFilepoints at it so~/.kube/configlinks to it. The CA stays a secret-path reference (it's public anyway).Action needed from you: fill the three placeholders via
sops edit users/secrets.yaml—kube/oidc-issuer-url,kube/oidc-client-id,kube/oidc-client-secret(plus the real CA PEM inkube/certificate-authority).Your draft's config, fixed on the way in:
command: kubectl→${pkgs.kubelogin-oidc}/bin/kubectl-oidc_login— the credential plugin is that binary, and it must be an absolute store path (kubectl execs it directly, no PATH)."oidc-login" "get-token"→ just"get-token"(the binary name iskubectl-oidc_login).--oidc-extra-scope=profile=email→ two args--oidc-extra-scope=profileand--oidc-extra-scope=email(the original passed one scope literally named "profile=email").env+$$OIDC_*indirection → secret values go in the exec args directly; kubelogin has no env fallback for--oidc-client-secret(flag set checked on 1.36.4, and exercised on the pinned 1.36.3), and the template substitution already keeps them out of the store.interactiveMode: Never→IfAvailable: on a freshly switched host the kubelogin token cache is empty andNeverhard-fails the first browser login. (IfNeededis not a valid value — client-go rejects it; valid:Never/IfAvailable/Always.)Also reverted: the OIDC attrset that had ended up as the default of
kubernetes.client.kubeconfiginusers/modules/kubernetes.nix. It referencedconfig/sops/kubectlat module scope where none are bound, used the'oidc-login'binary, and interpolated decrypted secret values throughpkgs.formats.yaml— i.e. straight into the Nix store — defeating the placeholder mechanism. Default isnullagain; the corrected store-safe pattern is now the option example, with a note pointing secret-embedding configs at templates +kubeconfigFile.Verified:
kubeconfigFile= rendered template path; template content shows/nix/store/…-kubelogin-1.36.3/bin/kubectl-oidc_login, placeholders in the three secret args, CA =/home/rpickler/.config/sops-nix/secrets/kube/certificate-authority; all fourkube/*secrets resolve againstusers/secrets.yaml.kubeconfig/kubeconfigFilenull, no.kube/configentry (host gate intact).interactiveModevalue bug above.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.