Give host services kubernetes access #40

New issue

Open

opened 2026-10-23 21:31:53 +00:00 by rpickler · 1 comment

rpickler commented

2026-10-23 21:31:53 +00:00

Owner

@hermes-bot I need a way for my systems to interact directly with kubernetes. Lurch has the credentials stored on it, and could do so itself, but I need other hosts to make kubectl calls as well. What would be the best way to do this? As a user, I'm using oidc through authentik for my access. Could this be done through authentik? Should it? This flows from #4, where chicken needs a way to inject secrets.

hermes-bot commented

2026-10-24 13:28:47 +00:00

Member

Short answer

Yes, this can be done through Authentik — and yes, it should be. The cluster already validates Authentik JWTs; what's missing is per-service machine identities. The one thing not to do is copy Lurch's admin credentials out to other hosts: the kubeconfig template sitting commented-out in modules/kubernetes/client.nix issues a cluster-admin (O=system:masters) cert — distributing that shape makes every host full admin, gives you one identity in the audit log, and revocation means CA rotation. Three tiers:

A. Humans (your kubectl): keep exactly what's wired. modules/kubernetes/master.nix already runs the apiserver against Authentik (--oidc-issuer-url=https://auth.richardpickler.com/application/o/kubeapiserver/, --oidc-client-id=kube-apiserver, username claim email, groups claim groups), and the bootstrapAddons binding maps Authentik group kubernetes → cluster-admin. This is working as designed — don't extend it to machines.

B. Host services: same issuer, machine identities via OAuth2 client credentials.

Authentik side: on the existing kubeapiserver OAuth2 app, enable the client_credentials grant with a confidential client per host/service (svc-chicken, …). Since the apiserver reads email and groups claims, bind an expression scope mapping that emits them for machine tokens, e.g. {"email": "svc-chicken@services.richardpickler.com", "groups": ["kubernetes-host-services"]}.
RBAC side: bind Group: oidcgroup:kubernetes-host-services to a deliberate ClusterRole. ⚠️ Do not add machine accounts to the Authentik kubernetes group — the bootstrapAddons binding would silently hand them cluster-admin.
Host side: kubelogin-oidc is already installed on client hosts (modules/kubernetes/client.nix); point a kubeconfig users[].exec plugin at it with --grant-type client-credentials (exact flags depend on the kubelogin generation; a 5-line curl-against-the-token-endpoint exec plugin works identically if the binary is the older int128 flavor). Client secret comes from sops-nix, one per host.
Why this is the right answer for you: one IdP you already run; per-host identity in audit logs; revocation = disable the client in Authentik; no long-lived certs proliferating.

C. The concrete #4 need (chicken injecting postgres credentials): a scoped ServiceAccount is better than general host access. pickler-postgres-k8s-sync only needs to create/update secrets in its target namespace(s) — it shouldn't hold anything broader:

kubectl -n "$NS" create sa postgres-sync
kubectl -n "$NS" create role postgres-sync --verb=get,create,update,patch --resource=secrets
kubectl -n "$NS" create rolebinding postgres-sync --role=postgres-sync --serviceaccount="$NS":postgres-sync
cat <<'EOF' | kubectl -n "$NS" apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: postgres-sync-token
  annotations:
    kubernetes.io/service-account.name: postgres-sync
type: kubernetes.io/service-account-token
EOF

Feed the resulting token into a single-context kubeconfig against https://lurch.lan:6443 and drop it into the sops secret PR #6 already gates on (pickler.postgres.kubernetes.kubeconfigSecret) — the sync service activates with zero broader privileges, and revocation is deleting one Secret.

What about client certs? They're the cluster-native machine identity and the repo is half-set-up for them (cfssl + certmgr in client.nix, services.certmgr renewing daily on lurch). But the commented template signs admin certs by hand, and K8s has effectively no cert revocation short of CA rotation. If you want this path anyway, do it as CSR auto-signing — each host generates its key locally, submits a CertificateSigningRequest with bootstrap creds, certmgr signs against a SigningProfile (per-host CN=host:<name>, O=kubernetes-host-services, short validity) — private keys never leave their host. More machinery than B; pick it only if you want the machine path to survive an Authentik outage.

Should it be Authentik? Yes, with two forward-looking notes:

For the actual flow from #4, consider reversing the direction in the cluster rebuild: external-secrets-operator pulling credentials from an external store (Vault/Vaultwarden/git+SOPS) means chicken needs no cluster access at all, which beats scoping host access down.
The NixOS-native services.kubernetes module this cluster runs on is legacy upstream, so the rebuild is coming regardless — the OIDC flags, RBAC bindings, and both B and C carry over unchanged to k3s/kubeadm, so adopt them now.

Suggested next step: pick the RBAC tier for host services (B), and I can PR the reusable NixOS module — sops secret for the client secret + exec-plugin kubeconfig template — plus the Authentik app/scope-mapping walk-through. C is ready to wire into PR #6 today.

## Short answer Yes, this can be done through Authentik — and yes, it should be. The cluster already validates Authentik JWTs; what's missing is per-service *machine* identities. The one thing not to do is copy Lurch's admin credentials out to other hosts: the kubeconfig template sitting commented-out in `modules/kubernetes/client.nix` issues a `cluster-admin` (O=system:masters) cert — distributing that shape makes every host full admin, gives you one identity in the audit log, and revocation means CA rotation. Three tiers: **A. Humans (your kubectl): keep exactly what's wired.** `modules/kubernetes/master.nix` already runs the apiserver against Authentik (`--oidc-issuer-url=https://auth.richardpickler.com/application/o/kubeapiserver/`, `--oidc-client-id=kube-apiserver`, username claim `email`, groups claim `groups`), and the `bootstrapAddons` binding maps Authentik group `kubernetes` → cluster-admin. This is working as designed — don't extend it to machines. **B. Host services: same issuer, machine identities via OAuth2 client credentials.** - **Authentik side:** on the existing `kubeapiserver` OAuth2 app, enable the `client_credentials` grant with a confidential client **per host/service** (`svc-chicken`, …). Since the apiserver reads `email` and `groups` claims, bind an expression scope mapping that emits them for machine tokens, e.g. `{"email": "svc-chicken@services.richardpickler.com", "groups": ["kubernetes-host-services"]}`. - **RBAC side:** bind `Group: oidcgroup:kubernetes-host-services` to a deliberate ClusterRole. ⚠️ Do **not** add machine accounts to the Authentik `kubernetes` group — the bootstrapAddons binding would silently hand them cluster-admin. - **Host side:** `kubelogin-oidc` is already installed on client hosts (`modules/kubernetes/client.nix`); point a kubeconfig `users[].exec` plugin at it with `--grant-type client-credentials` (exact flags depend on the kubelogin generation; a 5-line curl-against-the-token-endpoint exec plugin works identically if the binary is the older int128 flavor). Client secret comes from sops-nix, one per host. - **Why this is the right answer for you:** one IdP you already run; per-host identity in audit logs; revocation = disable the client in Authentik; no long-lived certs proliferating. **C. The concrete #4 need (chicken injecting postgres credentials): a scoped ServiceAccount is better than general host access.** `pickler-postgres-k8s-sync` only needs to create/update `secrets` in its target namespace(s) — it shouldn't hold anything broader: ```bash kubectl -n "$NS" create sa postgres-sync kubectl -n "$NS" create role postgres-sync --verb=get,create,update,patch --resource=secrets kubectl -n "$NS" create rolebinding postgres-sync --role=postgres-sync --serviceaccount="$NS":postgres-sync cat <<'EOF' | kubectl -n "$NS" apply -f - apiVersion: v1 kind: Secret metadata: name: postgres-sync-token annotations: kubernetes.io/service-account.name: postgres-sync type: kubernetes.io/service-account-token EOF ``` Feed the resulting token into a single-context kubeconfig against `https://lurch.lan:6443` and drop it into the sops secret PR #6 already gates on (`pickler.postgres.kubernetes.kubeconfigSecret`) — the sync service activates with zero broader privileges, and revocation is deleting one Secret. **What about client certs?** They're the cluster-native machine identity and the repo is half-set-up for them (cfssl + certmgr in `client.nix`, `services.certmgr` renewing daily on lurch). But the commented template signs admin certs by hand, and K8s has effectively no cert revocation short of CA rotation. If you want this path anyway, do it as CSR auto-signing — each host generates its key locally, submits a `CertificateSigningRequest` with bootstrap creds, certmgr signs against a SigningProfile (per-host `CN=host:<name>`, `O=kubernetes-host-services`, short validity) — private keys never leave their host. More machinery than B; pick it only if you want the machine path to survive an Authentik outage. **Should it be Authentik?** Yes, with two forward-looking notes: 1. For the actual flow from #4, consider **reversing the direction** in the cluster rebuild: external-secrets-operator pulling credentials *from* an external store (Vault/Vaultwarden/git+SOPS) means chicken needs **no** cluster access at all, which beats scoping host access down. 2. The NixOS-native `services.kubernetes` module this cluster runs on is legacy upstream, so the rebuild is coming regardless — the OIDC flags, RBAC bindings, and both B and C carry over unchanged to k3s/kubeadm, so adopt them now. **Suggested next step:** pick the RBAC tier for host services (B), and I can PR the reusable NixOS module — sops secret for the client secret + exec-plugin kubeconfig template — plus the Authentik app/scope-mapping walk-through. C is ready to wire into PR #6 today.

rpickler referenced this issue

2026-11-01 12:25:09 +00:00

Backup k8s Services #45

No labels

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

richardpickler.com/nixos#40

No description provided.

Rows
Columns