backup-client: restic backup of k8s state to pi4 (#45) #50
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-45-k8s-backup"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements Option A from the thread: OS-level restic on the storage hosts, zero cluster diff.
What lands
modules/backup-client.nix(new): one nightlyservices.restic.backupsprofile per host, creds via sops-nix (environmentFile, never inlined into unit files). Modes:btrfsSnapshot = true(horse/moose): pre-start stages read-only btrfs snapshots of every PV subvolume under/volume/k8sinto/volume/.backup-snapshots/<name>/, restic uploads that tree, post-stop deletes it. A missing source (e.g. unmounted/volume) or a non-btrfs tree fails the run loudly — verified behaviorally, never backs up an empty tree silently.forget --prunewhenpruneOptsis set);runCheckwith--read-data-subset=2%re-verifies every run.etcd-snapshot.timer(6h, Persistent) runsetcdctl snapshot save; 12 local files kept. The kubernetes module's etcd listens TLS-only on127.0.0.1:2379withclientCertAuth = true(verified by evaluating lurch's config), so the job authenticates with the same client certs kube-apiserver uses (/var/lib/kubernetes/secrets/kube-apiserver-etcd-client*.pem, certmgr-managed).backup.serverUsersextends the htpasswd with per-host users from bcrypt-hash secrets — one leaked host credential can't touch other repos. Server itself otherwise untouched; no new ports (8000 already open).flake.nixfor lurch/horse/moose; serverUsers set for pi4.Secrets to add before deploy (sops
modules/secrets.yaml)Values minted by you; repo password must be shared per-repo between client and any restore site:
restic-repo-k8s-horsek8s-horserestic-htpasswd-horserestic-repo-k8s-mooserestic-htpasswd-mooserestic-repo-k8s-etcdrestic-htpasswd-lurchrestic-htpasswd-hash-horsehtpasswd -bnB horseoutput (drop its trailing:*:padding line)restic-htpasswd-hash-mooserestic-htpasswd-hash-lurchHosts will refuse to activate until their secrets exist (sops-nix default) — deploy pi4 first (it only gains users), then the clients.
Known caveat: moose
/volumehosts/moose.nixstill has/volumecommented out (placeholder UUID) — the nightly job will fail loudly with "backup source missing" until you land that mount. Intentional; the alternative is a green checkmark over an empty tree.Verification (all executed against this branch)
nix evalofnixosConfigurations.{lurch,horse,moose,pi4}: units render —restic-backups-k8s(.timer)on horse/moose,restic-backups-k8s-etcd+etcd-snapshot(.timer)on lurch,EnvironmentFile→ sops-rendered env templates on all three.backup+horse/moose/lurchlines with SOPS placeholders.listenClientUrls=[https://127.0.0.1:2379],clientCertAuth=true, apiserver client cert =kube-apiserver-etcd-client.pem→ snapshot script renders matching--cacert/--cert/--keyflags.bash -nclean; staged-sweep logic behavior-tested with a stubbedbtrfs(fails loudly on missing source and on non-btrfs tree).pkgs.etcd(3.6.14) resolved for lurch includesetcdctl.Not done here (deliberately): the restore drill +
restic checkrun on live data, and moose's/volumemount — both need the hardware/credentials. Suggest running the drill before the nixos-anywhere migration, as discussed.Closes #45
There appears to be an error in your service definition
Root cause: not the unit definitions — the sops secrets this PR references were never added to
modules/secrets.yaml.The branch references nine vault keys that do not exist in the file:
restic-repo-k8s-horse,restic-htpasswd-horserestic-repo-k8s-moose,restic-htpasswd-mooserestic-repo-k8s-etcd,restic-htpasswd-lurchbackup.serverUsers)restic-htpasswd-hash-horse,restic-htpasswd-hash-moose,restic-htpasswd-hash-lurchsops-install-secretsis all-or-nothing: the first unknown key aborts the whole manifest, so nothing gets (re)installed and/run/secrets/rendered/backup-client-k8s-envnever appears. systemd then can't load the restic unit'sEnvironmentFile→Failed to load environment files/result 'resources'. On lurch it's thek8s-etcdunit, on horse thek8sunit — same cause both hosts. Confirm on either:⚠️ Hosts already running keep their pre-existing
/run/secretsfiles (a failed run skips install, it doesn't delete), but a horse/moose/lurch reboot before this is fixed boots with an empty/run/secrets— every sops-fed service on those hosts breaks, not just restic.Fix
sops edit modules/secrets.yamland add the nine keys (I can't do this step — no recipient key for the vault from my side). Values are exact strings with no trailing newline, and the REST password and its bcrypt hash must be a pair (client sends the password; pi4's htpasswd checks the hash):Rollout
nixos-rebuild switch(renders the three new users into the htpasswd; pi4's live system is still on main, which is why it is healthy today).nixos-rebuild switch, then verify:Expected after that: repo init on first run, then snapshot→upload→prune→check. moose will still fail nightly with
backup source missing: /volume/k8suntil its/volumemount is uncommented — intentional, it refuses to back up nothing.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.