Backup k8s Services #45
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I am very uncertain how to approach this. I have a number of k8s services that need to be backed up to my restic server. The services should be all running on the btrfs nfs csi storage driver now, although I know a few are not. Should this be done as an operator within the k8s deployment? Or should this be done at the OS level in nixos? Should we wait for #5 #40 and this comment: #6 (comment) so that we can reach the restic secrets required within the cluster.
@hermes-bot Give me some options. I'm not entirely certain how to approach this problem. It needs to be done, and be stable before we start migrating.
Where the data actually lives (grounds the whole question)
Before the options, three facts from the repo that shape this:
horseandmoosewithbasePath = /volume/k8s— every PV is a btrfs subvolume under/volume/k8sthere, NFS-exported to consumers. There is no other place k8s state hides.lurch.merged: false) — the age-key-on-controller → external-secrets route is not actually landed on main yet, despite the comment thread implying it. So the "reach the restic secret inside the cluster" machinery is still queued, not done.backup.enable_serveronpi4(REST server :8000, htpasswd from sops,/backupUSB disk). What's missing is a client side — nothing in the repo currently pushes to it.Option A — OS-level restic on the agent hosts (recommended)
Back up at the NixOS level on
horseandmoosewithservices.restic.backupspointed atrest:https://pi4…:8000/k8s-horse/…/k8s-moose.btrfs subvolume snapshot -r /volume/k8s/<vol> /volume/k8s-snapshots/<ts>/<vol>per PV (you already do this rootfs pattern inhosts/horse.nixviamakeBlankSnapshot), restic from the snapshot tree, delete after. btrfs snapshots are atomic → crash-consistent copies of every PVC without touching the cluster.sops.secretson horse/moose. sops-nix is already imported on both. No in-cluster secrets plumbing needed at all — this sidesteps the #6/external-secrets dependency entirely./volume.restic restoreinto a new subvolume or any NFS-mounted consumer, or just a throwaway pod — because it's plain files, restore is driver-agnostic (nice during a migration when the driver itself is in flux).prunePolices(keep-daily/weekly/monthly) + a weeklyrestic check --read-data-subset=2%.modules/backup-client.nix(generalizes for non-k8s hosts too), 2 sops secrets, wiring for horse/moose inflake.nix. Half a day + a restore drill.Option B — In-cluster (operator / CronJob)
BackupStorageLocationspeaks S3-compatible object storage, not the restic REST protocol. Pointing it at your pi4 server means running MinIO alongside/instead of restic server. It also needs restic/S3 creds as in-cluster Secrets — the exact blocker from the PR #6 thread. It would add real value later (per-PVC selectors, restore UX, hooks), but it drags in S3 + external-secrets + #5 chart delivery. Not the "stable before migrating" choice.Option C — driver-native push (watch, don't wait)
The agent has a REST API (:8080 in your config) and the roadmap lists btrfs send/receive + scheduled replication to another agent — the architecturally cleanest future (snapshot → send stream to pi4 → done, zero file-tree scanning, incremental at btrfs level). Also note the project is being superseded by ButterStore (drop-in fork, original repo to be archived). Neither send/receive nor the fork has shipped, so don't block on it — but it means Option A's snapshot-sweep design can likely be upgraded to send/receive later without changing the backup repository on pi4.
Don't forget etcd
The last state not covered by git: etcd on
lurch. A tiny systemd timer there:etcdctl --cacert … --cert … snapshot save /backup/etcd-<ts>.db, and let a restic backup (or the same client module,paths = ["/backup/etcd-snapshots"]) ship it to pi4. ~30 lines, closes the last gap."Should we wait for #5/#40/PR #6?"
No. Waiting only matters if you choose Option B (in-cluster creds). Option A needs nothing that isn't already on main: sops works on horse/moose today, the REST server is up on pi4, and the snapshot pattern already exists in your host configs. Given "needs to be stable before we start migrating," that ordering matters: the nixos-anywhere migration will repartition/reimage horse and moose — the backup is the safety net for the migration itself, so it has to exist and have passed a restore drill before the first host is reimaged, not after.
Recommended sequence: Option A on horse+moose now + etcd snapshot timer on lurch → run
restic checkand one full restore-to-scratch drill → then migrate. Revisit Velero/B1 afterwards only if you want in-cluster restore UX; watch ButterStore for the send/receive upgrade.If Option A sounds right I'll push a PR with
modules/backup-client.nix(restic client w/ snapshot pre/post hooks, prune, check timer) + horse/moose wiring + the etcd job — leaving the two sops secrets (repo password, REST creds) for you to add, since I can mint placeholders but only you should own those values.Give me a better overview of what option a would change in the nixos deployment.
Option A, concretely: the NixOS-side diff
What would actually change in this repo and on each host. Verified against
maintoday.Files in the repo
New:
modules/backup-client.nix(~80 lines, generalizes for any host, not k8s-specific)backup-client.enable,backup-client.repository(repo path on pi4),backup-client.paths(what to sweep),backup-client.btrfsSnapshot(bool: take read-only snapshots under/volume/.backup-snapshots/<ts>/pre-backup, delete post-backup — samebtrfs subvolume snapshot -ridiom yourmakeBlankSnapshotinhosts/horse.nixalready uses).services.restic.backups.<name>:preBackup/postBackuphooks for the snapshot sweep,timerConfig.OnCalendar(proposed:*-*-* 03:00+RandomizedDelaySec=3600),prunewith a keep-policy (prunePolices: daily×14, weekly×8, monthly×6 — knobs, argue freely).restic check --read-data-subset=2%systemd timer.environmentFilepointing at a root-only sops-rendered file — not the pattern inusers/modules/backup.nix, which embeds the REST password in the repo URL (that lands in a world-readable systemd unit file and is already committed in plaintext there; worth a rotation once we're in — I didn't quote it, but it's grep-visible).Modified:
flake.nix— 2 module-list entries (./modules/backup-client.nixonhorseandmoose), plus a small inline config block per host: repo name (k8s-horse/k8s-moose), path (/volume/k8s),btrfsSnapshot = true. Optionally lurch gets the same module pointed at/backup-etcdfor the etcd timer piece.That's the whole repo diff: 1 new file, 1 touched.
Secrets (sops)
modules/secrets.yamlgains per-host entries: restic repo encryption key + REST auth for horse and moose.modules/sops.nixuses host SSH ed25519 keys as age keys, so the re-encryption just needs horse/moose pubkeys added (sops updatekeysafter first deploy).modules/backup.nixrenders a singlebackuphtpasswd user. Either reuse it, or extend to one htpasswd entry per host so a leaked host credential can't rewrite other repos — ~5 lines server-side. I'd do per-host entries.What appears on each host after
nixos-rebuild switchrestic-backups-k8s.service+.timer,restic-check-k8s.timer/volume/.backup-snapshots/, deletes themetcd-snapshot.timer(optional piece):etcdctl snapshot save→ restic ships it to repok8s-etcd/backupExplicitly unchanged
modules/kubernetes/btrfs-csi-driver.nixuntouched —basePath = /volume/k8sstays, the backup just sweeps under it.pi4:8000.One landmine found while checking
hosts/moose.nixline 126:fileSystems."/volume"is commented out (horse mounts it by-UUID, moose doesn't — yet btrfs-csi on moose happily uses/volume/k8s, so it's mounted by hand or fstab outside the config). The module gates on path existence so it won't fire blind, but we should nail moose's/volumemount in the config as part of this PR — otherwise moose silently "backs up" an empty tree, which is the worst kind of green checkmark.Nothing here blocks on #5/#40/#6. Say the word and this ships as one PR:
modules/backup-client.nix+ flake wiring + etcd timer, with sops secret placeholders (names + structure documented), values minted by you.Go ahead. Implement and push a pr
Implemented and pushed as PR #50 (
issue-45-k8s-backup, againstmain) — Option A: newbackup-clientmodule (restic nightly to pi4; btrfs-snapshot mode for horse/moose, etcd-snapshot timer for lurch with the module's TLS client certs), pi4serverUsersextension, secrets checklist for you in the PR body. Verified by evaluating all four hosts' configs plus behavioral tests of the snapshot scripts (details in the PR).