Restore Drill Playbook: Prove Backups Before You Trust Them

Backups without restores are theater

A green cron job is not evidence. Evidence is: wipe a scratch host, restore from last night's artifact, and open the real app until you find last week's photo, vault entry, or git commit.

What must be in the backup set

For each compose project list: bind mounts / named volumes that hold durable data, database dump method (or volume that contains the DB files), `.env` / secret files, and the compose YAML itself. Thumbnails and caches are optional; originals and DB state are not.

Drill procedure (90 minutes or less)

1. Pick one critical service (photos, vault, or forge).
2. Snapshot or note the live health URL and a unique recent object (album date, password item name, commit hash).
3. Provision a scratch project on another path or host — never restore onto production volumes first.
4. Restore dumps + volumes from the artifact you claim is good.
5. Bring the stack up with pinned tags from STACK.md.
6. Prove the unique object exists and auth still works.
7. Log date, minutes elapsed, artifact ID, and who ran the drill.

# pattern only — adjust paths
mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
# restore volume tarball + .env + compose.yml here
docker compose up -d
# open the app; search for a known object; then tear down
docker compose down

Offsite and S3-compatible storage

Local snapshots die with the rack. Keep an encrypted copy off-site (restic/Borg/rclone to S3-compatible). Encrypt before upload. Store the key where the bucket alone cannot decrypt the archive.

Test download + decrypt on a machine that is not the production host. A bucket listing is not a restore.

Cadence

Quarterly minimum for household stacks; monthly for anything that holds identity or irreplaceable media. After major upgrades, run a mini-drill before you delete the previous artifact.

Calendar the next drill when you finish this one. Untimed "someday" restores never happen.

Failure modes to expect

Wrong volume restored; DB newer than media; `.env` missing; image tag floated; ACL blocked download; restore succeeded but SSO IdP is elsewhere. Write each failure into the drill log so the next run is faster.

[Add real environment data here]
Host / OS:
Backup tool + version:
Offsite target:
Last successful drill date:
Minutes to verify:
Artifact ID:

Related reading

See Docker Compose hardening, Updating without fear, Family handoff documentation, and What is self-hosting.