How to Run a Disaster-Recovery Drill (Restore Your Whole Server to a Spare Machine — Before You Have To)

Every backup tutorial in this series ends with "restore one file to prove it". That proves the archive opens. It doesn't prove you can get the server back — every service, every config, every password, in an order that works, in an afternoon rather than a week. A disaster-recovery drill does: once a year, on a spare machine or a VM, you pretend the server is gone and rebuild it from backups only. What you learn fixes the backups; what you write down becomes the runbook.

• Only the backups. No copying from the live server. If it isn't in a backup, it's lost — that's the point.
• Secrets must come from somewhere other than the server: the password manager (43), the printed sheet, the USB in a drawer. If the restic password only lives on the server, you've already found the first gap.
• Don't touch the live server's IP or DNS. The drill box gets a different IP; test by pointing a laptop's hosts file at it.
• Time every phase. The number is the answer to "how long would we be down?"

Install Ubuntu on the spare machine (tutorial 1), or spin up a VM. User, key, updates, hostname drill. If you have the Ansible playbook (140): ansible-playbook site.yml --limit drill and most of phases 1, 2 and 4 happen by themselves — the drill is also the test of the playbook.

Without it: tutorials 3–9 by hand, and note how long it took. That's a reason to write the playbook.

From the password manager or the sheet in the drawer:

• restic repository password(s) (55, 116) and rclone crypt passwords (54)
• the .env files for each stack (59) — or the Ansible vault password
• Cloudflare/DuckDNS tokens (48, 147), ntfy/Telegram tokens (93, 118)
• Vaultwarden's own backup: you can't look up the restic password in Vaultwarden if Vaultwarden is in the backup. Keep that one on paper.

Missing any of these ends the drill. Write it on the scorecard; it's the most valuable finding there is.

Mount the backup drive, or configure rclone to the cloud copy, or point restic at the REST server (116). Then:

source /root/.restic.env
restic snapshots
restic restore latest --target /tmp/restore --include /etc --include /home/john/docker

From /tmp/restore:

• /etc — only the files you edited (nginx sites, samba, fail2ban, ssh, msmtp, fstab lines for data drives, netplan). Not the whole tree.
• ~/docker into place: compose files, .env (or from secrets), each stack's ./data.
• Big data (media, photos, Nextcloud files): to whatever drive the drill box has, or a subset — for the drill, a few GB proves the path works; note the full size and the restore speed to extrapolate.
• Database dumps: mariadb < dump.sql into fresh containers (65), not copied files.

Time this phase per gigabyte; multiply by the real size. If "restore everything" is 30 hours from the cloud, that's a finding: a local copy (23) as well as the off-site one.

docker compose up -d in each folder, in dependency order (networks first, then proxy, then apps). Watch docker compose logs for the ones that fail: usually a missing .env, a wrong data path, a database that wants its dump. Nginx: nginx -t. Certificates: a drill box has no public name, so either restore /etc/letsencrypt (it's in the /etc backup) or accept warnings for the test.

From a laptop, with its hosts file pointing the names at the drill box:

• Log in to Nextcloud; open a file; upload one.
• Play something in Jellyfin; a photo in Immich.
• Vaultwarden: unlock the vault with the app pointed at the drill box.
• Home Assistant loads; a Zigbee device would need the stick moved — note it.
• Pi-hole answers DNS; Samba share opens; Paperless shows documents.
• The backup script itself runs on the drill box and writes somewhere.

Tick each. Every one that fails is a gap in the backup, the restore order, or the documentation.

In BookStack (88), a page per drill:

Fix every "Fix" item this week. Then the backup is known to work, and the runbook is the drill's own notes — the thing you'd follow at 2 am.

Once a year, and after any big change (new storage layout, new backup tool, a migration). A half-drill — restore one stack to a VM — every quarter is cheap and catches most drift.

It took all weekend. Normal the first time. The playbook (140), a local restic copy and a written order make the second one an afternoon.

A container starts but the app is empty. Data path mismatch between the compose file and where you restored; docker inspect the mounts (65).

Couldn't test the public services. Hosts-file trick on the laptop, or a temporary DNS record pointing at the drill box's IP for an hour.

Found a gap you can't fix cheaply (the 400 GB). Decide, and write the decision down: "Immich originals also on the backup drive; off-site covers documents only." A known gap is a plan; an unknown one is a disaster.

A spare box, a Saturday, backups only, secrets from outside the server, every service tested from a laptop, a scorecard with times and fixes. It's the one exercise that turns "I have backups" into "I can be back by tonight" — and the fixes it produces are the most valuable hour you'll spend on the server all year.