What to Do If Your Server Is Hacked (A Calm, Ordered Incident Response for a Home Server)

The CPU is at 100% on a process called kdevtmpfsi. There's a user you didn't create. Uptime Kuma says the site is serving someone else's page. last shows a login from a country you've never visited. It happens — usually through an exposed app with a default password, an old image with a known hole, or a forwarded port you forgot. The order of what you do next matters more than how fast you do it.

Pull the Ethernet cable, or on the router remove the port forwards and block the server's MAC; stop WireGuard/Tailscale (sudo systemctl stop wg-quick@wg0 tailscaled). The attacker loses access; the server keeps running so you can look. Don't reboot or power off yet — running processes, open connections and anything living only in memory disappear.

If it's a VM, snapshot it now (81, 141): a frozen copy to examine later.

At the console or a LAN-only SSH session, into a folder on a USB stick (not the compromised disk):

mkdir -p /mnt/usb/ir && cd /mnt/usb/ir
date > time.txt; uptime >> time.txt
ps auxwwf > ps.txt
ss -tunap > connections.txt
sudo lsof -nP > lsof.txt
who > who.txt; last -a -n 100 > last.txt; sudo lastb -a -n 100 > lastb.txt
cat /etc/passwd > passwd.txt; sudo cat /etc/shadow > shadow.txt
sudo cat /etc/sudoers /etc/sudoers.d/* > sudoers.txt
for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -l -u $u 2>/dev/null | sed "s/^/$u: /"; done > crontabs.txt
ls -la /etc/cron* /etc/systemd/system /home/*/.ssh > cron-systemd-ssh.txt
sudo find / -xdev -newermt "$(date -d '7 days ago' +%F)" -type f ! -path '/proc/*' ! -path '/var/lib/docker/*' > newfiles.txt 2>/dev/null
docker ps -a > docker-ps.txt; docker images > docker-images.txt
sudo journalctl --since "14 days ago" > journal.txt
sudo cp -r /var/log /mnt/usb/ir/var-log

Now you have what was running, who was connected to where, every user and sudoer, every scheduled job, and every file changed this week. That's enough to work from, and enough to hand to someone more experienced.

Work backwards from the evidence:

• The process: ps auxwwf shows the tree — a miner's parent is often a container (docker inspect the container → which app) or a cron entry. ls -l /proc/PID/exe and cwd say where it lives.
• Connections: connections.txt — an outbound connection to an odd IP on an odd port from a container names the container.
• Logins: last/lastb and journal.txt | grep -E "Accepted|session opened" — a key you don't recognise (71)? Then the key was planted after the first entry; keep looking.
• New files: newfiles.txt — a new script in /tmp, /dev/shm, /var/tmp, a changed authorized_keys, a new systemd unit, a new user's home.
• The exposed app: nmap from outside (171), the proxy's access log around the first odd timestamp, Loki (155) for the first "error" from that app. Default passwords (Grafana admin/admin, a Portainer left open, Mosquitto with anonymous on, a Jupyter without a token) and old images with known CVEs are the usual answer.

You want one sentence: "They got in through X on date Y." If you can't find it after an hour, assume everything the box could reach is compromised and go to step 4 anyway.

A "cleaned" server is one you hope is clean. A rootkit, a second backdoor, a modified binary: you won't find them all. The series has made rebuilding cheap:

1. Note what you need from the capture (step 2). Then power off and keep the disk (or the VM snapshot) for later study.
2. Fresh install on a wiped disk, or a new VM (1, 141).
3. The Ansible playbook (140) brings the baseline and hardening back in minutes — after you've fixed the hole in it.
4. Restore data from a backup taken before the compromise (restic snapshots have dates, 55; cold drive, 169) — never configs or scripts from after it. Review restored ~/docker compose files and .env by eye.
5. Pull fresh images (docker compose pull), never restored ones.

The DR drill (159) is exactly this; if you've done it, today is a long afternoon rather than a week.

Assume all of it was read: SSH keys (make new pairs on each client; remove old public keys everywhere — Gitea, VPS, the relative's box), the restic and rclone passwords, every .env secret (database passwords, API tokens, ntfy/Telegram tokens), Cloudflare/DuckDNS tokens, the WireGuard keys (new configs for every phone), Vaultwarden's master password and every password in it if the vault file could have been taken (43 — this is the painful one), app admin passwords, the Let's Encrypt account (revoke and reissue certificates), the Samba passwords. Enable 2FA where you hadn't (70, 95). Check Cloudflare, Gitea and any cloud accounts for new API keys or users.

Fix the specific hole first: the app behind Authelia or the VPN (95, 109), the image pinned and updated (50), the default password changed, the port closed. Then the checklist (39) and a scan (171) with fresh eyes. CrowdSec (107) if it wasn't there.

Then a page in BookStack (88): what happened, when, how you found it, how they got in, what you rotated, what you changed. The next incident — and there is always a next one, somewhere — starts from that page.

Someone else's data (a small business, customers' records) may trigger a legal duty to report (in the UK, the ICO within 72 hours for personal data). Personal photos and files are yours to decide about. If money or identity is involved, Action Fraud, the bank, and a password change on every account that shared a password.

ps shows nothing odd but the CPU is pegged. A hidden process (rootkit). top vs ps disagreeing is itself the evidence. Rebuild.

The miner comes back after killing it. A cron entry, a systemd unit, a container with restart: always, or a second compromised box on the LAN. Capture, then rebuild.

No backup from before the breach. Restore the most recent, review every script and config by eye, scan the data with ClamAV (clamscan -r), and treat every secret as burned.

You're not sure it was a breach. A misbehaving app can look like one. Isolate and capture anyway; the capture tells you.

Cable out, capture to a stick, find the door, fresh install plus playbook plus pre-breach backups, rotate everything, fix the hole, write it down. It's a bad day, not a catastrophe — because the backups, the playbook and the drill were built for exactly this.