How to Fix a Linux Server That Won't Boot (GRUB, fsck, fstab and Emergency Mode)
A server that won't boot looks catastrophic and almost never is. Nine times out of ten it's one of five things, each with a known fix: a drive mentioned in fstab that isn't there, a filesystem that wants checking after a power cut, a bootloader that lost its place, a /boot with no space, or a kernel update that doesn't like the hardware. Plug in a keyboard and monitor and work down the list.
Quick reference
| What you see | Likely cause | Fix |
|---|---|---|
| "You are in emergency mode" | bad fstab line, or a filesystem error | comment the line out / fsck |
| "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY" | dirty filesystem | fsck -y /dev/sdXN |
| "No bootable device" / "grub rescue>" | GRUB missing or BIOS boot order | reinstall GRUB / fix boot order |
| Boots to a black screen after an update | kernel problem | boot the previous kernel from GRUB |
| "error: no space left" during update, then no boot | /boot full | remove old kernels from recovery |
Step 0: Read the screen
Press Esc during the splash to see the text. The last few lines before it stops are the diagnosis. Scroll up with Shift + Page Up if you need to.
1. Emergency mode (bad fstab)
The screen says "You are in emergency mode" and offers a root prompt (press Enter; it may ask for the root password — if there's none, use recovery mode, tutorial 37).
journalctl -xb | grep -i -E "fail|mount" | tail
If it names a mount — a USB backup drive that's unplugged, an NFS server that's off — that's it. Make the disk writable and comment the line out:
mount -o remount,rw /
nano /etc/fstab
Put # in front of the offending line (or add nofail to its options — which you should always have had, tutorial 22). reboot.
2. Filesystem needs checking
"UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY" or "fsck failed" after a power cut. At the emergency prompt:
fsck -y /dev/sda2
Use the device it named (the root partition from lsblk). -y answers yes to every repair. For the root filesystem itself, it's cleaner from a live USB or by forcing a check at next boot:
touch /forcefsck
reboot
A data drive: umount /mnt/data then fsck -y /dev/sdb1.
3. Boots to the old kernel fine, new one doesn't
A kernel update that doesn't like the hardware shows as a black screen, a hang, or a kernel panic. At the GRUB menu (hold Shift or tap Esc at boot): Advanced options for Ubuntu → pick the previous kernel version. If it boots:
sudo apt-mark hold linux-image-generic linux-headers-generic
to stop it happening again until you've looked into it, and check journalctl -b -1 -p err for what the failed boot complained about. Often it's a proprietary driver or an odd NIC; the next kernel usually fixes it.
4. /boot is full
Updates fail with "No space left on device" on /boot, then the next reboot has a half-installed kernel. From recovery mode or the previous kernel:
df -h /boot
dpkg -l 'linux-image-*' | grep ^ii
uname -r
Remove old kernels that aren't the running one:
sudo apt purge linux-image-6.8.0-31-generic
sudo apt autoremove --purge -y
sudo apt install -f
sudo update-grub
5. GRUB is missing
"No bootable device", "Operating system not found", or grub rescue>. First check the BIOS boot order — a BIOS reset or a new drive often just moves things around. If the order is right:
Boot an Ubuntu Server USB, choose Try/rescue (or on the installer, Shift + F2? — easiest is a desktop live USB), then:
sudo mount /dev/sda2 /mnt # root partition
sudo mount /dev/sda1 /mnt/boot/efi # EFI partition, UEFI systems
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done
sudo chroot /mnt
grub-install /dev/sda
update-grub
exit
sudo reboot
On a UEFI system grub-install needs the EFI partition mounted as shown; efibootmgr afterwards shows the entry.
6. Still nothing
- Disk failing? From a live USB, sudo smartctl -H /dev/sda (tutorial 56). If it's bad, this is a restore-from-backup situation.
- RAM? Memtest86+ is on the GRUB menu of most installs; random panics are often memory.
- Just a VM? Snapshot rollback, then investigate at leisure.
Prevent the next one
nofail on every non-root fstab entry, a /boot with 2 GB rather than 500 MB on the next install, apt autoremove as part of your update habit, and a tested backup so the worst case is an afternoon rather than a loss.
Troubleshooting the troubleshooting
Keyboard doesn't work at the emergency prompt. Try another USB port; some only initialise after boot. A PS/2 keyboard always works on old hardware.
Emergency mode asks for a root password you don't have. Recovery mode from GRUB gives a root shell without one (tutorial 37).
fsck says the filesystem is mounted. You can't check a mounted filesystem; use /forcefsck or a live USB.
Fixed it, but it happens every reboot. You fixed the symptom. journalctl -b -1 from the previous boot has the cause.
Wrapping up
Read the screen, comment out the fstab line, fsck -y, pick the old kernel, clear /boot, reinstall GRUB. In that order, one of them is almost always it. The server you thought was dead is usually back in ten minutes — and the backup you tested is why you can stay calm while finding out.
