How to Diagnose Hardware Problems on a Linux Server (dmesg, memtest, sensors and USB)

Software problems leave logs that say what went wrong. Hardware problems leave a server that rebooted at 3 am with nothing in the journal, a USB drive that vanished mid-backup, or a crash every time the weather's warm. Linux still has the clues — the kernel log, temperature sensors, SMART, a memory test — you just have to know where each kind of fault shows up.

sudo dmesg -T | tail -50
sudo dmesg -T --level=err,warn
journalctl -k -b -1 -p warning # kernel messages from the PREVIOUS boot

-T gives real timestamps. After an unexplained reboot, the previous boot's last lines (-b -1) are what matter: a thermal warning, an MCE (machine check exception — CPU or RAM), I/O error on a disk, or nothing at all. Nothing at all points at the power supply or a hard lockup.

Watch live while you reproduce a fault:

sudo dmesg -wT

sudo apt install lm-sensors -y
sudo sensors-detect # answer YES to the defaults
sensors

CPU cores above about 85 °C under load, or any sensor at its "crit" value, is a cooling problem: dust, a dead fan, a dried-out thermal pad, or a mini PC in a cupboard with no airflow. Drive temperatures from SMART should stay under 45 °C. Log it to Netdata or Grafana (tutorials 66, 90) and the pattern — "crashes when the room is warm" — becomes obvious.

journalctl -k | grep -iE "thermal|throttl"

Throttling messages explain "slow when busy" without a crash.

Bad RAM is the classic cause of random crashes, corrupted files and failed builds. Test it properly from outside the OS:

sudo apt install memtest86+ -y

Reboot, pick Memory test (memtest86+) from the GRUB menu, and let it run at least one full pass (hours). Any red line means a bad stick: pull one at a time to find which. For a quick in-OS smoke test, sudo apt install memtester && sudo memtester 2G 1 tests 2 GB for one pass.

If the machine supports ECC RAM, sudo apt install rasdaemon && ras-mc-ctl --error-count shows corrected errors — a rising count is a stick on its way out.

"I/O error", "ata1: link is slow to respond", "resetting link", a drive that appears as sdb then sdc: often the cable rather than the drive, especially SATA cables and USB adapters. Order of suspicion:

1. sudo smartctl -a /dev/sdX (tutorial 56) — reallocated or pending sectors say the drive.
2. Swap the SATA cable / USB cable / port. If the errors move with the drive, it's the drive; if they stay with the port, it's the cable or controller.
3. UDMA_CRC_Error_Count rising in SMART = cable, almost always.

USB drives dropping out: usually power. A 2.5" drive on an unpowered hub or a Pi's USB port browns out under load. dmesg -w shows "device descriptor read error" or "over-current"; a powered hub or a drive with its own supply fixes it.

A failing PSU looks like nothing: reboots with no log, failures under load (every transcode, every backup), or "won't start first time". No software can test it directly. Clues: journalctl --list-boots showing many short uptimes, a UPS (tutorial 100) reporting the load jumping, and the problem following the PSU if you swap one in. Cheap PSUs and old capacitors are the usual suspects; a Pi with a non-official supply shows Under-voltage detected! in dmesg, and that's the whole diagnosis.

sudo ethtool enp3s0 | grep -E "Speed|Link"
sudo dmesg | grep -iE "link (up|down)|enp3s0"

A gigabit port negotiating at 100 Mb/s is a damaged cable or a bad plug. Repeated link up/down is a cable, a switch port or a NIC driver — try another port on the switch first. Realtek NICs on some kernels need the r8168 DKMS driver instead of r8169 if they drop under load.

sudo lshw -short
lspci
lsusb
sudo dmidecode -t memory

lshw lists what the kernel sees; if a device is missing from it, it's not a driver problem, it's not being detected — reseat it. dmidecode shows RAM slots, speeds and whether a stick dropped to a slower speed (a sign of trouble).

Change one thing and test. Note the time and journalctl -k -b -1 after each crash. Hardware faults are reproducible once you find the trigger (heat, load, a particular drive); until then they look random. And if a machine has had RAM, a PSU and a cable replaced and still crashes, the motherboard is the last suspect — and often the point to retire it (tutorial 74).

sensors shows nothing. sensors-detect wasn't run or needs a reboot; some VMs expose no sensors at all.

memtest86+ not in the GRUB menu. UEFI systems need the memtest86+ EFI build; the package installs it but you may need sudo update-grub, or boot it from a USB stick.

dmesg is empty for the previous boot. Journal isn't persistent: sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald.

dmesg -T for the kernel's complaints, sensors for heat, memtest overnight for RAM, SMART and a cable swap for disks, the "no log at all" rule for the power supply. Most hardware faults announce themselves in one of those places — and the ones that don't are usually the PSU.