How to Debug a Misbehaving Program with strace, lsof and ltrace (See What It's Actually Doing)

Tutorials 14 and 80 start with the logs. Sometimes the log is empty, or says "error" with no detail, or the program just hangs. Linux lets you watch what a process is doing at the kernel boundary: every file it opens, every network connection it makes, every permission it's denied, in real time. strace does that; lsof shows what's open right now; ltrace watches library calls; /proc holds the rest. They're how you find "it's reading the config from the wrong path" or "it's waiting on a DNS lookup that never returns" in a minute instead of an evening.

sudo apt install strace -y

Attach to something stuck:

sudo strace -p 12345

If the last line is read(5, … and nothing follows, it's blocked reading file descriptor 5 — ls -l /proc/12345/fd/5 says what that is (a pipe, a socket to a database, an NFS file). futex(…) forever is a thread waiting on a lock. connect(…) then a pause is a network timeout; poll/epoll_wait repeating is a healthy idle loop.

Run a program under it:

strace -f -o /tmp/trace.txt -s 200 -tt ./backup.sh

-f follows child processes (scripts spawn many), -o to a file (the trace is long), -s 200 shows more of each string, -tt timestamps. Then grep the file.

Filter to what matters:

strace -f -e trace=%file -o /tmp/files.txt ./program # open/stat/access — "which config does it read?"
strace -f -e trace=%network ./program # connect/sendto — "where is it talking to?"
strace -f -e trace=execve ./program # what commands a script runs

The classic finds: openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT — it's looking in /etc/app/, not where you put the file. connect(3, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("127.0.0.1")}) = -1 ECONNREFUSED — it's using localhost inside a container (101). openat(…, "/var/lib/app/data") = -1 EACCES — permissions (11).

Summary of time spent: strace -c -p PID for a few seconds, then Ctrl+C — a table of calls by count and time; a program spending all its time in read on one fd, or thousands of stat calls, tells you where it's slow.

The container's processes are visible on the host — docker top name shows PIDs — so sudo strace -p from the host works without changing the image. To run strace inside, the container needs cap_add: [SYS_PTRACE] and the tool installed; the host route is easier.

sudo apt install lsof -y
sudo lsof -p 12345 # every file, socket, library this process holds
sudo lsof -i :8096 # who's listening/connected on a port
sudo lsof /mnt/data # who's using the mount (the "target is busy" answer, 22)
sudo lsof +D /mnt/data/photos # everything open under a folder (slow on big trees)
sudo lsof -u www-data # everything a user has open
sudo lsof | grep deleted # deleted-but-open files eating disk (15)
sudo lsof -i -n -P | grep ESTABLISHED # live connections, numeric

lsof -p on a stuck process plus strace -p on the same process usually closes the case: the fd number from strace matches a line in lsof.

For a program that fails inside a library rather than at the kernel:

sudo apt install ltrace -y
ltrace -e 'getenv+fopen+dlopen' ./program

Shows which environment variables it reads (the getenv for DATABASE_URL that you spelt wrongly), which files via the C library, which plugins it loads. Less useful on Go/Rust static binaries; invaluable on C and Python extensions.

cat /proc/PID/status # state (S sleeping, D disk-wait, Z zombie), memory, threads
ls -l /proc/PID/fd # every open fd → what it points to
cat /proc/PID/cmdline | tr '\0' ' ' # the exact command line
sudo cat /proc/PID/environ | tr '\0' '\n' # its environment — the secret it's actually using
cat /proc/PID/limits # open-files limit (a server hitting "too many open files")
readlink /proc/PID/cwd /proc/PID/exe # where it runs, what binary
cat /proc/PID/stack # kernel stack: what a D-state process waits on (17)

Nextcloud's cron shows nothing and the admin page says it hasn't run. docker top nextcloud → the cron PID. sudo strace -f -p PID -e trace=%file,%network while triggering it: connect(… 6379 …) = -1 ETIMEDOUT — Redis. docker compose ps says Redis is up; lsof -i :6379 on the host shows nothing listening on the address Nextcloud uses. The compose file has REDIS_HOST: 192.168.1.50 but Redis isn't published — fix to redis (101). Three commands, no log needed.

"Operation not permitted" attaching. Ptrace scope: sudo, or echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope for your own processes.

Output is overwhelming. Always -e trace=… and -o file; grep for = -1 to see only failures.

strace slows the program. Expected (every syscall is intercepted); don't leave it attached to a database in production. -c summaries and short sessions.

Process is in D state and strace hangs. It's stuck in the kernel (disk/NFS); /proc/PID/stack and tutorial 117.

strace -p for the hang, strace -e trace=%file for the wrong path, %network for the wrong address, lsof for who holds what, /proc/PID/environ for the real config, ltrace for libraries. When the log won't say, the kernel will — and it's usually one line you can grep for = -1.