Finding Runaway Processes When pidstat Shows Everything Normal

The Invisible Resource Drain

You've been there: users complaining about sluggish response times, htop showing elevated system load, but when you run pidstat -u 5 or check the usual suspects, nothing jumps out. Every process looks reasonable. The load average sits stubbornly high whilst your per-process breakdowns suggest the server should be practically idle.

The problem often lies in short-lived processes that spawn and die faster than your monitoring interval can catch them. These transient processes can absolutely hammer your system whilst remaining nearly invisible to standard monitoring tools.

Why Standard Tools Miss the Culprits

Most monitoring tools sample at regular intervals - typically every 5, 10, or 30 seconds. A process that runs for 200 milliseconds every second can consume 20% of your CPU but never appear in a 5-second sample window.

Classic examples include:

• Cron jobs spawning subprocesses in rapid succession
• Web applications launching external scripts for image processing or PDF generation
• Log rotation scripts that fork dozens of compression processes
• Database maintenance tasks creating temporary worker processes

These workloads create what performance engineers call "CPU burst debt" - sustained resource consumption from processes that individually appear lightweight.

Catching the Ghosts

Linux's process accounting can track every process execution, regardless of runtime. Enable it with:

sudo systemctl enable psacct
sudo systemctl start psacct

After running for a few hours, analyse the accumulated data:

# Show processes sorted by total CPU time
sa -m

# Show command frequency and total runtime
sa -c

The -m flag reveals which commands consumed the most CPU cycles, whilst -c shows execution frequency. Look for patterns: a process appearing thousands of times per hour might explain your phantom load.

For real-time hunting, execsnoop from the BCC toolkit traces every execve() call:

sudo execsnoop-bpfcc

You'll see a live stream of process launches. Watch for rapid-fire execution of the same binary or unexpected external tools being called repeatedly.

The eBPF Alternative

On newer kernels, bpftrace provides surgical precision:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s -> %s\n", comm, str(args->filename)); }'

This traces every program execution with its parent process, helping you identify which service or daemon is spawning the resource-hungry children.

The Linux kernel documentation covers process accounting in detail if you need to dig deeper into the data format.

Prevention and Monitoring

Once you've identified the culprit, fix it at the source. Replace shell scripts that spawn multiple processes with single-binary alternatives. Cache expensive operations. Batch file processing instead of handling items individually.

Server Scout's real-time metrics help spot these patterns early. The dashboard tracks CPU usage at sub-minute intervals and can alert when load averages climb without obvious cause - giving you a heads-up before users start complaining.

If you're tired of playing detective with invisible processes, Server Scout's monitoring catches load anomalies as they develop rather than after the damage is done.