A client called last week: "Our new server has 8 cores, but only half of them show any activity. The other 4 sit at 0% while the active ones hit 80-90%. Something's wrong with our setup."
This wasn't a broken server. It was a perfectly configured one that had inherited some CPU affinity settings from a previous application deployment. The symptoms looked like hardware failure, but the real culprit was process-level CPU binding that nobody remembered configuring.
Finding Which Processes Are Pinned
Start by identifying which processes are restricted to specific CPU cores. The /proc/PID/status file contains the CPU affinity mask for each process:
for pid in $(ps -eo pid --no-headers); do
if [ -f /proc/$pid/status ]; then
cpus_allowed=$(grep "Cpus_allowed_list" /proc/$pid/status 2>/dev/null)
if [[ $cpus_allowed != *"0-"*$(nproc --all)* ]]; then
echo "PID $pid: $cpus_allowed"
fi
fi
doneThis script loops through all running processes and prints any that don't have access to the full CPU range. You'll typically find application servers, database processes, or legacy services that were explicitly pinned to specific cores.
Understanding NUMA Topology Effects
Modern multi-core systems often split CPUs across multiple NUMA nodes. Check your system's NUMA layout:
lscpu | grep NUMA
numactl --hardwareIf your 8-core system has 2 NUMA nodes with 4 cores each, and your applications are configured to run on only one node, you'll see exactly the pattern described: 4 cores busy, 4 cores idle. This often happens when applications explicitly request memory locality optimisation.
The kernel documentation at kernel.org provides detailed information about NUMA memory policies and their CPU binding effects.
Clearing Inherited CPU Affinity
To remove CPU pinning from a running process, use taskset:
# Allow process to use all available CPUs
taskset -cp 0-$(expr $(nproc) - 1) PIDFor systemd services, check for CPUAffinity directives in unit files:
grep -r "CPUAffinity" /etc/systemd/system/
grep -r "CPUAffinity" /lib/systemd/system/Remove or modify these directives, then reload and restart the affected services. Many organisations inherit these settings from performance tuning exercises that made sense on previous hardware configurations but create bottlenecks on newer systems.
Monitoring CPU Distribution Properly
Standard monitoring tools sometimes mask per-core utilisation patterns. Tools like htop show individual core activity, but many monitoring dashboards only display aggregate CPU metrics. When debugging CPU distribution issues, you need per-core visibility over time, not just current snapshots.
Look for monitoring that tracks per-core utilisation historically. This helps distinguish between applications that legitimately use fewer cores (single-threaded workloads) and those that are artificially constrained by affinity settings.
Similar distribution problems occur with network interrupts and memory allocation. The same debugging principles that help with high context switches when vmstat shows everything normal apply here: the aggregate numbers can look fine while the underlying resource distribution tells a different story.
Preventing Future CPU Pinning Issues
Document any intentional CPU affinity settings in your configuration management. What looks like smart optimisation on a 4-core system often becomes a performance bottleneck when you upgrade to 8 or 16 cores.
Regularly audit your service configurations for CPU binding directives. Many applications include CPU affinity settings in their default configurations, and these often go unnoticed until you change hardware.
Most importantly, establish monitoring that tracks per-core utilisation over time. Catching CPU distribution problems early prevents the "why is our expensive new server slower than the old one" conversations that waste hours of debugging time.
If you need per-core CPU monitoring with historical tracking, Server Scout's dashboard provides exactly this visibility without the complexity of heavyweight monitoring solutions.