Your Kubernetes dashboard reports healthy CPU utilisation across all pods. Response times are degrading, but nothing looks alarming in the metrics. Three minutes later, critical pods terminate due to resource pressure that never appeared in your standard monitoring stack.
This scenario plays out daily in production clusters running cgroups v2. The problem isn't your monitoring - it's that traditional Kubernetes metrics aggregate CPU usage over time windows that completely miss the throttling events that destroy application performance.
Why Standard Kubernetes Metrics Miss CPU Throttling Events
Kubernetes CPU metrics focus on average utilisation across time periods, typically 30-60 seconds. CPU throttling happens at microsecond granularity when pods exceed their resource quotas. By the time kubectl top shows elevated CPU, the damage is already done.
The kernel's cgroups v2 CPU controller tracks throttling events continuously in cpu.stat, but these metrics don't surface through standard Kubernetes monitoring. Applications experience sudden latency spikes while dashboards show normal resource consumption.
Throttling manifests as processes waiting for CPU time allocation rather than high CPU usage. Traditional monitoring interprets this as low system load, missing the resource contention entirely.
Understanding cgroups v2 CPU Controller Changes in Modern Clusters
Modern Kubernetes distributions default to cgroups v2, which changed how CPU throttling gets measured and reported. The cpu.stat file now uses throttledusec instead of the older cpu.cfsthrottled_us metric.
Key Differences from cgroups v1 Throttling Detection
The v2 controller tracks throttling time with microsecond precision rather than period counts. This provides more accurate detection of brief throttling events that would be averaged out in v1 metrics.
Container runtime interfaces expose these metrics differently. Docker's cgroups v2 implementation places CPU controller data in /sys/fs/cgroup/system.slice/docker-[container_id].scope/cpu.stat, while containerd uses different path structures.
Reading /proc/stat for Container-Level CPU Throttling
The /proc/stat file contains system-wide CPU accounting that reveals throttling patterns across all processes. The guest and guest_nice fields track CPU time allocated to containerised processes.
Monitoring the relationship between user CPU time and guest CPU time shows when containers consume disproportionate scheduling overhead. Throttled containers generate higher context switch rates visible in the ctxt field.
grep -E '^cpu |^ctxt' /proc/stat
cat /sys/fs/cgroup/cpu.stat | grep throttled_usecIdentifying Throttled vs Normal CPU Wait States
Normal CPU wait manifests as increased iowait in /proc/stat. Throttling appears as elevated steal time combined with low idle percentages. The key indicator is high context switches (ctxt) with normal interrupt counts (intr).
Throttled processes generate distinctive patterns in the processes and procs_blocked fields. Running processes remain constant while blocked processes increase during throttling events.
Calculating Throttling Percentage from Raw Metrics
Throttling percentage equals throttled_usec divided by total CPU time for the measurement period. Values above 5% indicate significant performance impact, while anything over 15% typically leads to application timeouts.
The calculation must account for multi-core systems where throttling can affect individual cores differently. Per-CPU lines in /proc/stat (cpu0, cpu1, etc.) reveal uneven throttling distribution that system-wide averages hide.
Building Early Warning Detection Before Pod Eviction
Effective throttling detection requires monitoring rate of change rather than absolute values. A pod consuming 200% of its CPU limit for 10 seconds causes more damage than steady 150% consumption over minutes.
Combining /proc/stat metrics with cgroup-specific cpu.stat data provides comprehensive throttling visibility. System-wide patterns indicate cluster-level resource pressure, while per-container metrics identify specific problematic pods.
Track the ratio between throttledusec and usageusec in container cpu.stat files. Ratios above 10% warrant investigation, while anything over 25% indicates immediate performance degradation.
Setting Meaningful Throttling Thresholds
Different application types tolerate throttling differently. Batch processing jobs handle brief throttling better than real-time APIs. Database containers require stricter thresholds than static web servers.
Set initial alerts at 15% throttling ratio for critical services, 25% for standard workloads, and 40% for batch processing. Tune these based on application-specific latency requirements rather than generic recommendations.
Correlating /proc/stat Data with Kubernetes Resource Requests
Kubernetes resource requests define minimum guaranteed resources, while limits set maximum consumption. Throttling occurs when pods exceed requests but stay within limits, creating performance degradation without resource violations.
Compare throttling metrics against pod resource specifications to identify misconfigured limits. Pods with low requests but high limits experience intermittent throttling that appears random without this context.
Container memory monitoring faces similar challenges with cgroups v2 reporting discrepancies. Memory and CPU throttling often occur together during resource pressure.
Server Scout's lightweight agent monitors both system-wide /proc/stat metrics and container-specific cgroup data without adding monitoring overhead that could worsen throttling. The bash-based architecture consumes under 3MB RAM while tracking throttling patterns across all containers.
Correlating system scheduling metrics with service health provides early warning before traditional Kubernetes monitoring detects problems. Service monitoring often reveals that slow startup times actually indicate CPU resource pressure rather than dependency issues.
Throttling detection through /proc/stat analysis gives 90 seconds advance warning before most Kubernetes clusters begin pod evictions. This early detection window enables proactive resource scaling or workload redistribution that prevents service disruptions.
FAQ
Can CPU throttling occur even when kubectl top shows low CPU usage?
Yes, kubectl top shows averaged CPU consumption over time windows, typically 30-60 seconds. CPU throttling happens at microsecond granularity when pods briefly exceed their limits. Short throttling bursts get averaged out in standard metrics while still causing significant performance degradation.
How does cgroups v2 throttling detection differ from cgroups v1?
cgroups v2 uses throttledusec in the cpu.stat file with microsecond precision, while v1 used cpu.cfsthrottled_us with period-based counting. The v2 approach provides more accurate detection of brief throttling events and better visibility into cumulative throttling impact over time.
What throttling percentage should trigger alerts in production environments?
Start with 15% throttling ratio for critical services, 25% for standard workloads, and 40% for batch processing. These thresholds should be tuned based on your application's latency requirements. Any throttling above 5% typically indicates performance impact, while 25%+ usually causes visible application slowdowns.