€12,000 Monthly Power Bill Revealed CPU Instruction Mining Patterns That Process Lists Never Showed

The Investigation Timeline

A hosting provider's electricity bill jumped from €2,800 to €14,800 per month across their 47-server infrastructure. Traditional monitoring showed normal CPU utilisation. Process lists looked clean. Load averages appeared reasonable. Yet something was consuming massive amounts of power.

The breakthrough came through /proc/stat analysis. While htop and ps showed nothing suspicious, the kernel's raw instruction counters told a different story.

When Traditional Process Lists Failed

The mining operation used sophisticated evasion techniques. Process names mimicked legitimate system services: [kthreadd/2], systemd-resolve, NetworkManager. CPU affinity was carefully managed to keep individual core usage below monitoring thresholds.

Running ps aux --sort=-%cpu showed the highest consumer at 8.3% CPU usage. Nothing triggered the standard 15% CPU alerts. Yet the power consumption patterns suggested intensive mathematical operations across all cores.

This is where /proc/stat becomes invaluable for forensic analysis. The file contains raw kernel counters that can't be manipulated by userspace processes.

CPU Instruction Pattern Anomalies

The first anomaly appeared in context switch patterns. Normal server workloads generate 2,000-4,000 context switches per second. These servers were seeing 18,000-22,000.

cat /proc/stat | grep ctxt
ctxt 847293847

Monero's RandomX algorithm generates excessive context switches due to its memory-hard design. The mining software was using thread pools with hundreds of worker threads, each switching rapidly between CPU cores.

The intr field showed similar anomalies. Interrupt rates were 300% above baseline, suggesting intensive memory access patterns characteristic of cryptocurrency hashing.

Forensic Analysis Through /proc/stat

/proc/stat contains fields that reveal computational fingerprints invisible to traditional monitoring. The investigation focused on three key metrics:

• Context switches per CPU cycle: Normal ratio is 0.002-0.004. Mining operations showed 0.012-0.018
• Interrupt vectors per second: Baseline was 1,200/sec. Mining periods showed 4,800/sec
• Idle time distribution: Legitimate loads show uneven idle patterns. Mining creates uniform distribution across cores

Interrupt Vector Analysis

The interrupt patterns revealed the smoking gun. /proc/interrupts showed excessive timer interrupts and IPI (Inter-Processor Interrupt) activity:

cat /proc/interrupts | grep -E "LOC:|IPI"
 LOC: 2847392847 2847398234 2847401923 2847399187
 IPI: 18472847 18473923 18474821 18475234

RandomX algorithm requires frequent memory synchronisation between CPU cores, generating these characteristic interrupt spikes. The pattern was consistent across all affected servers.

Context Switch Pattern Recognition

Mining operations create distinctive context switch signatures. The /proc/stat ctxt field showed steady, mechanical increases rather than the bursty patterns of normal server workloads.

Normal workloads: context switches correlate with network traffic, disk I/O, or user activity. Mining workloads: context switches occur at precise intervals regardless of external load.

Power Consumption Correlation

Baseline vs Anomalous Usage

Power monitoring revealed the mathematical relationship between CPU instruction patterns and electricity consumption. Each server's baseline power draw was 180-220 watts. During mining periods, consumption jumped to 380-420 watts.

The correlation was precise: every 10,000 additional context switches per second corresponded to 12-15 watts of additional power consumption. This mathematical relationship became the foundation for automated detection.

Socket-level analysis proved equally revealing. Mining operations established consistent outbound connections to pool servers, but used legitimate-looking process names to avoid network monitoring alerts.

Mining Operation Reconstruction

Monero-Specific CPU Signatures

The investigation reconstructed the entire mining operation. The malware used several sophisticated techniques:

1. Process masquerading: Legitimate system service names
2. CPU affinity rotation: Spreading load across cores to avoid individual core monitoring
3. Priority manipulation: Running at nice values that kept processes below monitoring thresholds
4. Memory mapping: Direct memory access to avoid traditional memory monitoring

The RandomX algorithm's memory-hard design creates distinctive patterns in /proc/meminfo. The AnonHugePages field showed sustained elevation during mining periods - a clear forensic marker.

Prevention and Detection Framework

Based on this investigation, effective mining detection requires monitoring CPU instruction patterns rather than just process lists. Key detection metrics:

• Context switch rates above 8,000/sec sustained for >5 minutes
• Interrupt rates exceeding 200% of historical baseline
• Power consumption increases >40% without corresponding load increases
• Uniform CPU idle time distribution across cores

The hosting provider implemented comprehensive monitoring that combines /proc/stat analysis with power consumption baselines. This approach catches sophisticated mining operations that traditional process monitoring misses.

Server Scout's plugin system now includes CPU instruction pattern analysis specifically designed to detect these evasion techniques. The lightweight bash-based approach can identify mining signatures without the resource overhead that made detection difficult in the first place.

The lesson: sophisticated threats require forensic-level monitoring. Process lists and CPU percentages aren't enough when attackers understand how traditional monitoring works. The raw kernel counters in /proc/stat provide ground truth that can't be manipulated by userspace malware.

FAQ