The IOPS Obsession Problem
You've just migrated your database cluster to a shiny new NVMe array. The vendor specs promise 500,000 IOPS, iostat confirms you're hitting those numbers, yet your applications are slower than they were on the old SATA SSDs. Welcome to the world where IOPS becomes a vanity metric.
The issue isn't that IOPS don't matter - they do. But for NVMe storage, especially in RAID configurations, IOPS tell you almost nothing about user experience. A single 4KB random read that takes 50 microseconds feels identical to one that takes 500 microseconds when you're counting operations per second. Your database doesn't care about throughput when it's waiting for a single critical index lookup.
What to Monitor Instead
Latency percentiles are your new best friend. While iostat shows average response times, NVMe performance lives in the outliers. Monitor the 95th, 99th, and 99.9th percentile latencies using tools like fio with the --output-format=json flag for historical tracking.
The /sys/block/nvme*/queue/ directory contains queue depth metrics that traditional tools miss. Check nr_requests against actual utilisation - NVMe can handle queue depths of 65,535, but most applications never come close to saturating this.
Thermal throttling is particularly nasty on dense NVMe arrays. Monitor /sys/class/thermal/thermal_zone*/temp and correlate temperature spikes with performance drops. Many enterprise NVMe drives start throttling at 70°C, long before the official 85°C limit.
The RAID Controller Bottleneck
Here's where things get interesting. Your NVMe drives might be capable of sub-100 microsecond latencies, but if they're behind a RAID controller designed for spinning disks, you're adding milliseconds of overhead.
Check your RAID controller's queue depth settings with cat /sys/block/sd*/queue/nr_requests. Many controllers default to queue depths optimised for mechanical drives (typically 128), which creates artificial bottlenecks for NVMe performance.
For software RAID using mdadm, monitor /proc/mdstat for rebuild activity that isn't obvious from standard disk metrics. A background reshape operation can cut NVMe performance in half while barely registering in iostat output.
Monitoring in Practice
Setting up proper NVMe monitoring means collecting metrics that traditional tools weren't designed for. You need something that can track latency percentiles over time, correlate thermal events with performance drops, and alert on queue depth saturation before it impacts applications.
Server Scout's bash-based monitoring includes NVMe-specific plugins that track these metrics without the overhead of heavyweight monitoring agents. The plugin system lets you collect thermal zone data, queue statistics, and custom latency measurements that actually matter for modern flash storage.
For NVMe arrays, the goal isn't maximum IOPS - it's consistent, predictable latency. Monitor what your applications actually experience, not what looks impressive in vendor benchmarks.