Your new monitoring system is firing alerts every twelve minutes. Database CPU usage is apparently "critical" when the server is running normally. Memory warnings flood your inbox while applications perform flawlessly. And nobody—not your predecessor, not the documentation, not the git history—can explain why these thresholds were chosen.
Welcome to monitoring archaeology.
The Documentation Ghost Town
It starts innocently enough. Someone leaves the company. They take with them the tribal knowledge of why the database memory alert fires at 73% instead of 80%, or why the load average threshold is set to 2.3 on the web servers but 4.1 on the batch processing machines. The alerts keep running, but the reasoning behind them vanishes.
One team we spoke to inherited a Nagios configuration with 847 custom check definitions. Not a single one had comments explaining the threshold values. When alerts started firing constantly after a kernel update, they faced a choice: disable everything or begin the painful process of understanding what each metric actually meant.
"We spent three weeks getting woken up at 2 AM for alerts that meant absolutely nothing," their systems administrator explained. "A disk space warning would fire when we had 40GB free. A load alert would trigger during normal application startup. We started ignoring everything, which is exactly when real problems slip through."
Six Months of 3 AM Wake-Up Calls
The team's first instinct was logical but wrong: adjust thresholds based on what "felt right." They raised the disk space warning from 85% to 90%, bumped memory alerts higher, and lengthened the CPU threshold periods. The false positives decreased, but so did their confidence in the entire system.
Months later, an actual performance crisis went undetected for six hours because the thresholds had been set too high. The monitoring system that should have provided early warning had become a collection of arbitrary numbers with no connection to business impact.
This is what happens when threshold decisions lose their historical context. Without understanding why previous administrators chose specific values, teams either accept constant noise or raise alerts so high they miss genuine problems.
Detective Work: Reverse Engineering Alert Logic
The breakthrough came when they stopped trying to guess and started measuring. Instead of debating whether 75% memory usage was "too high," they collected two weeks of baseline data during known-good periods.
The patterns were revealing. Their web servers typically ran at 68-72% memory during normal operation, with brief spikes to 84% during cache rebuilds every morning at 6 AM. The original 73% threshold wasn't arbitrary—it was carefully chosen to sit above normal operations but catch genuine memory leaks before they caused problems.
Load average told a similar story. The "weird" threshold of 2.3 made perfect sense once they realised their applications spawned exactly two worker processes per CPU core during busy periods. On their four-core systems, normal peak load was 2.1, making 2.3 a reasonable early warning.
This detective work revealed the fundamental truth about monitoring thresholds: they're not universal constants but carefully calibrated tools that reflect specific applications on specific hardware under specific workloads.
The Hidden Cost of Undocumented Thresholds
While the team spent six months fighting their monitoring system, they missed genuine capacity planning opportunities. Historical data showed gradual memory growth that would have justified hardware upgrades months earlier. Disk usage patterns revealed poor log rotation configurations that wasted storage.
The real cost wasn't just the time spent on false alerts—it was the lost confidence in infrastructure monitoring itself. Team members began making decisions based on gut feelings rather than metrics, leading to both over-provisioning (expensive) and under-provisioning (risky).
Documentation debt compounds exponentially. Each person who leaves takes irreplaceable context. Each new team member must either accept existing configurations blindly or spend weeks rebuilding knowledge that once existed.
Creating Future-Proof Alert Documentation
Their solution went beyond simple comments in configuration files. They built a decision log that connected each threshold to specific business metrics:
- Baseline measurements: "Web server memory usage: 68-72% normal, peaks to 84% during 6 AM cache rebuild"
- Business impact: "Memory above 85% causes 300ms response time degradation, affecting customer checkout success rate"
- Historical incidents: "Previous memory leak in July 2025 went undetected until 94% usage caused application failures"
- Review schedule: "Reassess quarterly as application deployment frequency increases"
This approach transformed thresholds from mysterious numbers into documented business decisions that future team members could understand and adjust appropriately.
They also implemented smart alerting with sustain periods, reducing noise from brief spikes while maintaining sensitivity to genuine problems. A memory alert now required three consecutive measurements above threshold rather than a single spike.
Building Monitoring That Survives Team Changes
The most important lesson was treating alert configurations as critical infrastructure documentation. When someone proposes changing a threshold, they must document both the old reasoning and the new justification.
Modern lightweight monitoring tools make this easier by providing built-in documentation features and reasonable defaults based on common application patterns. Server Scout, for example, includes contextual help that explains why each default alert condition is set at specific values for typical server workloads.
The goal isn't perfect thresholds—it's documented reasoning that survives team transitions. When the next person inherits your monitoring system, they should understand not just what alerts exist but why they were created and how they've evolved.
Prevention Strategies for New Teams
Smart teams document threshold decisions before crisis forces them to. Every monitoring configuration should include:
- Current baseline measurements during normal operations
- Business impact of threshold violations
- Historical context for why specific values were chosen
- Regular review dates to reassess as infrastructure evolves
The documentation effort pays dividends during incident response. Instead of wondering whether an alert represents genuine urgency, teams can quickly reference the business impact and respond appropriately.
Monitoring is ultimately about confidence—confidence that alerts represent real problems, that thresholds reflect actual business needs, and that the system will warn you before customers notice issues. Documentation debt destroys that confidence, turning monitoring from a strategic advantage into a source of constant friction.
Good monitoring documentation doesn't just prevent 3 AM false alarms—it preserves institutional knowledge that helps teams make better infrastructure decisions for years to come.
FAQ
How do you determine appropriate alert thresholds for a new system without historical data?
Start with conservative defaults and collect baseline data for 2-3 weeks during normal operations. Monitor the 95th percentile of each metric during known-good periods, then set initial thresholds 10-15% above these peaks. Document your reasoning and plan to adjust based on real operational experience.
What's the minimum documentation needed for each monitoring threshold?
Each threshold should include: current baseline values, business impact of violations, why this specific number was chosen, and when it should be reviewed. A single paragraph per alert explaining "normal is X, problems start at Y, so we alert at Z" prevents most documentation debt.
How often should alert thresholds be reviewed and updated?
Review quarterly for active systems, or whenever you change hardware, software, or application deployment patterns. Set calendar reminders rather than waiting for problems. Infrastructure changes faster than documentation, so regular reviews prevent thresholds from becoming obsolete.