Alert Severity Levels and Escalation

Server Scout uses a three-tier severity system to help you prioritise alerts and respond appropriately to different types of issues. Understanding these severity levels and how to leverage them effectively is crucial for maintaining a robust monitoring strategy.

Understanding the Three Severity Tiers

Server Scout categorises all alerts into three distinct severity levels:

Info: Low-priority notifications that provide awareness of system changes or non-urgent conditions. These alerts typically don't require immediate action but help maintain visibility into system behaviour.

Warning: Medium-priority alerts indicating potential issues that warrant investigation. These suggest problems that could escalate if left unaddressed but aren't immediately critical.

Critical: High-priority alerts requiring immediate attention. These indicate serious problems that could cause service disruption or system failure if not resolved quickly.

Default Alert Conditions

Server Scout applies consistent default thresholds across key system metrics:

• CPU Usage: Warning at 80%, Critical at 90%
• Memory Usage: Warning at 80%, Critical at 90%
• Disk Usage: Warning at 80%, Critical at 90%

These thresholds provide sensible starting points for most environments, giving you advance warning before resources become critically constrained.

Visual Indicators and Prioritisation

Severity levels directly impact how alerts appear in the Server Scout interface:

Colour Coding: Critical alerts display in red, warnings in amber/orange, and info alerts in blue or grey. This immediate visual distinction helps you quickly identify priority issues.

Sort Priority: The alert dashboard automatically sorts alerts by severity, with critical alerts appearing first, followed by warnings, then info alerts. Within each severity tier, alerts are typically ordered by timestamp.

Display Prominence: Critical alerts receive enhanced visual treatment, including bolder text, larger icons, or highlighted backgrounds to ensure they capture immediate attention.

Alert Evaluation Engine Priority

Server Scout's alert evaluation engine processes alerts using a strict hierarchy:

1. Critical (highest priority)
2. Warning (medium priority)
3. Info (lowest priority)

This ordering ensures that when multiple conditions exist simultaneously, the most severe alert takes precedence in notifications and display. For example, if a server experiences both high CPU (critical) and moderate disk usage (warning), the critical CPU alert will be prominently featured.

Designing Your Escalation Strategy

Effective alert management requires a structured approach to each severity tier:

Info Alerts: Awareness and Trend Monitoring

Use info alerts for:

• Routine maintenance notifications
• Service restarts or configuration changes
• Threshold breaches that are notable but not concerning
• Performance trends that warrant awareness

Action: Review during regular maintenance windows or daily system checks.

Warning Alerts: Investigation and Prevention

Deploy warning alerts for:

• Resource usage approaching concerning levels
• Performance degradation that doesn't immediately impact users
• Security events requiring investigation
• Backup failures or data integrity concerns

Action: Investigate within business hours and implement corrective measures to prevent escalation.

Critical Alerts: Immediate Response

Reserve critical alerts for:

• Service outages or imminent failures
• Security breaches requiring immediate containment
• Resource exhaustion causing system instability
• Hardware failures affecting availability

Action: Immediate response required, potentially including out-of-hours notifications and emergency procedures.

Customising Default Thresholds

While Server Scout's defaults work well for most scenarios, you can adjust thresholds to match your specific requirements:

# Example threshold adjustment
serverscout config set cpu.warning 75
serverscout config set cpu.critical 85
serverscout config set memory.warning 85  
serverscout config set disk.critical 95

Consider your specific environment when customising:

• High-performance systems may require lower warning thresholds
• Batch processing servers might need higher thresholds during expected load periods
• Development environments often warrant more relaxed thresholds than production systems

Best Practices

Regularly review your alert severity configuration to ensure it remains aligned with your operational needs. Avoid "alert fatigue" by ensuring info and warning alerts provide genuine value rather than noise. Test your escalation procedures periodically to verify that critical alerts reach the right people quickly.

Remember that effective monitoring isn't just about detecting problems—it's about providing the right information to the right people at the right time, enabling swift and appropriate responses to keep your systems running smoothly.