Recovery and Resolved Alerts

Understanding Recovery Notifications

When Server Scout detects that a previously firing alert has returned to normal conditions, it automatically sends a recovery notification to inform you that the issue has been resolved. This creates a complete monitoring lifecycle that keeps your team informed from problem detection through to resolution.

How Recovery Alerts Work

Recovery notifications trigger when a metric that was in a firing state returns to normal values. The moment Server Scout detects that the problematic condition has cleared, the alert status transitions from "firing" to "ok" and immediately dispatches recovery notifications to all configured notification channels.

These recovery messages include several key pieces of information:

• The current metric value (now within normal parameters)
• Clear indication that the alert has resolved
• Timestamp of when the recovery was detected
• Original alert details for context

Visual Indicators and Logging

Recovery notifications are designed to stand out from problem alerts through distinctive visual cues:

Webhook notifications use green severity colouring to immediately signal good news, contrasting with the red used for firing alerts. This colour coding helps teams quickly distinguish between new problems and resolved issues when notifications appear in Slack, Discord, or other webhook-enabled platforms.

In Server Scout's notification log, recovery alerts are clearly marked as "resolved" status, making it easy to track which issues have been addressed and which remain active.

Why Recovery Notifications Matter

Recovery alerts serve several critical functions for effective monitoring:

Automated Confirmation

Rather than requiring manual checks to verify that problems have been resolved, recovery notifications provide automatic confirmation that systems have returned to normal. This eliminates guesswork and reduces the time teams spend manually verifying fixes.

Closing the Loop

For on-call teams, recovery notifications complete the incident lifecycle. When an engineer receives an alert at 3 AM, they want definitive confirmation when their fix has worked—not uncertainty about whether the problem persists.

Reducing Alert Fatigue

By clearly communicating when issues resolve, recovery notifications help teams trust their monitoring system and reduce the tendency to ignore or disable noisy alerts.

Recovery and Cooldown Periods

Recovery notifications interact with Server Scout's cooldown periods in an important way: a recovery resets the alert state immediately, regardless of any configured cooldown period.

Here's how this works in practice:

1. An alert fires and enters cooldown (e.g., 30 minutes)
2. The underlying issue resolves after 10 minutes
3. Server Scout immediately sends a recovery notification
4. The alert state resets to "ok"
5. If the same condition occurs again, a new alert can fire without waiting for the original cooldown to expire

This behaviour ensures that genuine recoveries are reported promptly, whilst cooldowns continue to prevent notification spam for persistent, intermittent issues.

Viewing Alert Lifecycle

Server Scout's notification history provides a complete view of each alert's lifecycle. For any given alert, you can observe the full sequence:

1. Initial firing - when the problem was first detected
2. Reminder notifications - if configured and the issue persisted
3. Recovery notification - when the issue resolved

To view this information:

1. Navigate to the Alerts section in your Server Scout dashboard
2. Select Notification History
3. Filter by server or alert type if needed
4. Review the chronological sequence of notifications

This comprehensive logging helps with post-incident analysis, pattern identification, and understanding mean time to recovery (MTTR) for different types of issues.

Best Practices

Enable recovery notifications for all critical alerts—the small overhead of additional notifications is far outweighed by the operational benefits of confirmed issue resolution. Consider configuring different notification channels for recoveries if your team prefers to handle resolved issues differently from active problems.

Recovery notifications transform monitoring from a one-way alarm system into a complete communication tool that supports your team throughout the entire incident lifecycle.