Troubleshooting Alert Delivery

When alerts aren't being delivered as expected, it can leave you in the dark about critical server issues. This guide walks through systematic troubleshooting steps to diagnose and resolve alert delivery problems in Server Scout.

Testing Your Alert Channels

The first step in troubleshooting is to verify your notification channels are configured correctly using the built-in test feature:

serverscout test_channel email
serverscout test_channel webhook

This sends a test notification through your configured channels. If the test fails, the issue lies with your channel configuration rather than the alerting logic itself.

Troubleshooting Email Delivery Issues

Email delivery problems are often the most common alert delivery issue. Here are the key areas to investigate:

SMTP Configuration

Verify your SMTP settings in the Server Scout configuration:

# Check your current SMTP configuration
serverscout config show | grep -A 10 "email"

Common SMTP issues include:

• Incorrect server hostname or port number
• Wrong authentication credentials
• Missing TLS/SSL configuration for secure servers
• Firewall blocking outbound connections on SMTP ports (25, 587, 465)

Spam Filters and Sender Reputation

Server Scout alerts may be caught by spam filters. Check:

• Spam/junk folders for missing alerts
• Email server logs for rejected messages
• Your domain's SPF, DKIM, and DMARC records if sending from a custom domain
• Whether your server's IP address is on any blacklists

Consider using a dedicated email service like SendGrid or Amazon SES for improved deliverability.

Troubleshooting Webhook Delivery Issues

Webhook failures often have different root causes than email problems:

SSRF Protection Blocking Internal URLs

Server Scout includes SSRF (Server-Side Request Forgery) protection that blocks requests to private IP ranges. If your webhook endpoint uses internal addresses like 192.168.x.x or 10.x.x.x, you'll need to either:

• Configure your webhook to use a public endpoint
• Disable SSRF protection in the configuration (not recommended for production)

SSL Certificate Errors

For HTTPS webhooks, certificate issues can prevent delivery:

# Test SSL certificate validity
curl -I https://your-webhook-endpoint.com/alerts

Common SSL problems include expired certificates, self-signed certificates, or hostname mismatches.

Connection Timeouts and Failures

Network connectivity issues can prevent webhook delivery:

# Test basic connectivity
curl -X POST -H "Content-Type: application/json" \
  -d '{"test": "message"}' \
  https://your-webhook-endpoint.com/alerts

Check for:

• Firewall rules blocking outbound HTTPS traffic
• DNS resolution issues
• Target server downtime or overloading

Authentication Failures

If your webhook endpoint requires authentication, verify:

• API keys or tokens are correctly configured
• Authentication headers are properly formatted
• Credentials haven't expired or been revoked

Verifying Webhook Accessibility

Ensure your webhook endpoints are publicly accessible and responding correctly:

1. Test the endpoint from an external network
2. Verify the endpoint accepts POST requests with JSON payloads
3. Check that the endpoint responds with appropriate HTTP status codes (200-299)
4. Confirm any required authentication is working

Checking the Notification Log

Server Scout maintains logs of notification attempts:

# View recent notification attempts
serverscout logs --filter=notifications --tail=50

# Check for specific delivery failures
grep "notification failed" journalctl -u scout-agent

These logs will show delivery timestamps, success/failure status, and error messages that can help pinpoint issues.

Diagnosing Missing Alerts

If alerts aren't firing when expected, the issue may be with the alerting logic rather than delivery:

Sustain Period Configuration

Alerts require conditions to persist for a configured duration before firing:

# Check your alert sustain periods
serverscout alerts list --show-config

A high sustain period might delay alerts longer than expected.

Cooldown Periods

After an alert fires, cooldown periods prevent repeated notifications:

# Check active cooldowns
serverscout alerts status --show-cooldowns

Metric Collection Issues

Verify that the metrics your alerts depend on are being collected:

# Check recent metric collection
serverscout metrics status
serverscout metrics history cpu_usage --last=1h

Missing or stale metrics will prevent alerts from evaluating properly.

By working through these troubleshooting steps systematically, you should be able to identify and resolve most alert delivery issues, ensuring you stay informed about your server's health.