Integrating Alerts with PagerDuty

Connecting Server Scout with PagerDuty enables robust incident management and escalation for your server monitoring alerts. This integration ensures your team responds promptly to critical infrastructure issues through PagerDuty's powerful notification and escalation system.

Setting Up PagerDuty Integration

Creating a PagerDuty Service

1. Log into your PagerDuty account and navigate to Services → Service Directory
2. Click New Service to create a dedicated service for Server Scout
3. Provide a descriptive name such as "Server Scout Monitoring"
4. Under Integration Settings, select Events API v2 as the integration type
5. Configure your escalation policy (we'll cover best practices later)
6. Complete the service creation process

Obtaining the Integration Key

Once your service is created:

1. Navigate to the Integrations tab of your newly created service
2. Locate the Events API v2 integration
3. Copy the Integration Key - you'll need this for Server Scout configuration
4. Store this key securely as it authenticates Server Scout with your PagerDuty service

Configuring Server Scout

Setting Up the Webhook

To configure Server Scout to send alerts to PagerDuty:

1. Access your Server Scout dashboard
2. Navigate to Settings → Integrations → Webhooks
3. Add a new webhook with these settings:

- Name: PagerDuty Integration - URL: https://events.pagerduty.com/v2/enqueue - Method: POST - Content Type: application/json

Webhook Payload Configuration

Configure the webhook payload to include your PagerDuty integration key:

{
  "routing_key": "YOUR_INTEGRATION_KEY_HERE",
  "event_action": "trigger",
  "dedup_key": "{{server_id}}-{{check_type}}",
  "payload": {
    "summary": "{{alert_summary}}",
    "source": "{{server_name}}",
    "severity": "{{severity}}",
    "component": "{{check_type}}",
    "group": "server-monitoring",
    "custom_details": {
      "server": "{{server_name}}",
      "metric": "{{metric_name}}",
      "value": "{{current_value}}",
      "threshold": "{{threshold_value}}"
    }
  }
}

Replace YOURINTEGRATIONKEY_HERE with the integration key obtained from PagerDuty.

Severity Mapping

Server Scout alert severities map directly to PagerDuty severities:

• Critical alerts → PagerDuty critical severity
• Warning alerts → PagerDuty warning severity
• Info alerts → PagerDuty info severity

This mapping ensures appropriate escalation behaviour based on alert importance.

Alert Resolution Handling

Server Scout automatically sends recovery notifications when issues resolve. Configure a separate webhook payload for recovery events:

{
  "routing_key": "YOUR_INTEGRATION_KEY_HERE",
  "event_action": "resolve",
  "dedup_key": "{{server_id}}-{{check_type}}"
}

The dedup_key ensures PagerDuty properly matches resolution events with their corresponding incidents.

Best Practices for Escalation Policies

Structuring Your Escalation Policy

1. Immediate Response (0 minutes): Notify primary on-call engineer via push notification and SMS
2. Secondary Escalation (15 minutes): Alert backup engineer and team lead
3. Management Escalation (30 minutes): Notify management for critical infrastructure issues

Policy Recommendations

• Create separate services for different server environments (production, staging, development)
• Use severity-based routing to handle info alerts differently from critical alerts
• Configure maintenance windows to suppress alerts during planned maintenance
• Implement alert fatigue protection by grouping related alerts and setting appropriate thresholds in Server Scout

Team Scheduling

• Ensure 24/7 coverage for critical production systems
• Rotate on-call responsibilities to prevent burnout
• Maintain clear handoff procedures between shifts
• Document common scenarios and response procedures

Testing Your Integration

Before relying on the integration:

1. Trigger a test alert from Server Scout
2. Verify the incident appears in PagerDuty with correct details
3. Confirm notifications reach the appropriate team members
4. Test alert resolution to ensure incidents close properly

This integration provides a robust foundation for incident response, ensuring your team stays informed about server issues whilst maintaining proper escalation procedures.