Global vs Per-Server Alert Rules

Server Scout's alert system provides flexible rule management through global and per-server configurations, allowing you to create sensible defaults whilst maintaining fine-grained control over individual servers. Understanding how these rules interact is crucial for effective monitoring across diverse environments.

How Global and Per-Server Rules Work

Global Alert Rules

Global alert rules are conditions with no specific server_id (set to NULL in the database). These serve as default configurations that automatically apply to all servers in your monitoring setup. When you create a new server in Server Scout, it immediately inherits all global alert rules without requiring additional configuration.

Per-Server Alert Rules

Per-server rules are conditions assigned to a specific server_id. These rules override global settings for that particular server, giving you precise control over monitoring thresholds and behaviours on a server-by-server basis.

Alert Evaluation Engine

The alert evaluation engine follows a clear hierarchy when processing rules:

1. Check for per-server rules: The engine first looks for conditions specifically assigned to the server
2. Apply global rules: For any metrics without per-server rules, the engine applies the corresponding global rule
3. Override behaviour: When both global and per-server rules exist for the same metric, the per-server rule takes complete precedence

This merge process ensures that every server has appropriate monitoring coverage whilst allowing customisation where needed.

Practical Use Cases

Production vs Development Environments

A common scenario involves applying stricter monitoring thresholds to production servers whilst using more relaxed settings for development environments.

Global rules (suitable for development):

• CPU usage alert at 80%
• Memory usage alert at 85%
• Disk space alert at 90%

Production server overrides:

• CPU usage alert at 70%
• Memory usage alert at 75%
• Disk space alert at 80%

This approach ensures development servers don't generate excessive noise whilst production systems receive early warnings.

Disabling Specific Alerts

Some servers may require certain alerts to be disabled entirely. For example, a backup server might legitimately use high disk space during backup operations. You can create a per-server rule that disables disk space alerts for this specific server whilst maintaining the global rule for all others.

Resource-Specific Thresholds

Different server roles require different monitoring approaches:

• Database servers: Lower memory thresholds (databases typically use most available RAM)
• Web servers: Higher network traffic thresholds
• File servers: More generous disk space thresholds
• Build servers: Relaxed CPU usage limits during peak hours

Best Practices for Large Fleet Management

Start with Sensible Global Defaults

Begin by establishing global rules that work for the majority of your servers. These should be conservative enough to avoid false positives but sensitive enough to catch genuine issues. You can always create per-server overrides for special cases.

Use Server Groups Strategically

Whilst Server Scout doesn't have explicit server grouping, you can simulate this behaviour by:

1. Setting up global rules for your most common server type
2. Creating consistent per-server overrides for specific server classes
3. Using naming conventions to track which servers share similar alert profiles

Document Your Alert Strategy

Maintain documentation explaining:

• What your global rules represent (e.g., "suitable for standard web servers")
• Which servers have custom rules and why
• The rationale behind specific threshold choices

Regular Rule Auditing

Periodically review your alert configuration:

# Example: Review servers with custom alert rules
grep -r "server_id IS NOT NULL" journalctl -u scout-agent/

Implement Gradually

When rolling out new alert rules:

1. Test with a single server using per-server rules
2. Refine thresholds based on real-world data
3. Promote successful rules to global defaults
4. Add per-server overrides for servers that need different treatment

Managing Alert Noise

The global/per-server system helps reduce alert fatigue by allowing you to:

• Set conservative global defaults that avoid false positives
• Tighten thresholds only for critical systems that require immediate attention
• Disable problematic alerts for specific servers without affecting fleet-wide monitoring

This hierarchical approach ensures that your monitoring system scales effectively whilst maintaining the flexibility needed for diverse server environments.