Data Retention and Metric Pruning

Server Scout employs a sophisticated tiered data retention system designed to balance storage efficiency with the need for detailed historical metrics. Understanding how this system works will help you make the most of your monitoring data whilst keeping storage requirements manageable.

How Data Retention Works

Server Scout stores metric data using a three-tier retention system that automatically manages data lifecycle based on age and relevance:

Tier 1: Full Resolution Data (0-24 hours)

Raw metric data is collected every 5 seconds and stored at full resolution for the first 24 hours. This high-frequency data capture is essential for detailed troubleshooting of recent issues, allowing you to spot brief spikes, intermittent problems, or rapid changes in system behaviour that might otherwise be missed.

Tier 2: Averaged Data (24 hours - 7 days)

After 24 hours, the raw 5-second data points are consolidated into 1-minute averages. This reduces storage requirements by a factor of 12 whilst preserving meaningful trend information. One-minute resolution remains perfectly adequate for understanding system patterns over the past week.

Tier 3: Data Deletion (7+ days)

Data older than 7 days is automatically deleted. For most monitoring scenarios, keeping detailed metrics beyond a week provides diminishing returns whilst consuming increasing storage space.

Automatic Pruning Process

Server Scout handles data lifecycle management automatically through a cron job that runs every hour. This pruning process:

1. Identifies metric data older than 24 hours and averages it into 1-minute intervals
2. Removes raw data points that have been successfully averaged
3. Deletes all metric data older than 7 days
4. Cleans up resolved alert state data
5. Prunes notification history older than 30 days

No manual intervention is required - the system maintains itself efficiently in the background.

Why This Tiered Approach Matters

The retention tiers strike an optimal balance between storage efficiency and diagnostic value:

Recent troubleshooting benefits enormously from 5-second resolution data. When investigating a performance issue that occurred yesterday, you need to see exactly when CPU spiked, how long the spike lasted, and whether it coincided with other system events. Averaged data might smooth over these crucial details.

Weekly trend analysis works perfectly well with 1-minute averages. Understanding whether your server is generally becoming busier, identifying daily usage patterns, or spotting gradual resource increases doesn't require 5-second precision.

Storage optimisation is critical for long-term sustainability. Without pruning, metric data would grow continuously, eventually consuming significant disk space and impacting system performance.

Practical Impact on Graphs

This retention system directly affects what you see in Server Scout's graphs:

• 1-hour graphs: Display full 5-second resolution data, showing every spike and dip
• 6-hour and 24-hour graphs: Mix full resolution recent data with averaged older data
• 7-day graphs: Show entirely averaged data, providing smooth trend lines ideal for pattern recognition

What Data Is Affected

The retention schedule applies uniformly across all Server Scout metrics:

• System metrics: CPU usage, memory consumption, disk I/O, network traffic
• Mount point data: Disk usage and available space for all monitored filesystems
• Plugin metrics: Custom metrics from any installed monitoring plugins

Notification and Alert Data

Notification history follows a separate 30-day retention period, giving you a full month to review alert patterns and system events. Alert state data for resolved issues is cleaned up during the regular pruning cycle to prevent unnecessary accumulation.

This automated approach ensures Server Scout remains lightweight and efficient whilst providing the detailed recent data and meaningful historical trends you need for effective server monitoring.