Reading Agent Logs with journalctl

When troubleshooting Server Scout issues or monitoring your agent's performance, the systemd journal is your primary source of information. The Scout agent logs all its activities through systemd, making journalctl the ideal tool for viewing and analysing agent behaviour.

Basic journalctl Commands

The most common command you'll use for monitoring Scout agent activity is:

journalctl -u scout-agent -f

This follows the logs in real-time, displaying new entries as they occur. It's particularly useful when testing configuration changes or investigating active issues.

To view recent log entries without following, use:

journalctl -u scout-agent --since "1 hour ago"

You can adjust the time period using formats like "2 hours ago", "yesterday", or specific dates like "2024-01-15 10:00:00".

For viewing the most recent entries, specify the number of lines:

journalctl -u scout-agent -n 100

This displays the last 100 log lines, which is often sufficient for quick troubleshooting.

Understanding Log Format

Server Scout agent logs follow a consistent format with timestamps in square brackets showing UTC time:

[14:32:15] Agent started successfully (version 2.1.4)
[14:32:15] Monitoring URL: https://app.serverscout.ie
[14:32:15] Collection interval: 60 seconds

Each log entry includes the UTC timestamp in [HH:MM:SS] format, followed by the message describing the agent's activity.

Common Log Messages

Startup Messages

When the agent starts, you'll see several initialisation messages:

• Version information: Shows the current agent version
• Monitoring URL: Confirms which Server Scout endpoint the agent connects to
• Collection interval: Displays how frequently data is collected (typically 60 seconds)
• Network interface: Shows which interface is being monitored for traffic statistics
• Process ID: The agent's PID for system reference
• Active plugins: Lists enabled monitoring plugins

Normal Operation

During regular operation, look for:

• Successful data submission: Confirms metrics are being sent to Server Scout
• Plugin execution: Shows individual plugins collecting data
• Auto-update checks: Periodic checks for agent updates

Network Issues

Connection problems generate specific log entries:

• HTTP status codes: Failed submissions show codes like 500, 502, or 503
• Spool file creation: When connectivity fails, the agent creates local spool files
• Spool replay: When connectivity returns, spooled data is transmitted

Troubleshooting with Logs

Repeated Connection Failures

If you see continuous connection errors with HTTP status codes like 500 or timeout messages, check:

1. Network connectivity to app.serverscout.ie
2. Firewall rules allowing outbound HTTPS traffic
3. DNS resolution for Server Scout's API endpoints

Authentication Errors

401 Unauthorised or 403 Forbidden errors indicate API key issues:

• Verify your API key is correctly configured
• Check if the key has expired in your Server Scout dashboard
• Ensure the key hasn't been regenerated without updating the agent

Growing Spool Depth

Messages about increasing spool file depth suggest persistent network connectivity issues. The agent stores data locally when it cannot reach Server Scout's servers, but this indicates an ongoing problem requiring attention.

Plugin Errors

Plugin-related errors often stem from:

• Missing dependencies: Required system packages or tools aren't installed
• Permission issues: The scout user lacks necessary permissions to read system information
• Configuration problems: Invalid plugin settings or missing configuration files

Best Practices

When investigating issues, start with recent logs using the --since parameter to focus on relevant timeframes. Use the -f flag when actively testing fixes to see immediate results.

For complex issues, consider redirecting logs to a file for analysis:

journalctl -u scout-agent --since "24 hours ago" > scout-logs.txt

Regular log monitoring helps identify patterns and prevent minor issues from becoming major problems. The Scout agent's detailed logging makes troubleshooting straightforward when you know what to look for.