Monitoring Open File Descriptors

What Are File Descriptors?

File descriptors are handles that the Linux kernel uses to track open resources such as files, sockets, pipes, and directories. Every time a process opens a file, establishes a network connection, or creates a pipe, it consumes a file descriptor. This includes web server connections, database queries, log files, and even standard input/output streams.

Each system has limits on the number of file descriptors that can be open simultaneously, both per-process and system-wide. When these limits are exceeded, services begin failing with "too many open files" errors, and new connections are refused.

Enabling File Descriptor Monitoring

To monitor file descriptor usage with Server Scout, you'll need to enable the open file descriptor metric in your configuration:

1. Edit your Server Scout configuration file:

sudo nano /opt/scout-agent/scout.conf

1. Add or modify the following line:

open_fds=1

1. Restart the Server Scout service:

sudo systemctl restart serverscout

Once enabled, Server Scout will begin collecting file descriptor metrics and sending them to your dashboard.

Understanding the Metrics

Server Scout provides two key file descriptor metrics:

open_fds: The current count of open file descriptors across the entire system. This number fluctuates as processes open and close files, establish network connections, and perform I/O operations.

max_fds: The system-wide limit for open file descriptors. This is typically configured in /proc/sys/fs/file-max and represents the maximum number of file descriptors that can be open simultaneously.

The ratio between these two values is crucial for maintaining system stability. When openfds approaches maxfds, your system is at risk of file descriptor exhaustion.

Viewing File Descriptor Data

Once file descriptor monitoring is enabled, you can view the metrics in your Server Scout dashboard:

1. Navigate to your server's detail page
2. Locate the Load & Processes panel
3. You'll see both current open file descriptors and the system maximum

The display shows both absolute numbers and a percentage, making it easy to assess how close your system is to its limits.

Setting Up Alerts

To avoid service disruptions, configure alerts before file descriptor exhaustion occurs:

1. Go to your server's alert settings in the Server Scout dashboard
2. Create a new alert for the open_fds metric
3. Set the threshold to trigger when file descriptor usage exceeds 80-85% of the maximum
4. Configure notification methods (email, Slack, webhook, etc.)

This proactive approach gives you time to investigate and resolve issues before they impact your services.

Common Causes of High File Descriptor Usage

Several scenarios can lead to excessive file descriptor consumption:

Connection Leaks: Applications that open network connections but fail to close them properly. This is particularly common with HTTP clients, database connections, and API integrations.

Unclosed File Handles: Programs that open files for reading or writing but don't explicitly close them when finished. While garbage collection eventually cleans these up, it may not happen quickly enough under high load.

High-Traffic Web Servers: Legitimate high traffic can push file descriptor usage up, especially with web servers handling many concurrent connections. Each active connection typically requires at least one file descriptor.

Logging Issues: Applications writing to multiple log files simultaneously, or log rotation processes that temporarily keep old files open whilst creating new ones.

Troubleshooting High Usage

When file descriptor usage spikes, investigate using standard Linux tools:

# View current system-wide usage
cat /proc/sys/fs/file-nr

# Check per-process usage
lsof | awk '{print $2}' | sort | uniq -c | sort -nr | head -10

# View limits
ulimit -n

Regular monitoring with Server Scout helps you establish baseline usage patterns and quickly identify when something unusual occurs, allowing for prompt resolution before service disruptions occur.