The 70% Mystery: Socket Leaks That /proc/pid/fd Reveals Before ulimit Failures

A Python Flask application serving 400 concurrent users suddenly started throwing "too many open files" errors. The puzzling part? ulimit -n showed 65536 file descriptors available, but lsof counted only 45,000 in use - roughly 70% of the limit. Something was consuming file descriptors faster than the application could release them, but traditional monitoring showed nothing unusual.

The mystery deepened when restarting the application temporarily fixed the issue, only to see it resurface within hours. This wasn't a simple case of hitting system limits - it was a pattern that revealed itself only through careful analysis of the /proc/pid/fd directory structure.

The Hidden Socket Accumulation Pattern

Diving into /proc/PID/fd for the Flask application revealed an unexpected pattern. While most file descriptors pointed to expected resources (log files, database connections, network sockets), hundreds of entries showed identical socket patterns:

lrwx------ 1 www-data www-data 64 Feb 12 14:32 445 -> socket:[2847392]
lrwx------ 1 www-data www-data 64 Feb 12 14:32 446 -> socket:[2847394]
lrwx------ 1 www-data www-data 64 Feb 12 14:32 447 -> socket:[2847396]

These weren't active database connections. Running ss -tupln | grep :3306 showed far fewer active MySQL connections than the socket count in /proc/pid/fd suggested. The application was accumulating dormant sockets faster than the kernel could clean them up.

Cross-referencing with ss -tup state time-wait revealed the culprit: TIMEWAIT sockets from MySQL connections weren't being properly managed by the application's connection pool. Each web request created a new database connection, closed it immediately, but left the socket in TIMEWAIT state for the standard 60-second kernel timeout.

MySQL Connection Pool Leak Detection

The Flask application used SQLAlchemy with a connection pool configured for 20 persistent connections. However, a recent code change introduced a pattern where certain database queries bypassed the pool entirely, creating direct connections through mysql.connector.connect().

Tracing the pattern in /proc/pid/fd showed socket creation timestamps clustering around specific API endpoints. The /api/reports endpoint, called frequently by dashboard refreshes, was creating 3-4 new MySQL connections per request instead of reusing pooled connections.

This behaviour created a perfect storm: under normal load, the 60-second TIME_WAIT timeout was sufficient for socket cleanup. But during traffic spikes, new sockets accumulated faster than old ones expired. The application effectively leaked 200-300 file descriptors per minute during peak periods.

Building Proactive File Descriptor Monitoring

Standard monitoring focuses on percentage of ulimit usage, but this case demonstrated why that approach misses critical patterns. Socket accumulation problems become apparent long before hitting system limits if you monitor the right metrics.

The solution involved tracking three key indicators through /proc/pid/fd analysis: socket creation rate (new socket file descriptors per minute), TIME_WAIT accumulation patterns (matching socket states with fd counts), and connection pool bypass detection (comparing expected vs actual socket patterns).

A simple bash script parsing /proc/pid/fd and correlating with ss output provided 20 minutes of early warning before file descriptor exhaustion. This approach proved more reliable than application-level metrics, which often miss connection leaks that occur outside the monitored code paths.

The monitoring system we built (Server Scout's service monitoring) now tracks these patterns automatically, alerting when socket accumulation rates exceed sustainable levels rather than waiting for ulimit thresholds.

Prevention Through Pattern Recognition

Solving the immediate problem required fixing the code to use connection pooling consistently, but the broader lesson was about monitoring design. Traditional approaches wait for resource exhaustion to trigger alerts - by then, your application is already failing.

Effective file descriptor monitoring needs to understand application behaviour patterns. A web server should show predictable socket usage correlated with request patterns. Database applications should maintain relatively stable connection counts. Message queues should show socket creation and cleanup in regular cycles.

Deviations from these patterns, visible through /proc filesystem monitoring, provide early warning of resource leaks that percentage-based thresholds miss entirely. The key insight is monitoring rate of change rather than absolute values.

For production environments handling similar workloads, consider implementing custom scripts that parse /proc/pid/fd for your critical applications. The Linux kernel documentation provides detailed specifications for proc filesystem structures, enabling precise monitoring without performance overhead.

Socket leaks represent a class of problems where system limits aren't the real constraint - application behaviour is. By the time you hit 90% of ulimit, you've already experienced degraded performance for hours. Monitoring the patterns that lead to exhaustion provides the lead time necessary for proactive intervention.

FAQ