Application Ghost Resources: Finding the File Descriptors and Memory Segments Your SIGTERM Handler Forgot

Your application stops cleanly. The logs show successful shutdown. systemd reports the service as inactive. Yet something lingers - connections that should be closed, memory segments that should be freed, file descriptors that should disappear.

This is the ghost resource problem. Applications implement SIGTERM handlers that appear comprehensive but miss critical cleanup paths. The process exits gracefully, but system resources remain allocated, causing subtle problems during restarts or gradual resource exhaustion.

The Silent Resource Leak Pattern in Graceful Shutdowns

Most applications handle SIGTERM by setting a flag that triggers an orderly shutdown sequence. They close database connections, flush buffers, and call cleanup functions. The process exits with status 0, and everyone assumes the job is done.

The reality is more complex. Child processes inherit file descriptors from their parents. Shared memory segments created with shmopen() persist until explicitly unlinked. Network sockets in TIMEWAIT state can accumulate. Each restart leaves behind a small collection of system resources that eventually add up to significant problems.

These leaks are particularly insidious because they don't trigger immediate failures. Your monitoring shows clean shutdowns and successful restarts. The application appears healthy. Only after weeks or months do you notice connection pool exhaustion, file descriptor limits being reached, or mysterious memory usage that doesn't correspond to any running process.

Investigating What Survives the SIGTERM Handler

The /proc filesystem provides forensic tools for tracking resources that outlive their parent processes. The key is capturing snapshots before and after shutdown to identify what persists.

Using /proc/pid/fd to Track File Descriptor Leaks

File descriptor analysis starts with a simple directory listing that reveals more than most application logs:

# Before shutdown - record all file descriptors
ls -la /proc/12345/fd/ > /tmp/fd_before.txt

# After graceful shutdown attempt
ls -la /proc/12345/fd/ > /tmp/fd_after.txt 2>/dev/null || echo "Process gone"

The /proc/pid/fd directory contains symbolic links to every file descriptor held by the process. Each link shows the target - regular files, sockets, pipes, or device nodes. When processes don't clean up properly, these descriptors can be inherited by child processes that outlive the parent.

Look for socket descriptors that point to network connections. These appear as links to socket:[12345] entries. Cross-reference these with /proc/net/tcp to see which connections remain in TIMEWAIT or CLOSEWAIT states after shutdown.

Pipes and FIFOs are another common leak. These show as pipe:[67890] entries. Applications that spawn child processes for background tasks often create pipes for communication but forget to close both ends during shutdown.

Mapping Shared Memory with /proc/pid/maps

Shared memory segments require explicit cleanup that many SIGTERM handlers overlook. The /proc/pid/maps file shows all memory mappings for a process, including shared memory regions that survive process termination.

# Check for shared memory regions
grep -E "/(shm/|SYSV|dev/shm)" /proc/12345/maps

Shared memory created with shmopen() appears as mappings in /dev/shm/. POSIX shared memory requires both munmap() to release the mapping and shmunlink() to remove the backing store. Many applications handle the first but forget the second, leaving memory segments allocated indefinitely.

System V shared memory shows up differently in the maps file and requires ipcs to fully investigate. Check for segments that persist after process termination with ipcs -m. These can accumulate over time, especially in applications that create unique shared memory keys for each instance.

Network Connection State Analysis

Network connections present a more complex cleanup challenge. TCP connections follow a specific state sequence during termination, and problems can occur at any step. The /proc/net/tcp file shows connection states that reveal improper shutdown handling.

Connections stuck in CLOSE_WAIT indicate the application received a FIN from the remote end but never sent its own FIN response. This usually means the application didn't properly close its end of the socket during shutdown.

TIMEWAIT connections are normal and expected, but excessive numbers suggest connection churning or improper connection pooling. Applications that create many short-lived connections instead of reusing pooled connections will accumulate TIMEWAIT entries.

UNIX domain sockets also require attention. These appear in /proc/net/unix and can leak if the application doesn't unlink the socket file during shutdown. Check for socket files in /tmp or /var/run that persist after process termination.

Common SIGTERM Handler Implementation Gaps

The most effective service monitoring catches these patterns before they cause production problems. Understanding where SIGTERM handlers typically fail helps build better cleanup procedures.

Race Conditions in Cleanup Sequences

Many applications attempt cleanup in a specific order but don't account for timing dependencies. A common pattern is closing database connections before ensuring all worker threads have finished their current operations. This can leave transactions in an inconsistent state or connections in an indeterminate state.

The solution involves careful sequencing with proper synchronisation. Set shutdown flags first, then wait for worker threads to acknowledge the shutdown signal before beginning resource cleanup. Use timeouts to prevent indefinite waits, but make them long enough for normal operations to complete.

Signal handlers themselves have limitations. They can't safely call most library functions due to signal safety restrictions. Complex cleanup operations need to be triggered by the signal handler but executed in the main thread context. This requires coordination mechanisms that many implementations get wrong.

Child Process Resource Inheritance

Child processes inherit file descriptors from their parents unless explicitly prevented. Applications that spawn background processes often forget to close unused file descriptors in the child, leading to resource leaks that survive the parent's termination.

The FD_CLOEXEC flag prevents file descriptors from being inherited across exec() calls, but many applications spawn children with fork() without exec(). In these cases, explicit cleanup in the child process is required.

Shared memory segments and System V IPC objects also persist across process boundaries. Child processes that exit without proper cleanup leave these resources allocated. This is particularly problematic in applications that crash or are killed with SIGKILL after SIGTERM fails.

Building Bulletproof Shutdown Procedures

Effective shutdown procedures require systematic resource tracking and cleanup verification. Start by inventorying all resources your application creates - file descriptors, memory mappings, child processes, and IPC objects.

Implement cleanup in reverse order of allocation. Resources created last should be cleaned up first. This prevents dependencies that could cause cleanup operations to fail or hang.

Use verification steps during shutdown. After closing file descriptors, verify they're actually closed by checking /proc/self/fd. After unmapping shared memory, confirm the mappings are gone from /proc/self/maps. These checks catch cleanup failures before they become persistent leaks.

Set strict timeouts for all cleanup operations. systemd gives processes 90 seconds to respond to SIGTERM before sending SIGKILL. Use this time budget wisely by setting shorter timeouts for individual cleanup steps. If any step takes too long, skip remaining cleanup and exit - it's better to have a known leak than an unkillable process.

Monitoring Resource Cleanup Success

Building automated monitoring for resource cleanup involves scripting the manual investigation techniques described above. Create monitoring scripts that capture resource snapshots before shutdown and verify complete cleanup afterward.

Track file descriptor counts over time to detect gradual leaks. Monitor shared memory usage with ipcs output parsing. Watch for accumulating socket files in temporary directories. These metrics reveal cleanup failures that don't show up in application logs.

The debugging PostgreSQL connection pool exhaustion investigation shows similar principles applied to database connections. Resource leaks follow predictable patterns regardless of the specific resource type.

Set up alerts for resource count increases that don't correspond to increased application activity. A slowly climbing file descriptor count or growing shared memory usage indicates systematic cleanup failures that need immediate attention.

Testing shutdown procedures in isolation helps catch problems before they affect production. Create test harnesses that spawn your application, exercise its resource allocation patterns, then verify complete cleanup after SIGTERM. This testing should be part of your continuous integration pipeline, not a manual process performed during crises.

FAQ