Process and Zombie Monitoring

Understanding Process Monitoring in Server Scout

Server Scout provides comprehensive process monitoring capabilities to help you keep track of system performance and identify potential issues before they impact your infrastructure. The monitoring agent automatically collects process-related metrics that give you valuable insights into your server's health.

How Server Scout Detects Processes

The Server Scout agent uses the ps command to gather process information during each monitoring cycle. This lightweight approach ensures minimal system overhead whilst providing essential data about running processes and system state.

Two key metrics are collected:

• zombie_count: Tracks the number of zombie processes on your system
• top_processes: Identifies the most resource-intensive processes

What Are Zombie Processes?

Zombie processes, also known as defunct processes, are child processes that have completed execution but still have entries in the process table. When a child process terminates, it becomes a zombie until its parent process reads its exit status using system calls like wait() or waitpid().

Under normal circumstances, zombies exist only briefly. However, if a parent process fails to reap its children (read their exit status), these zombie processes accumulate in the process table. Whilst zombies consume minimal resources—they only retain their process ID, exit status, and usage statistics—large numbers can exhaust the system's process table, potentially preventing new processes from starting.

Zombie Process Detection

Server Scout's zombie_count metric specifically monitors for these defunct processes by parsing the output of ps and counting entries with a status of 'Z' (zombie). This metric appears on your server's detail page, allowing you to track zombie accumulation over time.

A consistently high zombie count often indicates:

• Poorly written applications that don't properly handle child processes
• Parent processes that have crashed or are unresponsive
• System-level issues affecting process management

Top Processes Monitoring

The top_processes metric provides detailed information about your system's most resource-intensive processes. Server Scout collects:

• Top 3 CPU-consuming processes: Process ID, name, and CPU percentage
• Top 3 memory-consuming processes: Process ID, name, and memory percentage

This information helps you quickly identify processes that might be causing performance issues or consuming excessive resources.

Viewing Process Information

Process metrics are displayed prominently on each server's detail page within your Server Scout dashboard. You'll find:

1. Current zombie process count with historical trends
2. Real-time listings of top CPU and memory consumers
3. Process details including PIDs for easy identification

The dashboard updates regularly, providing near real-time visibility into your server's process landscape.

Investigating and Cleaning Up Zombie Processes

When Server Scout alerts you to zombie processes, follow these steps to investigate and resolve the issue:

1. Identify Zombie Processes

ps aux | grep -w Z

Or for more detailed information:

ps -eo pid,ppid,state,comm | grep -w Z

2. Find the Parent Process

ps -o pid,ppid,state,comm -p <zombie_pid>

3. Investigate the Parent

Examine the parent process to determine why it's not reaping its children:

ps aux | grep <parent_pid>

4. Clean Up Zombies

Restart the parent process (if safe to do so):

sudo systemctl restart <service_name>

Kill and restart the parent process:

sudo kill <parent_pid>

For persistent issues, consider:

• Checking application logs for errors
• Reviewing code for proper signal handling
• Implementing proper cleanup procedures in parent processes

Best Practices

• Monitor zombie trends rather than reacting to single occurrences
• Investigate applications that consistently create zombies
• Ensure proper signal handling in custom applications
• Use Server Scout's alerting features to notify you of sustained zombie accumulation

Regular monitoring of these process metrics through Server Scout helps maintain optimal server performance and prevents resource exhaustion issues before they impact your services.