Production Redis clusters fail in ways that traditional monitoring misses. Connection pooling masks node failures, authentication errors prevent query-based health checks, and by the time client applications report issues, your cluster has been degraded for minutes.
Socket-level monitoring through /proc/net/tcp reveals Redis cluster health without making a single connection to your Redis instances. This approach works even when Redis processes hang, authentication fails, or network partitions prevent traditional monitoring.
Understanding Redis Cluster Socket Patterns
Redis clusters create predictable socket patterns that change during failover events. Each Redis instance maintains connections on both the primary port (typically 6379) and the cluster bus port (primary port + 10000).
Normal Operation Connection States
During healthy operation, Redis master nodes show specific socket characteristics. Masters maintain more ESTABLISHED connections to slaves and clients, whilst the cluster bus port shows consistent peer-to-peer communication patterns.
Slaves exhibit different socket signatures. They maintain fewer direct client connections but show regular communication patterns to their master through both the data port and cluster bus.
The /proc/net/tcp file uses hexadecimal notation for addresses and ports. A Redis instance running on 127.0.0.1:6379 appears as 0100007F:18EB (IP bytes reversed, port in hex).
Master-Slave Socket Signatures
Master nodes typically maintain 2-3 connections per slave in the cluster plus client connections. The cluster bus port shows connections to all cluster members, creating a predictable baseline pattern.
Slave nodes show a more constrained pattern: one persistent connection to their master for replication, cluster bus connections to all nodes, and varying numbers of client connections depending on read traffic distribution.
Parsing /proc/net/tcp for Redis Monitoring
The monitoring script needs to convert hexadecimal addresses back to readable format and track connection states over time. Focus on ESTABLISHED, TIMEWAIT, and SYNSENT states as they indicate different cluster health conditions.
TCP State Analysis Script
Create a parser that extracts Redis-specific connections from /proc/net/tcp. Filter for your Redis ports and cluster bus ports, then group connections by state and destination.
awk '$2 ~ /:18EB$|:4E40$/ { print $2, $3, $4 }' /proc/net/tcp |
while read local_addr remote_addr state; do
# Convert hex addresses and track connection patterns
echo "Port $(printf "%d" 0x${local_addr#*:}) State $state"
doneTrack connection counts per port over time. Sudden drops in ESTABLISHED connections to the cluster bus port often indicate node failures before application-level health checks detect issues.
Slot Migration Detection
During slot migrations, Redis cluster topology changes create temporary socket pattern variations. Monitor for increased TIME_WAIT states on cluster bus ports, which indicate rapid connection cycling as nodes redistribute slot ownership.
Connection count changes between cluster members often precede visible slot migration events. A 30% increase in cluster bus connection cycling typically indicates migration activity.
Building Failover Detection Logic
Failover events create distinctive socket state transitions. Masters that fail show immediate drops in client connections followed by cluster bus connection changes as remaining nodes elect a new master.
Connection Pattern Changes
Master failures follow predictable patterns: client connections drop first (within 1-2 seconds), then cluster bus connections show rapid state changes as the cluster detects the failure and initiates election.
Slave promotions create inverse patterns. A slave being promoted to master shows rapid increases in ESTABLISHED connections as clients reconnect and other slaves establish replication connections.
Monitor the ratio of ESTABLISHED to TIME_WAIT connections per node. Ratios above 1:0.3 typically indicate healthy operation, whilst ratios below 1:0.8 suggest connection instability.
Automated Alert Triggers
Set thresholds based on your cluster's normal connection patterns. Alert when cluster bus connections drop by more than 25% within a 30-second window, or when client connection counts show sudden 50% reductions.
False positives occur during planned maintenance or high client connection churn. Implement time-of-day baselines and require sustained threshold breaches (60+ seconds) before triggering alerts.
Production Implementation
Deploy the monitoring script to run every 10 seconds via systemd timer. Store connection state history in simple flat files rather than databases to avoid dependencies that might fail during Redis issues.
Performance Considerations
Parsing /proc/net/tcp on busy systems requires optimisation. Use awk rather than shell loops for hex conversion. Filter for specific ports early in the pipeline to reduce processing overhead.
On systems with thousands of connections, focus monitoring on cluster bus ports only. These show cluster health changes without the noise of client connection variation.
Integration with Existing Monitoring
This socket-based approach complements rather than replaces query-based Redis monitoring. Building backup verification tests using similar zero-dependency techniques ensures your monitoring remains functional when primary systems fail.
Socket analysis works alongside traditional Redis monitoring. Use it as an early warning system that alerts before query-based monitoring fails. Many teams find socket patterns change 60-90 seconds before Redis client libraries detect problems.
Server Scout's plugin system supports custom Redis monitoring scripts using this approach. The pure bash implementation avoids dependencies that might conflict with Redis installations whilst providing the socket-level analysis needed for early failure detection.
This monitoring approach particularly benefits hosting companies managing multiple Redis clusters. Socket-level detection scales across hundreds of instances without the authentication complexity or query overhead of traditional monitoring.
For teams already using intrusion detection systems, geographic attack clustering patterns can be adapted to identify Redis cluster communication anomalies that suggest security issues rather than just performance problems.
Socket state monitoring reveals Redis cluster health through the Linux kernel's networking layer rather than Redis-specific interfaces. This provides monitoring that continues working even when Redis processes become unresponsive or authentication systems fail.
The kernel's socket state information updates immediately when connections fail, making this approach faster than health check timeouts. Combined with Server Scout's email notifications, teams receive failover alerts before applications experience connection errors.
Production Redis environments benefit from monitoring strategies that don't depend on the systems they're watching. Socket-level analysis provides this independence whilst revealing cluster topology changes that query-based monitoring often misses.
FAQ
How often should I parse /proc/net/tcp for Redis monitoring?
Every 10-15 seconds provides good balance between responsiveness and system overhead. More frequent parsing rarely improves detection time but increases CPU usage on busy systems.
Can this approach detect Redis authentication failures?
Yes, authentication failures show as rapid SYNSENT to TIMEWAIT transitions as clients fail to complete the Redis AUTH handshake, creating distinctive socket patterns different from network-level connection failures.
What about Redis Sentinel monitoring?
Sentinel instances create their own socket patterns on port 26379. Apply the same techniques to monitor Sentinel health, looking for loss of connections between Sentinel instances as an indicator of monitoring infrastructure problems.