TCP Handshake Ratio Analysis: Detecting SYN Floods That fail2ban Rate Limits Never See

A hosting company with 200+ servers discovered their fail2ban logs showed normal activity during a 20-minute period when customer sites became completely unreachable. The culprit wasn't a brute-force attack triggering traditional rate limits, but a sophisticated SYN flood that exploited the mathematical blind spots in connection-based monitoring.

This scenario highlights why modern DDoS attacks require behavioural analysis rather than signature detection. The answer lies in TCP connection state mathematics through /proc/net/tcp analysis.

Why Traditional Rate Limiting Fails Against Modern DDoS Patterns

fail2ban monitors authentication failures and connection attempts per IP address within specific time windows. However, volumetric attacks bypass these thresholds by distributing requests across thousands of source addresses, each staying below your configured limits.

The mathematical reality: if you've set fail2ban to trigger after 10 connections per minute, an attacker using 500 source IPs can generate 5,000 connection attempts whilst each individual address appears completely legitimate.

The SYN Flood Blind Spot

SYN floods consume server resources by initiating TCP handshakes without completing them. Your server allocates memory for half-open connections in the SYN_RECV state, but traditional monitoring only counts completed connections or failed authentication attempts.

A healthy server typically maintains 85-95% TCP handshake completion rates. During a SYN flood, this ratio drops to 30-60% as thousands of incomplete handshakes accumulate in your connection table.

Connection State Distribution Anomalies

Normal traffic patterns show predictable connection state distributions: mostly ESTABLISHED connections, minimal SYN_RECV states, and consistent ratios between inbound and outbound connections. Volumetric attacks create statistical anomalies that stand out when you analyse connection state mathematics rather than individual IP behaviour.

Reading /proc/net/tcp for Security Intelligence

The /proc/net/tcp file contains real-time connection state data in a format that enables mathematical analysis of traffic patterns:

sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt
0: 0100007F:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000
1: 0100007F:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000

Decoding Connection State Values

The st column contains hexadecimal connection states that reveal traffic behaviour patterns:

• 01 = ESTABLISHED (normal active connections)
• 02 = SYN_SENT (outbound connection attempts)
• 03 = SYN_RECV (incomplete inbound handshakes)
• 08 = CLOSE_WAIT (connections awaiting cleanup)
• 0A = LISTEN (services accepting connections)

During volumetric attacks, the ratio of SYN_RECV states increases dramatically compared to ESTABLISHED connections.

Calculating TCP Handshake Success Ratios

Handshake completion analysis requires counting connection states and calculating ratios:

• Completion ratio = ESTABLISHED / (ESTABLISHED + SYN_RECV)
• Flood indicator = SYN_RECV / Total active connections
• State distribution variance = Standard deviation of state counts over time

Healthy servers maintain completion ratios above 85%. Ratios below 60% indicate potential SYN flooding, while ratios below 30% confirm active attacks.

Building Scalable Detection Scripts

Connection state monitoring through /proc/net/tcp requires minimal CPU overhead compared to packet capture approaches. A bash script can parse connection states and calculate ratios every 30 seconds without impacting server performance.

The key is establishing baseline ratios during normal traffic periods, then alerting when current measurements deviate significantly from historical patterns. This mathematical approach adapts to your specific traffic patterns rather than relying on fixed thresholds.

Monitoring Connection State Thresholds

Effective detection combines multiple mathematical indicators:

1. Absolute SYN_RECV count - Alert when exceeding 500 half-open connections
2. Completion ratio degradation - Trigger when ratios drop 40% below baseline
3. State distribution entropy - Detect unusual patterns in connection state variety
4. Time-based trend analysis - Identify rapid changes rather than gradual shifts

Alert Logic That Adapts to Traffic Volume

Static thresholds fail during legitimate traffic spikes. Adaptive alerting uses percentage-based calculations that scale with your normal traffic volume.

For example, instead of alerting on "more than 100 SYNRECV connections," monitor for "SYNRECV states exceeding 15% of total connections when baseline is typically 3%." This approach prevents false positives during busy periods whilst maintaining sensitivity to attacks.

Implementation and Performance Considerations

Parsing /proc/net/tcp every 30 seconds typically consumes less than 1MB RAM and negligible CPU time. The architecture decisions behind lightweight monitoring show how bash-based analysis outperforms heavyweight monitoring solutions for this type of statistical work.

The mathematical approach scales efficiently because connection state analysis complexity remains constant regardless of traffic volume. Whether you're processing 100 or 100,000 connections, the calculation overhead stays minimal.

Server Scout's agent includes connection state ratio monitoring as part of its comprehensive network analysis capabilities, providing automated baseline calculation and adaptive threshold alerting without the resource overhead of traditional DDoS detection tools.

Building scalable monitoring solutions often requires choosing mathematical analysis over signature-based detection to achieve both accuracy and performance at scale.

For production implementation, consider storing historical baseline data and implementing gradual threshold adjustment based on legitimate traffic pattern evolution. This prevents alert fatigue whilst maintaining security sensitivity.

FAQ

Visit serverscout.ie to learn more about kernel documentation at kernel.org for complete /proc/net/tcp format specifications.