3-Hour Head Start: How TCP Connection Analysis Exposed Lateral Movement Before SIEM Alerts

Last Tuesday at 14:30, the first anomalous connection appeared in /proc/net/tcp on a customer-facing web server. Three hours later, the SIEM finally triggered its first lateral movement alert. By then, the attackers had compromised four additional systems and exfiltrated 2.3GB of customer data.

This incident demonstrates how fundamental Linux networking data can provide earlier warning of advanced persistent threats than expensive security appliances. More importantly, it shows the manual analysis techniques that work when you don't have enterprise detection tools.

The Initial Compromise Signal

The attack started with a successful credential stuffing attempt against an exposed SSH service. Standard authentication logs showed nothing unusual - just another failed login among thousands. But the TCP connection patterns told a different story.

At 14:30, /proc/net/tcp revealed an ESTABLISHED connection from the compromised web server to an internal database host on port 5432. This connection persisted for 847 seconds - far longer than typical application database queries that complete in milliseconds.

cat /proc/net/tcp | grep "0101A8C0:1542 0201A8C0:15B3 01"

The hex encoding shows the local address (192.168.1.1:5442) connecting to 192.168.1.2:5555, with connection state 01 (ESTABLISHED). Normal PostgreSQL connections from this web server typically lasted under 2 seconds. This 847-second persistence indicated interactive access rather than application queries.

Baseline TCP Connection Patterns

Effective lateral movement detection requires understanding normal connection behaviour. Our web servers typically maintained:

• 12-15 ESTABLISHED connections to the load balancer (port 80/443)
• 2-3 short-lived database connections per minute (lasting 0.1-2.0 seconds)
• 1 persistent connection to the monitoring system
• 4-6 outbound HTTPS connections for API calls

The anomalous 847-second database connection immediately stood out against this baseline. More concerning was the timing - it began exactly 23 seconds after the suspicious SSH authentication, suggesting manual database enumeration rather than automated application logic.

Parsing /proc/net/tcp for Anomalous Connections

The /proc/net/tcp format requires hex decoding, but provides timestamp precision that netstat lacks. Each line contains the connection tuple, state, and kernel timestamps critical for timeline reconstruction.

Connection state analysis revealed the attack progression. State 01 (ESTABLISHED) connections persisted between systems that normally never communicated directly. State 02 (SYN_SENT) attempts appeared on unusual ports, indicating reconnaissance scanning.

Connection State Analysis Methodology

We built a simple parser to track connection anomalies:

while true; do
  awk '{print $2, $3, $4}' /proc/net/tcp | sort | uniq -c
  sleep 30
done

This revealed connection patterns invisible to application logs. Normal business hours showed consistent connection counts to expected destinations. The lateral movement introduced new destination addresses and unusual port combinations.

Identifying Pivot Points Through Port Patterns

The attackers followed predictable lateral movement patterns. After establishing database access, they attempted connections to:

• Port 445 (SMB) across the subnet range
• Port 3389 (RDP) to Windows systems
• Port 5985 (WinRM) for PowerShell remoting
• Port 22 (SSH) to additional Linux hosts

These port combinations don't appear in normal application traffic. Our web servers had no business reasons to initiate SMB or RDP connections. The TCP analysis revealed this reconnaissance activity 2 hours and 17 minutes before the SIEM detected unusual authentication patterns.

Timeline Reconstruction from Network Data

The /proc/net/tcp timestamps enabled precise timeline reconstruction. The attack progression became clear:

14:30:00 - Initial SSH compromise 14:30:23 - First anomalous database connection (847-second duration) 15:15:41 - SMB reconnaissance across subnet 15:47:12 - Successful lateral movement to file server 16:22:08 - RDP connection to domain controller 17:30:00 - SIEM finally triggers first lateral movement alert

This 3-hour detection gap allowed extensive data access. The TCP connection analysis provided forensic evidence that SIEM logs couldn't match. More importantly, it demonstrated detection techniques available on every Linux system without additional software.

Cross-System Connection Correlation

By collecting /proc/net/tcp data across multiple servers, we correlated the attack path. The same source IP appeared in connection logs on four different systems, but at different times corresponding to the lateral movement timeline.

This cross-system correlation revealed the attack scope before log aggregation systems processed the events. Server Scout's unified dashboard now provides this correlation automatically, but the manual analysis techniques remain valuable for incident response.

Response Actions Based on TCP Analysis

The TCP connection data guided immediate containment actions. We identified compromised systems by their anomalous connection patterns, not by waiting for behavioral analysis or file integrity checks.

Network-level containment proved more effective than host-based responses. Blocking the attacking IP addresses at network firewalls immediately severed all lateral movement connections, preventing further system compromise.

Containment Strategy Without EDR Tools

Without enterprise endpoint detection tools, we relied on network-level indicators:

1. Immediate firewall rules blocking source IPs with anomalous connection patterns
2. SSH key rotation on all systems showing suspicious connection activity
3. Database credential changes based on connection timing analysis
4. Network segmentation to prevent further lateral movement

This network-focused approach proved faster than host-based incident response. The TCP connection data provided actionable intelligence within minutes of analysis.

Lessons Learned and Detection Improvements

This incident exposed gaps in our security monitoring strategy. Traditional security tools focused on log analysis and signature detection, missing the network-level indicators visible through system data.

We implemented continuous /proc/net/tcp monitoring as part of our standard server monitoring approach. Understanding connection baselines became critical for detecting anomalous network activity.

The most valuable lesson: expensive security tools can't replace understanding fundamental system behaviour. The Linux networking stack provides rich forensic data that many organisations never examine. Manual analysis skills remain essential when automated tools fail or aren't available.

Building detection capabilities into existing monitoring infrastructure proved more sustainable than deploying additional security appliances. Unified infrastructure monitoring that includes network connection analysis provides security benefits without tool sprawl.

Network-level detection often provides earlier warning than application-level monitoring. Understanding TCP connection patterns gives incident responders a critical head start in containing advanced threats.

FAQ