Day One: Safe Reconnaissance (Hours 1-24)
Last Tuesday, Sarah stared at an email that made her stomach drop. Three production servers, no documentation, previous admin unreachable, and 400 customers depending on systems she'd never seen before. Sound familiar?
Taking over undocumented infrastructure feels like being handed car keys in a foreign country without a map. You know something important is running, but touching the wrong thing could bring everything down. The secret isn't moving fast - it's building confidence through systematic, non-invasive discovery.
Identifying What's Running Without Disruption
Start with the safest reconnaissance possible. Your first command should be ss -tuln to see what services are listening without affecting anything. This shows you the network footprint without generating logs that might worry existing monitoring systems.
Next, check what's actually consuming resources with top and ps aux. Don't restart anything yet - you're gathering intelligence, not optimising. Look for unfamiliar process names and write them down. That mysterious legacy_batch_processor consuming 40% CPU might be critical to payroll processing.
Examine /etc/systemd/system and /etc/init.d to understand what services are supposed to start automatically. The enabled services tell you what the previous admin considered essential, even if they forgot to document it.
Mapping Network Dependencies
Use lsof -i to correlate network connections with specific processes. This reveals the application stack's communication patterns. When you see MySQL connections from three different processes, you've found your database tier. When port 443 connections cluster around nginx worker processes, you've mapped your web frontend.
Check /etc/hosts and DNS resolution patterns in /var/log/messages. Legacy systems often rely on hardcoded hostnames that reveal integration points. That reference to old-payment-gateway.internal might explain the mysterious cron job you spotted earlier.
Document every listening port and its corresponding service. This network map becomes your safety net for understanding what depends on what.
Initial Service Inventory
Build a simple spreadsheet with columns for service name, purpose (even if guessed), resource usage, and network dependencies. When you see postfix running alongside a web application, note "probably handles contact form emails" in the purpose column. When you find Redis consuming 2GB RAM, mark it as "likely session storage or caching".
Don't worry about being wrong initially - you're building hypotheses to test safely later.
Day Two: Deeper Analysis (Hours 25-48)
Application Stack Discovery
Now start examining configuration files, but read-only. Check /etc/nginx/sites-enabled or /etc/apache2/sites-enabled to understand web application routing. The virtual host configurations reveal which domains point where and often contain comments from previous administrators.
Look through /etc/cron.d and user crontabs with crontab -l -u username for each user. Scheduled tasks reveal business logic that isn't visible in running processes. That 3am database backup script explains why disk usage spikes every night.
Examine log files in /var/log to understand normal system behaviour. Recent error patterns show you what's already broken but still limping along. Application logs often contain stack traces that reveal technology choices and integration points.
Data Flow Mapping
Trace data paths by following configuration chains. If nginx proxies to port 8080, check what's listening there. If that's a Java application, examine its configuration for database connections, external API calls, or file system dependencies.
Check database connection strings in application configs. The connection parameters reveal whether you're dealing with local databases, remote clusters, or third-party services. Note authentication methods - hardcoded passwords suggest you'll need to audit security practices later.
Document file system dependencies by examining recent file access patterns. Use find /opt -type f -mtime -7 to see what application files were modified recently. This reveals active codebases versus legacy directories.
Security Posture Assessment
Run ss -tuln again and compare listening services to public port scans. Anything listening on 0.0.0.0 that shouldn't be publicly accessible needs firewall attention. Check iptables -L or ufw status to understand current firewall rules.
Examine SSH configuration in /etc/ssh/sshd_config. Look for non-standard ports, key-only authentication, or user restrictions. Review /var/log/auth.log for recent login patterns and failed authentication attempts.
Check for obvious security issues like world-writable directories (find / -type d -perm 777 2>/dev/null) or SUID binaries (find / -type f -perm -4000 2>/dev/null), but don't fix anything yet.
Day Three: Documentation and Validation (Hours 49-72)
Creating Your Infrastructure Map
Consolidate your discoveries into a clear diagram showing data flow between services. Use simple boxes and arrows - this isn't architecture documentation, it's an operational map. Show which services depend on which others and note critical resource requirements.
Create a service restart order based on dependencies. Database services start before applications, load balancers start after backend services. This sequence becomes crucial when you eventually need to perform maintenance.
Document your confidence level for each component. Mark services you understand well as "verified", partially understood ones as "probable", and complete mysteries as "unknown". This guides your future investigation priorities.
Testing Your Understanding Safely
Validate your service inventory by checking status without restarting anything. Use systemctl status servicename to confirm your understanding of what each service does. Compare process arguments with configuration files to verify your assumptions.
Test your network map by temporarily blocking non-critical connections and observing the results. If blocking port 6379 (Redis) causes web application errors, you've confirmed the caching dependency.
Perform read-only database queries to understand data structures and relationships. This validates your application stack assumptions and reveals business logic you might have missed.
Building Monitoring Coverage
Now that you understand the infrastructure, you can deploy proper monitoring. This is where a lightweight agent becomes invaluable - you can monitor all your newly discovered servers without adding complexity to systems you're still learning.
Start with basic system metrics: CPU, memory, disk space, and load averages. These provide the foundation for understanding normal behaviour patterns. Add service-specific monitoring as your confidence grows.
Establish baseline thresholds conservatively. Set disk space alerts at 90% rather than 80% until you understand normal usage patterns. Configure smart alerting with longer sustain periods to avoid false alarms while you're still learning the system's personality.
Set up email notifications immediately so you'll know if your exploration activities accidentally affect production services. Better to get an alert about a service restart than discover an outage from angry customers.
The goal isn't comprehensive monitoring on day three - it's establishing basic visibility so you can learn safely while protecting the business from unknown system failures.
By the end of 72 hours, you'll have transformed from inheriting mystery servers to managing known infrastructure. You'll understand what's running, why it's important, and how to monitor it properly. Most importantly, you'll have built this knowledge without causing outages or compromising system stability.
FAQ
What if I accidentally break something during the discovery process?
Stick to read-only commands for the first 48 hours. Use ss instead of netstat, examine logs with less rather than tail -f, and never restart services until you understand their dependencies. If you must test something, do it during scheduled maintenance windows with stakeholder awareness.
How do I prioritise which systems to investigate first when managing multiple servers?
Start with the servers showing the highest resource utilisation in top output, then move to systems with the most network connections from ss -tuln. These typically indicate primary application servers or databases that support multiple services.
What documentation format works best for handoff to the next admin?
Create a simple text file with three sections: "Services and their purposes", "Known dependencies", and "Things I still don't understand". Include specific commands that reveal key information and note any quirks you've discovered. For comprehensive monitoring setup documentation, the Server Scout knowledge base provides detailed guidance on documenting monitoring configurations properly.