Last Tuesday, a three-person hosting company discovered that their control panel installation had been quietly modified over two weeks. No alerts, no notifications, just slowly escalating customer complaints about odd behaviour. The attackers had persistence in /etc/apache2/conf.d/ and were harvesting customer credentials.
File integrity monitoring feels like enterprise-grade security complexity, but Linux's built-in inotify system makes it surprisingly approachable for small teams. You don't need complex commercial tools to detect when someone changes your critical system files.
Understanding File Integrity Monitoring for Small Teams
File integrity monitoring tracks changes to important files and directories on your systems. When someone modifies /etc/passwd, creates new files in /var/www/html/, or changes configuration in /etc/ssh/sshd_config, your monitoring should notice immediately.
Small teams need monitoring that works without dedicated security specialists. This means focusing on the directories that matter most and building alerts that actually get attention without drowning your operations staff.
The inotify system built into Linux kernels watches filesystem events in real-time. Unlike scheduled checksum comparisons that run every few minutes or hours, inotify catches changes as they happen.
Setting Up inotify for Security Detection
Linux distributions include inotify in the kernel, but you'll need the userspace tools to build monitoring scripts.
Installing and Configuring inotify-tools
Install the inotify-tools package through your distribution's package manager. On Ubuntu and Debian systems, run apt install inotify-tools. RHEL and CentOS systems use yum install inotify-tools or dnf install inotify-tools.
The key command is inotifywait, which monitors files and directories for specific events. The most useful events for security monitoring are INMODIFY (file contents changed), INCREATE (new file created), INDELETE (file deleted), and INMOVED_TO (file moved into watched directory).
Before building production monitoring, check your system's watch limits. The kernel parameter /proc/sys/fs/inotify/maxuserwatches defaults to 8192 watches per user. If you're monitoring many directories recursively, you might need to increase this limit through /etc/sysctl.conf.
Creating Your First File Monitor Script
Start with a simple monitoring script that watches a single critical directory. Create a file called watch-etc.sh with basic inotifywait functionality.
Your script should specify the directories to monitor, the events to catch, and the format for logging detected changes. Include timestamps, the specific file affected, and the type of change detected.
For production use, wrap your inotifywait command in a while loop so monitoring continues after each detected event. Without this, the script exits after the first change.
Monitoring Critical System Directories
Not every directory requires monitoring. Focus on locations where unauthorised changes indicate security problems or system compromise.
Essential Directories to Watch
The /etc/ directory contains system configuration that attackers often modify. Monitor /etc/passwd, /etc/shadow, /etc/sudoers, and the entire /etc/ssh/ directory for authentication changes.
Web server directories like /var/www/html/ and /var/www/vhosts/ catch web shell uploads and site defacements. Include both document roots and configuration directories like /etc/apache2/ or /etc/nginx/.
System service directories need watching too. Monitor /etc/systemd/system/ for unauthorised service installations and /etc/cron.d/ for malicious scheduled tasks.
Binary directories like /usr/bin/, /usr/sbin/, and /usr/local/bin/ rarely change during normal operations. Changes here often indicate rootkit installation or system compromise.
Handling Alert Notifications
When your monitoring detects changes, the alert needs to reach someone who can investigate. Build notification into your monitoring script using email, webhook calls, or integration with your existing alerting infrastructure.
Include enough context in alerts that your team can respond appropriately. Log the username who made the change (if available through who or w commands), the specific file affected, and the type of modification detected.
Not every file change represents a security incident. System package updates legitimately modify files in /etc/ and /usr/bin/. Build your team's understanding of normal change patterns so they can quickly distinguish routine updates from suspicious activity.
Building Team Workflows Around File Monitoring
File integrity monitoring only provides security value when your team knows how to respond to alerts and can distinguish genuine threats from routine maintenance.
Training Team Members on Alert Response
Document your response procedures before deploying monitoring. When someone receives a file change alert, they should know which changes require immediate investigation versus those that can wait until normal business hours.
Create simple decision trees for common scenarios. Changes to SSH configuration files during business hours by known administrators might be routine. The same changes at 2 AM from unknown processes require immediate attention.
Your response procedures should include steps for investigating suspicious changes: checking system logs, reviewing recent login activity, and verifying whether legitimate maintenance caused the detected modifications.
Integrating with Existing Monitoring Setup
File integrity monitoring works best alongside your existing infrastructure monitoring. When Server Scout detects unusual CPU spikes or memory usage, correlate those alerts with any file system changes happening at the same time.
If you're already using Server Scout's service monitoring features, extend your monitoring scripts to integrate with the same notification channels your team already monitors.
For teams managing multiple servers, consider how you'll scale file monitoring across your entire infrastructure. A centralized approach where monitoring results feed into your main dashboard provides better visibility than scattered individual server scripts.
Troubleshooting Common inotify Issues
File monitoring scripts can consume significant system resources if configured incorrectly. Recursive monitoring of large directory trees generates enormous volumes of events.
If your monitoring script stops working, check whether you've exceeded the kernel's watch limits. Use find /proc/*/fd -lname anon_inode:inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' | uniq -c | sort -nr to see which processes are consuming inotify resources.
Network filesystems like NFS don't support inotify monitoring. If your critical directories exist on network mounts, you'll need alternative monitoring approaches or need to monitor the underlying storage directly.
Some applications create temporary files during normal operation, triggering excessive alerts. Web applications might create session files, or log rotation might generate numerous file events. Fine-tune your monitoring to exclude known temporary file patterns while still catching genuine security events.
File monitoring provides early warning about system compromise, but it works best as part of a comprehensive security approach. Regular system updates, proper access controls, and network monitoring all contribute to overall security posture.
For detailed implementation guidance and advanced configuration options, see our Linux Service Status Monitoring article, which covers integrating custom monitoring scripts with system service management.
Small teams can implement effective file integrity monitoring without enterprise security budgets. Starting with basic inotify monitoring of critical directories provides valuable security visibility that scales as your infrastructure grows.
FAQ
How much system overhead does inotify monitoring add?
Properly configured inotify monitoring uses minimal CPU and memory resources. Each watch consumes approximately 1KB of kernel memory. The main overhead comes from processing events, so focus monitoring on directories with infrequent legitimate changes.
Can inotify monitoring detect changes made by root users?
Yes, inotify detects all filesystem changes regardless of which user account makes them. However, sophisticated attackers might use kernel-level rootkits that bypass filesystem monitoring entirely.
What happens if the monitoring script crashes?
When your monitoring script stops running, you lose real-time change detection until you restart it. Consider using systemd or supervisor to automatically restart crashed monitoring processes, and build alerts that notify you when monitoring stops working.