When df Shows Plenty of Space but You Can't Create Files: The Inode Exhaustion Debug Guide

You're staring at a server that's refusing to create new files. touch test.txt throws "No space left on device", but df -h clearly shows 15GB available on the filesystem. Your log rotation has stopped, temporary files won't write, and users are complaining.

Welcome to inode exhaustion - one of the most confusing disk space problems you'll encounter as a sysadmin.

Understanding the Inode Layer

Every file on a Unix filesystem requires two things: disk blocks for the actual data and an inode for the metadata. Think of inodes as the filesystem's filing cabinet - each drawer (inode) holds information about one file: permissions, timestamps, ownership, and pointers to the data blocks.

When you format a filesystem, the number of inodes is typically fixed based on the partition size and expected file size. The default ratio is usually one inode per 16KB of disk space, but this assumption breaks down when applications create millions of small files.

Check your current inode usage with df -i. You'll see something like this:

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      1310720 1310720       0  100% /var

That 100% usage is your problem, even if df -h shows plenty of free space.

Common Culprits Behind Inode Exhaustion

Mail servers are notorious for this. Postfix queue directories can accumulate hundreds of thousands of small bounce messages. Each email generates multiple files: the message body, headers, and queue control files.

Session storage in web applications creates another hotspot. PHP applications writing session data to /tmp or /var/lib/php/sessions can generate massive numbers of small files, especially if session garbage collection isn't working properly.

Log files with aggressive rotation policies contribute too. Some applications create individual log files per request or transaction instead of writing to shared logs. This becomes particularly problematic in high-traffic environments where small files accumulate faster than cleanup processes can handle them.

Backup systems sometimes create temporary files during operations and fail to clean them up properly. Check directories like /tmp, /var/tmp, and application-specific temporary locations.

Hunting Down the Space Hogs

Start by identifying which directory is consuming your inodes. This one-liner counts files recursively and sorts by the highest count:

find / -xdev -type f | cut -d"/" -f2 | sort | uniq -c | sort -nr

For a more targeted approach, count files in suspect directories:

find /var/spool/postfix -type f | wc -l
find /var/lib/php/sessions -type f | wc -l
find /tmp -type f | wc -l

Remember that deleted files still consume inodes if processes have them open. Use lsof +L1 to find deleted files that are still holding inodes.

Prevention and Monitoring

Set up monitoring for inode usage alongside disk space monitoring. A proper monitoring setup should alert you when inode usage hits 80-85%, well before you hit the wall.

Consider filesystem choices for high-file-count scenarios. XFS allows dynamic inode allocation, unlike ext4's fixed inode table. For mail servers and applications that create millions of small files, this flexibility prevents inode exhaustion.

Implement proper cleanup procedures in your applications. Session cleanup, temporary file removal, and mail queue maintenance should run regularly and be monitored for success.

Tune inode density at filesystem creation time if you know you'll be storing many small files. Use mkfs.ext4 -i 4096 to create more inodes (one per 4KB instead of the default 16KB).

Recovery Strategies

When you're in the middle of an inode exhaustion crisis, focus on quick wins. Delete the largest collections of unnecessary files first - often these are in /tmp, mail queues, or session directories.

For mail servers, flush the queue of obvious spam or bounce messages. For web applications, clear old session files and temporary uploads.

If you cannot free enough inodes immediately, consider moving some data to a different filesystem temporarily while you resolve the underlying cause.

Server Scout's disk monitoring includes inode usage tracking, helping you spot these issues before they become emergencies. The lightweight agent monitors both space and inode consumption without adding overhead to already-stressed systems.