Managing Disk Space

This document examines disk space management as a critical system resource, exploring how programs consume storage through binaries, data, caches, logs, and temporary files. It covers diagnostic approaches for identifying space usage patterns, understanding performance degradation as disks fill up, and implementing strategies to prevent disk exhaustion that can cause application crashes and potential data loss.

Understanding Disk Space Usage

Why Programs Need Disk Space

Another resource that might need attention is the disk usage of computers. Programs may need disk space for lots of different reasons.

Common disk space consumers:

Potential Causes of Space Exhaustion

If a computer is running out of space, it’s possible that there’s an attempt to store too much data in too little space.

Space exhaustion scenarios:

Maybe there are too many applications installed, or an attempt to store too many large files in the drive.

Program Misuse of Disk Space

But it’s also possible that programs are misusing the space allotted to them, like by keeping temporary files or caching information that doesn’t get cleaned up quickly enough or at all.

Misuse patterns:

Performance Impact of Disk Exhaustion

System-Wide Performance Degradation

It’s common for the overall performance of the system to decrease as the available disk space gets smaller.

Performance degradation stages:

Data Fragmentation

Data starts getting fragmented across the disk, and operations become slower.

Fragmentation effects:

Fragmentation performance comparison:

Application Crashes

When a hard drive is full, programs may suddenly crash while trying to write something into disk and finding out that they can’t.

Write failure scenarios:

Risk of Data Loss

A full hard drive might even lead to data loss, as some programs might truncate a file before writing an updated version of it, and then fail to write the new content, losing all the data that was stored in it before.

Data loss patterns:

Warning

A full disk can cause catastrophic data loss when applications truncate files before writing updates. If the truncation succeeds but the write fails due to no space, all original data is permanently lost. Always monitor disk space to prevent reaching this critical state.

Error Messages

If it gets to this point, errors like “no space left on the device” will probably be seen when running applications or in the logs.

Common disk full errors:

Diagnosing Disk Space Issues

User Machine Solutions

So what should be done if a computer runs out of disk space? If it’s a user machine, it might be easily fixed by uninstalling applications that aren’t used, or cleaning up old data that isn’t needed anymore.

User machine cleanup approaches:

Server Investigations

But if it’s a server, a closer look at what’s going on might be needed. Is the issue that an extra drive needs to be added to the server to have more available space, or is it that some application is misbehaving and filling the disk with useless data?

Server diagnostic questions:

Space Usage Analysis

To figure this out, examining how the space is being used and what directories are taking up the most space is needed, then drilling down until finding out whether large chunks of space are taken by valid information or by files that should be purged.

Analysis workflow:

Common disk usage commands:

 1# Check overall disk usage
 2df -h
 3
 4# Find directories using most space
 5du -sh /* | sort -rh | head -10
 6
 7# Find large files over 100MB
 8find / -type f -size +100M -exec ls -lh {} \;
 9
10# Check disk usage by directory, sorted
11du -h /var | sort -rh | head -20
12
13# Find files modified in last 7 days
14find /var/log -mtime -7 -type f -exec du -sh {} \; | sort -rh

Expected vs Anomalous Usage

For example, on a database server, it’s expected that the bulk of the disk space is going to be used by the data stored in the database. On a mail server, it’s going to be the mailboxes of the users of that service.

Expected space usage by server type:

But if most of the data is found to be stored in logs or in temporary files, something has gone wrong.

Anomalous usage indicators:

Common Misbehavior Patterns

Excessive Error Logging

One common pattern of misbehavior is a program that keeps logging error messages to the system log over and over. This can happen for lots of different reasons.

Excessive logging scenarios:

OS Retry Loops

For example, the OS might keep trying to start a program that fails because of a configuration problem. This will generate a new log entry with every retry and can take up a lot of space if there are several retries per second.

Retry pattern impact:

Example error log loop:

1Nov 11 10:15:01 server systemd[1]: Starting myapp.service...
2Nov 11 10:15:01 server myapp[1234]: Configuration file not found: /etc/myapp/config.yml
3Nov 11 10:15:01 server systemd[1]: myapp.service: Main process exited, code=exited, status=1/FAILURE
4Nov 11 10:15:01 server systemd[1]: myapp.service: Failed with result 'exit-code'.
5Nov 11 10:15:02 server systemd[1]: Starting myapp.service...
6Nov 11 10:15:02 server myapp[1235]: Configuration file not found: /etc/myapp/config.yml
7# ... repeats thousands of times ...

High-Volume Legitimate Logging

Or it could be that the server has a lot of activity and the logs are real, but there are just too many of them.

High-activity logging management:

In that case, tweaking the configuration of the tools that rotate the logs more frequently might be needed to make sure that only what’s needed is being kept.

Log rotation configuration strategies:

Example logrotate configuration:

 1/var/log/myapp/*.log {
 2    daily                  # Rotate daily
 3    rotate 7              # Keep 7 days
 4    compress              # Compress old logs
 5    delaycompress         # Don't compress most recent
 6    missingok            # Don't error if log missing
 7    notifempty           # Don't rotate if empty
 8    create 0644 root root # Create new file with permissions
 9    size 100M            # Also rotate if >100MB
10    postrotate
11        systemctl reload myapp
12    endscript
13}

Temporary File Issues

Uncleaned Temporary Files

In other cases, the disk might get full due to a program generating large temporary files and then failing to clean those up.

Temporary file problems:

Cleanup on Normal vs Abnormal Exit

For example, an application might clean up temporary files when shutting down cleanly, but leave them behind if it crashes.

Cleanup behavior comparison:

Programming Errors

Or it could simply be a programming error of creating temporary files and never cleaning them up.

Programming error patterns:

 1# Bad: Temporary file never cleaned up
 2def process_data(input_file):
 3    temp_file = "/tmp/processing_" + str(time.time())
 4    with open(temp_file, 'w') as f:
 5        # Process data...
 6        f.write(processed_data)
 7    # File left behind forever
 8    return result
 9
10# Good: Explicit cleanup
11def process_data_better(input_file):
12    temp_file = "/tmp/processing_" + str(time.time())
13    try:
14        with open(temp_file, 'w') as f:
15            f.write(processed_data)
16        # Process temp file
17        result = process(temp_file)
18    finally:
19        if os.path.exists(temp_file):
20            os.remove(temp_file)
21    return result
22
23# Best: Use tempfile module
24import tempfile
25
26def process_data_best(input_file):
27    with tempfile.NamedTemporaryFile(mode='w', delete=True) as temp_file:
28        temp_file.write(processed_data)
29        temp_file.flush()
30        result = process(temp_file.name)
31    # File automatically deleted when context exits
32    return result

Solutions for Temporary File Problems

In a case like this, ideally there would be some housekeeping to fix the program and delete those files correctly.

Temporary file solutions:

But if that’s not possible, writing a custom script that gets rid of them might be needed.

Cleanup script example:

 1#!/bin/bash
 2# cleanup_temp_files.sh - Remove old temporary files
 3
 4# Find and delete temp files older than 7 days
 5find /tmp -type f -name "processing_*" -mtime +7 -delete
 6
 7# Find and delete empty directories
 8find /tmp -type d -empty -delete
 9
10# Log cleanup action
11echo "$(date): Cleaned up old temporary files" >> /var/log/temp_cleanup.log

Cron job for automated cleanup:

1# Run cleanup script daily at 2 AM
20 2 * * * /usr/local/bin/cleanup_temp_files.sh

Deleted But Open Files

The Tricky Debugging Situation

A situation that might be tricky to debug is when the files taking up the space are deleted files.

How deleted files consume space:

If a program opens a file, the OS lets that program read and write in the file regardless of whether the file is marked as deleted or not.

Intentional Deletion Pattern

So lots of programs delete the temporary files they create right after opening to avoid issues with failing to clean them up later.

Temporary file lifecycle with immediate deletion:

That way, the process can read from and write to the file while the file is open. Then when the process finishes, the file gets closed and actually deleted.

Benefits of immediate deletion pattern:

When Things Go Wrong

Now, this system is widely used and works fine for most processes. But if for some reason this temporarily deleted file starts becoming super large, it can end up taking up all the available disk space.

Deleted-but-open file problems:

If that happens, confusion will result when trying to figure out where most of the data went, since these deleted files won’t be seen.

Important

Deleted but open files consume disk space without appearing in directory listings, making them extremely difficult to diagnose. The df command shows disk usage, but du cannot account for the space because the files don’t exist in the filesystem namespace anymore.

Detecting Deleted Open Files

To check for the specific condition, the currently opened files need to be listed and combed for the ones that are known to be deleted.

Detection commands:

 1# Linux: List open files marked as deleted
 2lsof | grep deleted
 3
 4# Alternative: Check /proc for deleted files
 5find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'
 6
 7# Show processes with deleted files and their sizes
 8lsof -nP | grep '(deleted)' | awk '{print $2, $9, $7}' | \
 9  while read pid name size; do
10    echo "PID $pid: $name ($size bytes)"
11  done
12
13# Find large deleted files (>100MB)
14lsof -nP | grep '(deleted)' | \
15  awk '$7 > 104857600 {print $2, $9, $7/1048576 " MB"}'

lsof output interpretation:

1COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
2myapp    1234 root   3w   REG  253,0 10737418240 1234 /tmp/bigfile (deleted)

Resolution strategies:

General Troubleshooting Approach

Consistent Problem-Solving Process

Of course, there are all kinds of other reasons why the disk may be getting too full. Just remember that whenever this happens, the process will remain the same.

Universal disk troubleshooting workflow:

Investigation Phase

Time will need to be spent looking into what’s using the disk.

Investigation checklist:

Classification Phase

Check to see if it’s expected or an anomaly.

Expected vs anomaly decision tree:

Resolution Phase

Figure out how to solve it.

Resolution strategies by problem type:

Prevention Phase

Most important of all, how to prevent it from happening again.

Prevention strategies:

Monitoring script example:

 1#!/bin/bash
 2# disk_monitor.sh - Alert when disk usage exceeds threshold
 3
 4THRESHOLD=80
 5EMAIL="admin@example.com"
 6
 7df -h | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | \
 8while read output; do
 9  usage=$(echo $output | awk '{ print $1}' | sed 's/%//g')
10  partition=$(echo $output | awk '{ print $2 }')
11
12  if [ $usage -ge $THRESHOLD ]; then
13    echo "ALERT: Partition $partition is ${usage}% full" | \
14      mail -s "Disk Space Alert on $(hostname)" $EMAIL
15  fi
16done

Conclusion

Disk space management represents a critical resource concern where programs consume storage through installed binaries and libraries, application data, caches, logs, temporary files, and backups, with exhaustion potentially caused by legitimate data growth requiring more capacity or program misbehavior through inadequate cleanup of temporary files and logs. As available disk space decreases, overall system performance degrades through data fragmentation causing slower operations, with full disks leading to application crashes when write operations fail and potential catastrophic data loss when programs truncate files before writing updates that then fail due to insufficient space, generating “no space left on device” errors. User machines can often be fixed through simple cleanup like uninstalling unused applications and deleting old data, but servers require detailed investigation to determine whether adding storage capacity is needed or if applications are misbehaving by filling disks with useless data, using commands like df and du to analyze space usage patterns and identify whether large chunks are legitimate data like databases and mailboxes or anomalies like excessive logs and temporary files. Common misbehavior patterns include programs logging error messages repeatedly when continuously failing to start due to configuration problems (potentially generating gigabytes per day), legitimate high-activity logging requiring more frequent log rotation to manage volume, and temporary files that programs fail to clean up either due to crashes preventing cleanup execution or simple programming errors never implementing deletion logic. The particularly tricky scenario of deleted but open files occurs when programs delete temporary files immediately after creation for guaranteed cleanup but before closing them, consuming disk space invisibly without appearing in directory listings since the OS maintains file access for open file descriptors regardless of deletion status, requiring lsof commands to detect these hidden space consumers. The consistent troubleshooting approach involves spending time investigating what uses the disk through filesystem and directory analysis, classifying whether usage is expected based on server role or anomalous requiring intervention, figuring out how to solve the immediate issue through cleanup or capacity addition, and most importantly implementing prevention strategies including disk monitoring with alerts at 80% usage, proper log rotation configuration, automated cleanup scripts for temporary files, and capacity planning to forecast and prevent future exhaustion events.

FAQ