Solution¶

Triage¶

Assess disk usage:

df -h /
du -sh /var/log/* | sort -rh | head -10

Find the largest log files:

find /var/log -type f -size +1G -exec ls -lh {} \;

Check logrotate configuration:

cat /etc/logrotate.d/webapp
cat /var/lib/logrotate/status | grep webapp

Check for deleted-but-open files:
```
lsof +L1 | grep deleted
```

Root Cause¶

The logrotate configuration for /var/log/webapp/access.log uses the default rotation method (rename the file, then create a new one). However, the web application holds the old file descriptor open and continues writing to the renamed file. The new empty access.log receives no writes, while the rotated file grows unbounded.

The logrotate config is missing either the copytruncate directive or a postrotate script to send SIGHUP to the application. As a result, effective rotation never happens, and the single access log has grown to 72 GB over 45 days.

Fix¶

Immediate (free disk space):

Truncate the active log file without stopping the application:
```
> /var/log/webapp/access.log
```
This zeroes the file while the application continues writing to the same file descriptor.
If there are deleted-but-open files, restart the holding process to release them:
```
systemctl restart webapp
```

Permanent fix:

Fix the logrotate configuration:

# /etc/logrotate.d/webapp
/var/log/webapp/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    maxsize 1G
}

Test the configuration:

logrotate -d /etc/logrotate.d/webapp    # dry run
logrotate -f /etc/logrotate.d/webapp    # force run

Verify logrotate cron is running:

systemctl status logrotate.timer    # systemd timer
cat /etc/cron.daily/logrotate       # or cron-based

Set journal size limits:

# /etc/systemd/journald.conf
SystemMaxUse=2G
systemctl restart systemd-journald

Rollback / Safety¶

Truncating a log file with > is safe for the running process but destroys log data. Archive first if needed.
copytruncate has a tiny window where log lines can be lost during rotation. For most applications this is acceptable.
If the application supports SIGHUP for log reopening, prefer a postrotate script over copytruncate.

Common Traps¶

Deleting the log file instead of truncating. rm access.log frees the directory entry but the process still holds the fd open, so disk space is not freed until the process exits.
Assuming logrotate is running. Check that the cron job or systemd timer is actually active and not failing silently.
Not testing logrotate with -d (debug/dry-run). Syntax errors in the config silently prevent rotation.
Forgetting delaycompress. Without it, the file is compressed immediately after rotation, which can conflict with copytruncate.
Ignoring journald. On systemd systems, the journal can consume many gigabytes independently of /var/log files.