Linux: deleted files keeps consuming disk space

Sometimes, usually log files, keeps consuming disk space even after deletion. It’s caused by locks of a process which still keeps using them.

In order to diagnose the problem we have to find those files.

lsof | grep deleted | grep /var/log/apache2 | sort | tail

The output should look something like:

apache2 8728 www-data 79w REG 254,2 23002246 524353 /var/log/apache2/iframes.espoluprace.cz-access.
log.1 (deleted)
apache2 8728 www-data 80w REG 254,2 5629092 524396 /var/log/apache2/vip.espoluprace.cz-data-access
.log.1 (deleted) 
...
...

You can see there is process apache2, pid 8728 run by user www-data which holds the files for 79 weeks. The only thing you need to do is to restart the service in this example and the disk space gets freed up.