, , , ,


In almost every script I write I include extensive logging so that I can keep track of issues and so that I can automate much of the administration of the Linux systems I need to maintain.I also use a NRPE plugin to scan the logs for the work ERROR which I ensure gets written when the script or application finds itself in an error situation, this then ends up in the monitoring system (see Nagios articles).

In the process I generate a LOT of log files which I keep separate to the standard system logs and http logs you find under the common /var/log filesystem. I’m also pedantic to impose that all log files are date stamped and a new log file is created every day.

As a result these files grow and need to be cleaned up over time. I usually keep stuff from 7 to 90 days depending on what it is, and I can compress files into an archive log server if I need to just by using rsync to move the logs over to another server and let it do the archive work.

Below is my standard cleanup-directories.sh script, its role is to remove files that are older than “X” days, where “X” is passed on the command line with the target directory. The cleanup script creates its own log file in /logs/cron/cleanup called dircleanup-YYYY-MM-DD.log


Using the script is pretty easy, the command line parameters its takes are -r and -d where:

  • -r “nn”  -> retention time (in days) i.e. -r 14
  • -d “/path/directory” -> The directory to cleanup old files in


Cleanup old entries for the cleanup program itself!

root/cleanup-directories.sh -r 14 -d /logs/cron/cleanup

Recursive Clean up

The Script will clean up the directory recursively, so I always start at the root with the longest any log file will remain, say 90 days then run the 60, 45 adn 30 day cleanup tasks after that, the do the specific directories for 14 and 7 day clean up.

root/cleanup-directories.sh -r 90 -d /logs
root/cleanup-directories.sh -r 30 -d /logs/cron
root/cleanup-directories.sh -r 14 -d /logs/dbbackups
root/cleanup-directories.sh -r 7 -d /logs/app-logs


Typical CRONTAB entries

Below is an extract from the crontab file of a live database server, not only do we cleanup the log files, but we get rid of old backups as well that are more than 31 days old. It is suggested you select suitable times late at night and perform cleanup of suitable directories, be very careful of which ones you choose, there is no recovery once they are gone!

#crontab -l | grep cleanup
30 21 * * * /root/cleanup-directories.sh -r 14 -d /logs/cron/cleanup
31 21 * * * /root/cleanup-directories.sh -r 21 -d /logs/cron
32 21 * * * /root/cleanup-directories.sh -r 28 -d /logs
33 21 * * * /root/cleanup-directories.sh -r 31 -d /data/db-backups

Initial Setup

Basically the initial setup is for you to create a /logs directory off the root partition while running as the superuser and give the /logs directory permissions of 777 using the following commands:

mkdir /logs
chmod 777 /logs

The script – cleanup-directories.sh

# Directory Cleanup Script - Remove any old files in a specified directory
# Free to use commercially or non-commercially provided acknowledgement 
# of the author is retained.
# History
# -------
# 2008-12-22  Sid Young  Created script to clean up old files to reduce disk space

# Common Logging Code
function LogStart
        echo "====== Log Start =========" >> $LF
        echo "Time: `date`" >> $LF
        echo " " >> $LF
function LogEnd
        echo " " >> $LF
        echo "Time: `date`" >> $LF
        echo "====== Log End   =========" >> $LF

function LogMsg
        echo "`date '+%Y-%m-%d|%H:%M:%S|'`$$|OK|$1" >> $LF

function LogError
        echo "`date '+%Y-%m-%d|%H:%M:%S|'`$$|ERROR|$1" >> $LF

function LogCritical
        echo "`date '+%Y-%m-%d|%H:%M:%S|'`$$|CRITICAL|$1" >> $LF

FILE_DATE=`date '+%Y-%m-%d'`
mkdir -p $LF_DIR
chmod 777 /logs/cron
chmod 777 /logs/cron/cleanup
touch $LF
chmod 644 $LF

# Process any command line parameters

set -- getopt dr: "$@"
while [ $# -gt 0 ]
    case "$1" in
        -d)     TARGET_DIR="$2" ;;
        -r)     KEEP_DAYS="$2" ;;

LogMsg "Target directory: $TARGET_DIR"
LogMsg "Retension Period: $KEEP_DAYS"
if [ -d "$TARGET_DIR" ]
        cd $TARGET_DIR
        LogMsg "$TARGET_DIR exists - Locating files to delete"
        CNT=`find $TARGET_DIR -type f -mtime +$KEEP_DAYS -print | wc -l`
# Find the files and list them to the log
        find $TARGET_DIR -type f -mtime +${KEEP_DAYS} -print > /tmp/cleanup-list-$$

        LogMsg "File list is:"
        LogMsg "-------------"
        for file in `cat /tmp/cleanup-list-$$`
                LogMsg $file
        LogMsg " "
        for file in `cat /tmp/cleanup-list-$$`
                rm $file > /dev/null 2>&1
                if [ $? -eq 1 ]
                        LogMsg "File $file Not Found?"
        LogMsg "Done $CNT files removed"
        LogCritical "Aborting - $TARGET_DIR does not exist"
        echo "ERROR - Target for cleanup [ $TARGET_DIR ] does not exist - aborting" | mailx -s "[CRON] Cleanup failed" $EMAIL
rm -f /tmp/cleanup-list-$$
# End of file

Where to from here?

I would suggest using the Log code in your shell scripts so they generate suitable log files with meaningful messages that both report status of apps that run through the day as well as debug code for apps that may cause problems from time to time. If the I/O to the /logs file system starts to get too high the use LVM to create a new logical volume and mount it at the /logs mount point.