Tags

, , , ,


Overview

In almost every script I write I include extensive logging so that I can keep track of issues and so that I can automate much of the administration of the Linux systems I need to maintain.I also use a NRPE plugin to scan the logs for the work ERROR which I ensure gets written when the script or application finds itself in an error situation, this then ends up in the monitoring system (see Nagios articles).

In the process I generate a LOT of log files which I keep separate to the standard system logs and http logs you find under the common /var/log filesystem. I’m also pedantic to impose that all log files are date stamped and a new log file is created every day.

As a result these files grow and need to be cleaned up over time. I usually keep stuff from 7 to 90 days depending on what it is, and I can compress files into an archive log server if I need to just by using rsync to move the logs over to another server and let it do the archive work.

Below is my standard cleanup-directories.sh script, its role is to remove files that are older than “X” days, where “X” is passed on the command line with the target directory. The cleanup script creates its own log file in /logs/cron/cleanup called dircleanup-YYYY-MM-DD.log

Usage

Using the script is pretty easy, the command line parameters its takes are -r and -d where:

  • -r “nn”  -> retention time (in days) i.e. -r 14
  • -d “/path/directory” -> The directory to cleanup old files in

Example:

Cleanup old entries for the cleanup program itself!

root/cleanup-directories.sh -r 14 -d /logs/cron/cleanup

Recursive Clean up

The Script will clean up the directory recursively, so I always start at the root with the longest any log file will remain, say 90 days then run the 60, 45 adn 30 day cleanup tasks after that, the do the specific directories for 14 and 7 day clean up.

root/cleanup-directories.sh -r 90 -d /logs
root/cleanup-directories.sh -r 30 -d /logs/cron
root/cleanup-directories.sh -r 14 -d /logs/dbbackups
root/cleanup-directories.sh -r 7 -d /logs/app-logs

 

Typical CRONTAB entries

Below is an extract from the crontab file of a live database server, not only do we cleanup the log files, but we get rid of old backups as well that are more than 31 days old. It is suggested you select suitable times late at night and perform cleanup of suitable directories, be very careful of which ones you choose, there is no recovery once they are gone!

#crontab -l | grep cleanup
30 21 * * * /root/cleanup-directories.sh -r 14 -d /logs/cron/cleanup
31 21 * * * /root/cleanup-directories.sh -r 21 -d /logs/cron
32 21 * * * /root/cleanup-directories.sh -r 28 -d /logs
33 21 * * * /root/cleanup-directories.sh -r 31 -d /data/db-backups

Initial Setup

Basically the initial setup is for you to create a /logs directory off the root partition while running as the superuser and give the /logs directory permissions of 777 using the following commands:

mkdir /logs
chmod 777 /logs

The script – cleanup-directories.sh

#!/bin/bash
#
# Directory Cleanup Script - Remove any old files in a specified directory
#
# Free to use commercially or non-commercially provided acknowledgement 
# of the author is retained.
#
# History
# -------
# 2008-12-22  Sid Young  Created script to clean up old files to reduce disk space

TARGET_DIR=/tmp
KEEP_DAYS=90
EMAIL=<YOUR.EMAIL@your.domain.com>
#
# Common Logging Code
#
function LogStart
{
        echo "====== Log Start =========" >> $LF
        echo "Time: `date`" >> $LF
        echo " " >> $LF
}
function LogEnd
{
        echo " " >> $LF
        echo "Time: `date`" >> $LF
        echo "====== Log End   =========" >> $LF
}

function LogMsg
{
        echo "`date '+%Y-%m-%d|%H:%M:%S|'`$$|OK|$1" >> $LF
}

function LogError
{
        echo "`date '+%Y-%m-%d|%H:%M:%S|'`$$|ERROR|$1" >> $LF
}

function LogCritical
{
        echo "`date '+%Y-%m-%d|%H:%M:%S|'`$$|CRITICAL|$1" >> $LF
}

UMASK=002
FILE_DATE=`date '+%Y-%m-%d'`
LF_DIR=/logs/cron/cleanup
LF=$LF_DIR/dircleanup-$FILE_DATE.log
mkdir -p $LF_DIR
chmod 777 /logs/cron
chmod 777 /logs/cron/cleanup
touch $LF
chmod 644 $LF

#----------------------------------------
#
# Process any command line parameters
#
#----------------------------------------
LogStart

set -- getopt dr: "$@"
while [ $# -gt 0 ]
do
    case "$1" in
        -d)     TARGET_DIR="$2" ;;
        -r)     KEEP_DAYS="$2" ;;
    esac
    shift
done

LogMsg "Target directory: $TARGET_DIR"
LogMsg "Retension Period: $KEEP_DAYS"
if [ -d "$TARGET_DIR" ]
then
        cd $TARGET_DIR
        LogMsg "$TARGET_DIR exists - Locating files to delete"
        CNT=`find $TARGET_DIR -type f -mtime +$KEEP_DAYS -print | wc -l`
#
# Find the files and list them to the log
#
        find $TARGET_DIR -type f -mtime +${KEEP_DAYS} -print > /tmp/cleanup-list-$$

        LogMsg "File list is:"
        LogMsg "-------------"
        for file in `cat /tmp/cleanup-list-$$`
        do
                LogMsg $file
        done
        LogMsg " "
        for file in `cat /tmp/cleanup-list-$$`
        do
                rm $file > /dev/null 2>&1
                if [ $? -eq 1 ]
                then
                        LogMsg "File $file Not Found?"
                fi
        done
        LogMsg "Done $CNT files removed"
else
        LogCritical "Aborting - $TARGET_DIR does not exist"
        echo "ERROR - Target for cleanup [ $TARGET_DIR ] does not exist - aborting" | mailx -s "[CRON] Cleanup failed" $EMAIL
fi
rm -f /tmp/cleanup-list-$$
LogEnd
#
# End of file
#

Where to from here?

I would suggest using the Log code in your shell scripts so they generate suitable log files with meaningful messages that both report status of apps that run through the day as well as debug code for apps that may cause problems from time to time. If the I/O to the /logs file system starts to get too high the use LVM to create a new logical volume and mount it at the /logs mount point.

Enjoy!

Advertisements