HowTo: Using the Cron Shell Tool to Automate Procedures - Part 1 of 3

Novell Cool Solutions: Feature
By Stomfi

Digg This - Slashdot This Posted: 25 Jan 2005

Learning to use Linux at Home and Work Welcome to my ongoing series of HowTo articles designed to help Linux newbies get comfortable with Linux. Before trying any of these HowTos, take a few minutes to study the prerequisites so you can hit the ground running. --Stomfi

This HowTo explains how you can spend more time doing quality pursuits by using a Linux PC to automate general administrative tasks and procedures. There are three parts. The whole scheme is written using the Linux shell, as it is designed to run without any user intervention. This first part covers the strategy and most of the shell scripts. The second part covers the main shell script, auto email system setup, testing, and the cron script. The third deals with automatically producing reports using Gnuplot.

When doing general administrative work, much of it involves acquiring information, getting it into a computer, and performing analysis and processing, which may create follow up tasks, creating reports and charts for others to appraise. By utilising the cron daemon, many tasks which are performed manually on a Microsoft system, can be performed automatically by your Linux computer, when information is received, events happen, or criteria are satisfied.

Depending how data is received, some of it can be recognised by the computer. But some of it may have to be manually entered, or scanned in with an OCR program. Even a small business can benefit by using low cost data entry clerks for this activity, while other clerical processing is automated by their Linux machines.

Some of the types of processes that can be automated under cron are:

• Checking for due information
• Sending reminders for overdue information
• Checking for new information
• Posting new information to storage and creating links
• Processing new information
• Creating, sending or printing reports, charts, letters, or agendas
• Requesting information
• Analysing information
• Displaying information on screen depending on criteria

Because cron looks at its action tables once every minute, and actions can be initiated by cron once a minute, hour, day, once a month, year or combinations of those at specific times or every 9999 years, true office automation can be achieved with absolutely no additional user input required. This is something that is difficult if not impossible to set up on a Microsoft system as they were designed to create an industry that was not necessary and didn't need to exist in the planned development of office automation machinery.

A simple practical example will show you how to implement a system using Linux shell tools for processing and shell scripts for actions. Cron table entries will look for new information, and send required information. You will gain the knowledge to apply to your own work processes, and be able to kid the boss you are being busy doing your work, when in reality having some quality time, while your Linux computer does it all for you.

The example will be an emailed newsletter and list of readers. Your quality time can be spent creating new newsletters, as the clerical administrative functions will be automated. Each reader requests their first newsletter via email, then they email a renewal for each subsequent one. If they don't renew by a cut off date, they are first sent a reminder and is still no reply are removed from the list. If the system finds a current renewal date and they haven't received one, it emails them a new newsletter. Nice a simple, but shows off the power of real office automation.

Create a folder in your HOME directory and call it "newsletter". Create a sub folder called "bin" for our shell scripts. Create a sub folder called "issues" for the newsletters.

Welcome to newsletter 1
If you want to receive another, click on the email link below
and send an empty email.
email: renewal <myname@myisp.com>

Welcome to newsletter 2
If you want to receive another, click on the email link below
and send an empty email.
email: renewal <myname@myisp.com>

The above links have a subject line which should say "renewal"

Also you need a web page advertising your newsletter. It has the following line:

If you would like to receive a copy of our newsletter, click on the email link below
and send an empty email.
email: newsletter <myname@myisp.com>

The subject line of the above should be "reader"

You will also need a Please Confirm email like this one which was created in Thunderbird mail and saved as the file $HOME/newsletter/pconfirm.eml.

X-Mozilla-Status: 0001

X-Mozilla-Status2: 01000000

FCC: mailbox://myname@pop-server.myisp.net/Sent

X-Identity-Key: id1

Date: Tue, 23 Nov 2004 14:24:40 +1000

From: myname <myname@anotherisp.com>

X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; uuencode=0

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4

X-Accept-Language: en-us, en

MIME-Version: 1.0

Subject: Please Confirm Newsletter

Content-Type: text/html; charset=us-ascii

Content-Transfer-Encoding: 7bit




<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>

<head>

<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">

<title></title>

</head>

<body>

You have not renewed your myname newsletter<br>

If you wish to receive another please send an empty email<br>

with the subject: renewal<br>

to the link below<br>

<a href="myname@myisp.com">renewal myname@myisp.com</a><br>

</body>

</html>

That's all the dummy information. Now for the received email setup.

If you've got DSL or cable you can open Kmail and minimise it so any new mail goes into the correct folders. If you want to be truly automated you can use the shell tool "fetchmail" and a daily cron table entry to fetch the mail and go play sports while your work is being done, but for now we shall let Kmail do the work in our normal GUI interface.

The Linux login account for this to work should be the same name as the email user name.

e.g. If your login name was "stomfi", your email name would be "stomfi@myisp.com"

If you are going to use Kmail, you can create "mbox" type folders called renewals and readers, and apply filters to your inbox which will put incoming emails with the above subject lines into the relevant folders.

We can use the shell tool "grep" to scan these mbox mail folders for the email address of the sender and the date. One problem with this approach is that every email received is saved in this file whether it has been processed or not. Kmail can be set to empty unread mail from these folders after a certain time. This is such a handy feature for our application, I suggest you use it to get this mail and set the "unread" time to 7 days, since only the computer will read it.

We can get the computer to extract the key lines into a file each time and compare it with the last one, so it can process only the new differences. Yes, there are Linux tools that do that, and they were invented long before Microsoft decided they could make money by getting uninformed people to do it manually on their cheap primitive systems. Now that BSD & GNU/Linux is on their preferred hardware however, the uninformed are becoming the informed, like you are by following these HowTo.

The first script we write is to get the new readers from the email and put them into our Rlist file and the date into our Rdate file. This one will get called each day at 1:00 P.M. by cron.

The second script gets the renewals from the email and puts them into the Rdate file. We will run this one at 3:00 P.M. via cron.

The third script checks to see if a new issue is available and puts its name in the issues file.

The fourth script sends the current newsletter to readers with a renewal date for the current period, and puts a date sent into the Rsent file with the ID and issue. If the renewal date is not current a "please confirm" script is called. If it is current the "mail letter" script is called.

Here are the scripts. The first and second scripts can be the same one, as they almost do the same work. We can separate the difference by calling them with two arguments, being email folder name and type.

#!/bin/bash

#$HOME/newsletter/bin/extract.sh Folder=renewals or readers Type= renewal or reader

#extract email lines from folder for type of email

#

#This is so we can use this script for both renewals and new readers 

FOLDER="$HOME/Mail/$1"

TYPE=$2

#

#check for required number of arguments

if [ $# -lt 2 ]

then

   echo "Usage: extract.sh mailfoldername type"

   echo "Where type is renewal or reader"

   exit 1

   fi

#Set up the application folder

AHOME="$HOME/newsletter"

#

#Set up TYPE variables

if [ $TYPE == "renewal" ]

then

   NEW="$AHOME/newrenewals"

   OLD="$AHOME/oldrenewals"

else

   NEW="$AHOME/newreaders"

   OLD="$AHOME/oldreaders"

fi

#Set up some common variables for less typing

RLIST="$AHOME/Rlist"

RDATE="$AHOME/Rdate"

#

#Put temporary files into $AHOME/tmp

ATMP="$AHOME/tmp"

#Make sure it exists

mkdir $ATMP

#

#set up the temporary variables

NEWLIST="$ATMP/newlist"

TMPLIST="$ATMP/tmplist"

FDATE1="$ATMP/fdate1"

FDATE="$ATMP/fdate"

NADDR="$ATMP/newaddr"

THISNEW="$ATMP/thisnew"

#

#rename existing file 

if [ -e $NEW ]

then

   mv $NEW $OLD

fi

#

#Look for lines containing the word "From" and an @ sign

grep "From" "$FOLDER" | grep "@" >> $NEW

#

#print each line in new that does not exist in old into a file.

if [ ! -e $OLD ]

then

   cp $NEW $NEWLIST

else

   diff $OLD $NEW > $TMPLIST

   #If this is empty then are are no new ones

   #so we can exit this script with no further action

   if [ ! -s $TMPLIST ]

   then

      exit

   fi 

   #else its size is greater than zero

   #create a new file from the diff file

   #Remove the old renewals file and recreate it empty

   rm -f $NEWLIST

   touch $NEWLIST

   #Use patch to apply the differences

   patch $NEWLIST $TMPLIST

fi

#

#cut each line into email address and date formatted how we want

#

#Get all the dates into a file

cat $NEWLIST | awk -F" " '{ print $4 " " $5 " " $7 }' > $FDATE1

#

#Use date with the -f file option to change them

date -f $FDATE1 +%D > $FDATE

#

#Get all the addresses

cat $NEWLIST | awk -F" " '{print $2}' > $NADDR

#

#paste these together

paste -d"#" $NADDR $FDATE > $THISNEW

#

#Now we've got a file in the right format 

#Do actions depending on TYPE

#

if [ $TYPE == "renewal" ]

then

   #we append it to the Rdate file and sort it for easy retrieval

   cat $THISNEW $RDATE > $ATMP/rupdate

   cat $ATMP/rupdate | sort| uniq > $RDATE 

   #we have got to check that our renewals are valid by checking to see 

   #if they are in the readers list, if not presume it may be a

   #deleted reader come back for more by reinstalling them.

   #we can update easily by appending this file to the RList, sorting

   #and printing only those that are unique. Easy.

   cat $THISNEW $RLIST > $ATMP/lupdate

   cat $ATMP/lupdate | sort | uniq > $RLIST

else

   #TYPE will be readers 

   cat $THISNEW $RLIST | sort | uniq > $RLIST

   #We make sure the new readers get their date into the Rdate file

   cat $THISNEW $RDATE | sort | uniq > $RDATE 

fi

#

#Done it. End of script so exit

exit

Here is the script that moves issue names into a file so that another script can use it.

#!/bin/bash

#issues.sh

ls -l --time-style=+%D $HOME/newsletter/issues | awk '{print $6 "#" $7}' > $HOME/newsletter/Issues

This sendletter script sends the current newsletter to readers with a renewal date for the current period, and puts a date sent into the Rsent file with the ID and issue. If the renewal date is not current a "please confirm" email is sent and a record put into the Rconfirm file.

This script is the "please confirm" script.

#!/bin/bash

#pconfirm.sh 

#

#Send please confirm email to newsletter readers

#

SDATE=`date +%D`

#The standard file to send is called $HOME/newletter/pconfirm.eml

#

#send the current newsletter to addressees in file 

#$HOME/newsletters/tmp/pconfirm.tmp

ALIST=`cat $HOME/newsletter/tmp/pconfirm.tmp`

for ADDR in $ALIST

do

   mutt -s "Confirmation Reminder" $ADDR < $HOME/newletter/pconfirm.eml

   echo "$ADDR#$SDATE" >> $HOME/newsletter/Rconfirm

   done

#

#This is a one on one script. In a real situation a mail

#list would be used. 

And this is the "mletter script" to send the newsletter.

#!/bin/bash

#mletter.sh issue 

#

SDATE=`date +%D`

#send the current newsletter to addressees in temp file

ALIST=`cat $HOME/newsletter/tmp/nsend.$1`

for ADDR in $ALIST

do

   mutt -s "Newsletter $1" $ADDR < $HOME/newsletter/issues/$1

   echo "$ADDR#$SDATE" >> $HOME/newsletter/Rsent

done

#

#This is a one on one script. In a real situation a mail

#list would be used.

Next time we will write the sendletter script which does all the analysing to find out whether to do any of the email actions. We will also set up our computer to use postfix to send the mail via mutt. We will use the Webmin program to do this. Lastly we will check that all the scripts work, and set up cron to do it.

Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com