Novell Home

HowTo: Using the cron shell tool to automate procedures Part 3 of 3

Novell Cool Solutions: Feature
By Stomfi

Rate This Page

Reader Rating  stars  from 1 ratings

Digg This - Slashdot This

Posted: 16 Feb 2005 		 

StomfiLearning to use Linux at Home and Work
Welcome to my ongoing series of HowTo articles designed to help Linux newbies get comfortable with Linux. Before trying any of these HowTos, take a few minutes to study the prerequisites so you can hit the ground running.
--Stomfi

 

 

 

In the first part of this HowTo we wrote several shell scripts to automate procedures. In the second part we wrote the last script which controls the sending of emails and the cron table entries. In this part we write the Gnuplot shell script which creates the data files and the plot configuration files for some bar charts which are saved as png pictures. An html file script is used to create a report. Lastly we look at this report and see how to use its methods to create a newsletter.

The first thing to be done is a modification of the extract.sh script so it includes a history file about the new readers. The two lines are marked with a ##########################

#!/bin/bash

#$HOME/newsletter/bin/extract.sh Folder=renewals or readers 
Type= renewal or reader

#extract email lines from folder for type of email

#

lots of missing script here

#

#Set up some common variables for less typing

#######################################

#Added variable for history file for charting

#

HLIST="$AHOME/Hlist"

RLIST="$AHOME/Rlist"

RDATE="$AHOME/Rdate"

#

#

lots of missing script here

#

else

   #TYPE will be readers so we repeat the above

   cat $THISNEW $RLIST | sort | uniq > $RLIST

   #Then we make sure the new readers get their date into 
   the Rdate file

   cat $THISNEW $RDATE | sort | uniq > $RDATE 

   ########################################

   #Added line for history file for charting

   cat $THISNEW $HLIST | sort | uniq >> $HLIST

   #

fi

#

#Done it. End of script so exit

exit

I left out most of the script, and only showed the added bits.

This is the plotting script. I have included lots of comments so you can see what it is doing.

#!/bin/bash

#plot.sh 

#If you need to print this file use lpr plot.sh

#

#uses gnuplot to create a png chart from newsletter data which is 

#contained in the files Hlist, Rdate, Rconfirm and Rsent

#All files are in the same format being address#date

#

#We get how many new readers per month

#by creating a history file as an extra step in the extract process

#We sort the Hlist file by the date field and make counts for

#each month

#

#From Rdate we get how many renewals per month 

#From Rconfirm we get how many reconfirms per month

#From Rsent we get how many newsletters per month

#From Rsent & Hlist gives how many dropouts per month 

#which is a count of Hlist addresses which are not in the 
Rsent per month 

NHOME="$HOME/newsletter"

PTMP="$NHOME/tmp"

#

#Each of the first 4 plots are basically the same.

#We create individual folders for the plot names

#and save the data relevant to each one

PREAD="$PTMP/pread"

mkdir $PREAD

PRENEW="$PTMP/prenew"

mkdir $PRENEW

PCONF="$PTMP/pconf"

mkdir $PCONF

PSENT="$PTMP/psent"

mkdir $PSENT

PDROP="$PTMP/pdrop"

mkdir $PDROP

#Now we've got files to sort

#We've got to sort so we can total each date count

sort -t"#" -k2,2n -o $PREAD/chartsrt.txt $NHOME/Hlist

sort -t"#" -k2,2n -o $PRENEW/chartsrt.txt $NHOME/Rdate

sort -t"#" -k2,2n -o $PCONF/chartsrt.txt $NHOME/Rconfirm

sort -t"#" -k2,2n -o $PSENT/chartsrt.txt $NHOME/Rsent

#

#All that's left is the dropouts chart

#We now have the data in the PSENT and PREAD directories

#We need to find those addresses in the PREAD/chartsrt.txt 
data that are not in

#the PSENT/chartsrt.txt data

#

#We can do this with one of the Linux shell database tools

#join will print out unpairable lines of two sorted files

#with the -v filenumber option

join -t"#" -v 2 $PSENT/chartsrt.txt $PREAD/chartsrt.txt > $PDROP/chartsrt.txt

#If we were using English dates we could use awk with a 

#field separator of "/" which is the date separator to 

#compare the the 2nd field which is the English month,

#but since the Americans have to be different and put the month

#out of its natural order, we have to do two processes to extract the

#month. 

#

#extract the file, replacing the # with a / so we get a / in front 

#of the date to make this work

#

awk -F"#" '{print $1 "/" $2}' $PREAD/chartsrt.txt > $PREAD/chartslash.txt 

awk -F"#" '{print $1 "/" $2}' $PRENEW/chartsrt.txt > $PRENEW/chartslash.txt 

awk -F"#" '{print $1 "/" $2}' $PCONF/chartsrt.txt > $PCONF/chartslash.txt 

awk -F"#" '{print $1 "/" $2}' $PSENT/chartsrt.txt > $PSENT/chartslash.txt 

awk -F"#" '{print $1 "/" $2}' $PDROP/chartsrt.txt > $PDROP/chartslash.txt 

#

#The logic of this may be a bit hard to work out in awk, so a good way

#is to write what is called stomfi's psuedo code

#

#for each line in input file

#do the following actions

   # if its the first line, then

      # set OLDDATE = Field 2"/"Field 3"/"Field 4

      # set OLDTYPE = Field 2

      # set OLDCOUNT = 1

   # else (not the first line)

      # if Field 2 = OLDTYPE, then

        # OLDCOUNT = OLDCOUNT plus 1

      # else (its a different type)

         # print OLDDATE "#" OLDCOUNT appended to output file

         # set OLDDATE = Field 2"/"Field 3"/"Field 4

         # set OLDTYPE = Field 2

         # set OLDCOUNT = 1

      # end if

   # end if

   #at the end print OLDDATE "#" OLDCOUNT appended to output file

   #as there will not be a new record to compare

#done

#

#We've got to do this for each file in the five directories

#So we can give the script a list or repeat the statements five

#times, once for each directory

#This how to make the list

DLIST="$PREAD $PRENEW $PCONF $PSENT $PDROP"

#This is how to process for each item in the list

for DNAME in $DLIST

do

   #This is how the psuedo code looks when translated into awk

   #Since awk reads each line at a time we leave out the for 
   do done bit

   #Awk also uses ++ to increment OLDCOUNT and ; to separate 
   multiple statements

   #Remember the \ is used to tell the shell this is all on one line

   #It makes it easier to read by setting out the lines separately 
   like this

   #

   awk -F"/" '{if(NR == 1){ OLDDATE = $2"/"$3"/"$4; OLDTYPE = $2; OLDCOUNT = 1 }\

   else { if($2 == OLDTYPE) { OLDCOUNT++ }\

      else {print OLDDATE"#"OLDCOUNT;\

      OLDDATE = $2"/"$3"/"$4;\

      OLDCOUNT = 1;\

      OLDTYPE = $2\

      }\

   }\
  
   }END{print OLDDATE"#"OLDCOUNT}' $DNAME/chartslash.txt > $DNAME/chartawk.txt 

   #

   #Now we sort this file numerically descending on the count field

   sort -t"#" -k 2 -n -r -o $DNAME/chartsort.txt $DNAME/chartawk.txt

   #

   #We use the dates for the chart bar labels

   #We build the gnuplot config file

   #

   #manipulate data for pareto chart


   #For the x labels use this awk command to create a file with 

   #name position pairs

   awk -F# '{print $1 "#" NR}' $DNAME/chartsort.txt > $DNAME/xlabels1.dat

   #

   #Word smith each line from xlabels1.dat into a gnuplot label format

   #This is a one line file with each name, position pair separated by a comma

   #and each name surrounded by quotes.

   awk -F# 'BEGIN{ORS=", "};{print "\"" $1 "\" " $2}' $DNAME/xlabels1.dat > $DNAME/xlabels2.dat

   #

   #replace the last "," with ")" avoiding any spaces at the end

   #We will use sed the stream editor and a Regular Expression pattern

   #The s means substitute /First pattern/for second pattern/

   #The first pattern is the comma , then [\ ][\ ]* for one or more spaces, the

   #backslash says interpret the next character as its literal meaning ie space

   #next is $ meaning the end of the line.

   #So our pattern means a comma and one or more spaces at the end of the line

   #

   #The substitute pattern is the literal meaning of )

   #

   sed -e s/,[\ ][\ ]*$/\)/ $DNAME/xlabels2.dat > $DNAME/xlabels3.dat

   #

   #add leading "set xtics rotate (" to tell Gnuplot to 
   put the labels sideways

   awk '{print "set xtics rotate (" $0}' $DNAME/xlabels3.dat > $DNAME/xlabels4.dat

   #
 
   #We create the Title from the Selection criteria using the directory variables

   if [ "$DNAME" == "$PREAD" ]

   then

      PTITLE="New Readers"

   fi

   if [ "$DNAME" == "$PRENEW" ]

   then

      PTITLE="Renewals"

   fi

   if [ "$DNAME" == "$PCONF" ]
   then

      PTITLE="Confirmations Sent"

   fi

   if [ "$DNAME" == "$PSENT" ]

   then

       PTITLE="Newsletters Sent"

   fi

   if [ "$DNAME" == "$PDROP" ]

   then

       PTITLE="Readers Dropped Out"

   fi

   #

   #We need to create the pareto settings file to include 
   the bar labels

   #and the Title

   #The pareto2.dem file is a template which contains 
   the Gnuplot settings

   cp $NHOME/pareto2.dem $DNAME/pareto.dem 

   #

   #If you need to print this file use lpr plot.sh

   #Notice it sets the terminal type to png and then 
   to the output file name

   #Notice the \" used to get echo to put a " into the 
   pareto.dem file.

   #This next line says echo ctrl@. 

   #To insert this symbol in vi press Ctrl-v and then Ctrl-@

   echo "^@" >> $DNAME/pareto.dem

   echo "set title \"$PTITLE\"" >> $DNAME/pareto.dem 

   echo "^@" >> $DNAME/pareto.dem

   echo "set terminal png" >> $DNAME/pareto.dem 

   echo "^@" >> $DNAME/pareto.dem

   echo "set output \"$DNAME/pareto.png\"" >> $DNAME/pareto.dem 

   echo "^@" >> $DNAME/pareto.dem

   cat $DNAME/xlabels4.dat >> $DNAME/pareto.dem 

   echo "^@" >> $DNAME/pareto.dem

   echo "plot \"$DNAME/pareto.dat\"" >> $DNAME/pareto.dem 

   #

   #For the plot data use this awk command which creates a file

   #with line numbers tabs and reject counts, telling gnuplot the

   #number of each bar column and its height

   awk -F# '{print NR "\t" $2}' $DNAME/chartsort.txt > $DNAME/pareto.dat

   #

   #Use Gnuplot with png output file, boxes option to chart file

   gnuplot $DNAME/pareto.dem

   #

   #end of the loop for the charts

done

#

#All the plots are saved in the corresponding tmp directory

#Another script will be used to create an html report for 
on screen viewing

#

#End of plot script

You can look up "info gnuplot" to find out how to make different charts and colours.

This next script creates the html file to report your plots. I made it by using Open Office and dragging the plots into place, resizing them then saving the file. I opened the html file in my editor to make the changes that resulted in this script.

The file rep1.txt looks like this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

<HTML>

<HEAD>

<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8">

<TITLE>rcharts</TITLE>

<META NAME="GENERATOR" CONTENT="OpenOffice.org 1.1.3 (Linux)">

<META NAME="CREATED" CONTENT="20041128;21591000">

<META NAME="CHANGED" CONTENT="20041128;22061500">

<STYLE>

<!--

H1 { color: #000000 }

P { color: #000000 }

A:link { color: #000099 }

A:visited { color: #990099 }

-->

</STYLE>

</HEAD>

<BODY LANG="en-GB" TEXT="#000000" LINK="#000099" VLINK="#990099" 
BGCOLOR="#ffffcc" DIR="LTR">

<H1>Newsletter Performance Charts</H1>

Select this link to view the report page layout.

$ crontab -e

This opens the vi editor for your crontab file.

Start typing by typing the "i" key to put you in "insert" mode. You can use the "delete" and the arrow keys to fix mistakes. To escape from insert mode use the ESC key. Be careful not to type anything in the ESC mode unless you know how to use vi, because the keys do editing commands in this mode. e.g. typing the x key deletes the letter under the cursor.

I presume you have typed the " i " key. The word " Insert" should appear at the bottom of the page.

Modify the sendletter.sh line and add the three following lines. Press the ESC key when finished.

SHELL=/bin/bash

#Run at 1pm every week day

0 13 * * 1-5 $HOME/newsletter/bin/extract.sh renewals renewal

#Run at 3pm every week day

0 15 * * 1-5 $HOME/newsletter/bin/extract.sh readers reader

#Run once on Wednesday at 10am in the first week of any month

* 10 1-7 * 3 $HOME/newsletter/bin/issues.sh 

#Run once on Tuesday at 9.15 except in the first week of any month 
stopping on the 27th

15 9 8-27 * 2 $HOME/newsletter/bin/sendletter.sh

#Run these two once on the 28th

5 9 28 * * $HOME/newsletter/bin/plot.sh

10 10 28 * * $HOME/newsletter/bin/report.sh

Now hold down the Shift key and type ZZ As long as you pressed the ESC key this will save and quit the vi editor.

Creating the newsletter is similar to creating the report html file, except the png files can be replaced with separate html files, each of which goes into a particular area of the layout, and can include their own graphics. To make it easy, create the whole layout as a table and put each file into one of its boxes. save the files and using the shell script method recreate the html file with shell variable names for each html file for each table box. If you like using style sheets, you can do a similar script using these instead. Then its just a matter of creating a new html file for each layout box and running the script to create the newsletter, or if you know the files will be there, you can use cron do create it for you.

This HowTo has demonstrated the use of cron for automating administrative tasks in a general work and home environment. It has shown how simply written UNIX/Linux shell scripts can get a computer to do very powerful automatic information processing of the type usually done by manual labour in a Microsoft environment. It has shown how to use Gnuplot to create charts of data, and how to create html files using the shell. It has also shown a lot of the shell tools that are commonly used for processing information in a work environment.

We shall be using these tools more and more as these HowTos continue, so I hope you will practice using them for doing everything you can think of.


Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com

Novell® Making IT Work As One™

© 2009 Novell, Inc. All Rights Reserved.