AuditDataStageJobLogs - Script & DataStage job to check the status of DataStage jobs

  • Home
  • /
  • AuditDataStageJobLogs - Script & DataStage job to check the status of DataStage jobs
AuditDataStageJobLogs - Script & DataStage job to check the status of DataStage jobs

AuditDataStageJobLogs - Script & DataStage job to check the status of DataStage jobs

Data stage 16 Nov 2014
Overview

In most of the applications with a batch cycle, IT and Business stakeholders will be interested to know the status of the batch cycle. It will be a cumbersome task to monitor the status of all cycles manually.

Here, lets walk through to an automated process to check the status of a given set of DataStage jobs and to create a formatted file with job status. This file can be used to email user with the status of the jobs. AuditDataStageJobLogs process has two parts

  1. Script(AuditDataStageJobLogs.sh) to pull the last two entries of a given job.

  2. DataStage job(AuditDataStageJobLogs.dsx) to process the output of AuditDataStageJobLogs.sh and to create a user friendly output file.

*** Source code for the .sh and .dsx can be downloaded by clicking the respective links.

Now lets take a deep dive into each of this

Script
#!/bin/sh
#######################################################################################################
#  Script Name       : AuditDataStageJobLogs.sh                                                       #
#  created by        : Entechlog                                                                      #
#  created date      : 26-Oct-2014                                                                    # 
#  Syntax            : AuditDataStageJobLogs.sh <ENVIRONMENT> <INPUTFILE>                             #
#  Input file format : SYSTEM_NAME, AUTOSYS_JOB_NAME, DATASTAGE_JOB_NAME, PROJECT_NAME                #
#                    : SYSTEM_NAME & AUTOSYS_JOB_NAME can be skipped by using UNKNOWN                 #
#  Example           : UNKNOWN, UNKNOWN, TestJob01, WAREHOUSE_DEV                                     #  
#######################################################################################################

#######################################################################################################
# Find the location of scripts and check for temp  and logs dir, create if don't exist                #
#######################################################################################################

THISDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -d "$THISDIR/logs" ]; then
   echo "logs directory exists"
else
   mkdir -p "$THISDIR/logs" 2>/dev/null
   chmod 755 "$THISDIR/logs"
fi

if [ -d "$THISDIR/temp" ]; then
   echo "tmp directory exists"
else
   mkdir -p "$THISDIR/temp" 2>/dev/null
   chmod 755 "$THISDIR/temp"
fi

#######################################################################################################
# Read Arguments                                                                                      #
#######################################################################################################

echo "Total arguments.." $#
if [ $# == 0 ];
then
  echo " "
  echo "Syntax: AuditDataStageJobLogs.sh <ENVIRONMENT> <FileNameWithProjectAndJobName>"
  echo " "
  echo "Usage:  Job to pull the status of Datastage job from Datastage Log Files"
  exit;
fi

ENVIRONMENT=$1
INPUTFILE=$2

echo ENVIRONMENT = $ENVIRONMENT
echo INPUTFILE = $INPUTFILE

FILE_NAME=$(echo $INPUTFILE | cut -d'.' -f1)
echo  FILE_NAME = $FILE_NAME

#######################################################################################################
# Location of log files                                                                               #
#######################################################################################################

LOGDIR=$(echo $THISDIR"/logs" | tr '[:upper:]' '[:lower:]')

TEMPDIR=$(echo $THISDIR"/temp" | tr '[:upper:]' '[:lower:]')

INPUTDIR=$(echo $THISDIR"/input" | tr '[:upper:]' '[:lower:]')

AuditDataStageJobLogs_parms=$INPUTDIR"/"$INPUTFILE

echo LOGDIR = $LOGDIR
echo TEMPDIR = $TEMPDIR
echo INPUTDIR = $INPUTDIR
echo AuditDataStageJobLogs_parms = $AuditDataStageJobLogs_parms

#export MAILLIST=replace_with_your_email_Address

#######################################################################################################
# Protect files created by this script.                                                               #
#######################################################################################################

umask u=rw,g=rw,o=rw

#######################################################################################################
# Read Current System time stamp                                                                      #
#######################################################################################################

CURR_DATE=`date "+%Y-%m-%d%H:%M:%S"`

#######################################################################################################
# Open up a new log for todays run								      #
#######################################################################################################

LOGFILE=$LOGDIR"/"$ENVIRONMENT"_"$FILE_NAME"_"$CURR_DATE".log"
TEMPFILE=$TEMPDIR"/"$ENVIRONMENT"_"$FILE_NAME".dat"

#######################################################################################################
# Delete previous tmp file                                                                            #
#######################################################################################################

rm $TEMPFILE > $TEMPFILE

#######################################################################################################
# Create and write the log header message                                                             #
#######################################################################################################

echo `date "+%Y-%m-%d%H:%M:%S"` "- AuditDataStageJobLogs.sh Started" > $LOGFILE

#######################################################################################################
# Insert Seperator                                                                                    #
#######################################################################################################

echo ' ' >>$LOGFILE

#######################################################################################################
# Run Script to List the job logs                                                                     #
#######################################################################################################

cd  `cat /.dshome`
. ./dsenv 
cd bin

while read job_details
do
     echo `date "+%Y-%m-%d%H:%M:%S"` "- Processing Job log for : " $job_details 	>>$LOGFILE

     system_name=$(echo $job_details | cut -d',' -f1)
     autosys_name=$(echo $job_details | cut -d',' -f2)
     batch_name=$(echo $job_details | cut -d',' -f3)
     project_name_org=$(echo $job_details | cut -d',' -f4)
     project_name=$(echo "${project_name_org/'$ENVIRONMENT'/$ENVIRONMENT}" | tr '[:lower:]' '[:upper:]')
     project_name_batch_name="$project_name $batch_name" 

     echo "system_name              : " $system_name					>>$LOGFILE
     echo "autosys_name	            : " $autosys_name					>>$LOGFILE
     echo "batch_name               : " $batch_name					>>$LOGFILE
     echo "project_name             : " $project_name					>>$LOGFILE
     echo "project_name_batch_name  : " $project_name_batch_name 			>>$LOGFILE
     echo ' ' >>$LOGFILE

     dsjob -logsum -type STARTED -max 2 $project_name_batch_name | tr "\\n" ",">>$TEMPFILE
     echo "">>$TEMPFILE

done < $AuditDataStageJobLogs_parms

#######################################################################################################
# write trailer log record                                                                            #
#######################################################################################################
echo `date "+%Y-%m-%d%H:%M:%S"` '- AuditDataStageJobLogs.sh finished' >> $LOGFILE

#mailx -s"Monthly Job Audit Status" $MAILLIST < $TEMPDIR/AuditDataStageJobLogs.dat
exit 0;
  • Script uses IBM DataStage command “dsjob -logsum” to pull the log details.
  • Script needs input directory in the same level as script is, with the input file in it. This is one of the prerequisite to run this script, The temp and logs directory will be auto created in same level as script
post thumb
  • Script needs 2 parameters as input - ENVIRONMENT and INPUTFILE.
    • Environment will be the environment or region(could be DEV, TEST, PROD as per your shop) and this will be the dynamic part of project name.
    • INPUTFILE will have the list of all jobs and respective project names in below format( Please ignore the first two fields named UNKNOWN. We have been using it to hold the system name and scheduler job name so that the output can be summarized in various format).
post thumb
  • Output and Logs files of this script will be captured in temp and logs dir respectively at same level, Below is the snippet of a sample output file.
post thumb
DataStage job

Download Link:(AuditDataStageJobLogs.dsx)

  • Processes the output from AuditDataStageJobLogs.sh. The prerequisite is script should be complete with RC 0.

  • Requires a cutoff time stamp. This cutoff time stamp will be used to compare against job start time and end time to drive the cycle status.

post thumb
  • Below is a sample output file which has the derived JOB_STATUS based on the cutoff time stamp and output from the script.
post thumb

Hope this helps !

Top

Share: