AuditDataStageJobLogs - Script & DataStage job to check the status of given set of DataStage jobs

Sunday, November 16, 2014

AuditDataStageJobLogs - Script & DataStage job to check the status of given set of DataStage jobs

In most of the applications with a batch cycle, IT and Business stakeholders will be interested to know the status of the batch cycle. It will be a cumbersome task to monitor the status of all cycles manually.

Here, lets walk through to an automated process to check the status of a given set of DataStage jobs and to create a formatted file with job status. This file can be used to email user with the status of the jobs. AuditDataStageJobLogs process has two parts
  1. Script(AuditDataStageJobLogs.sh) to pull the last two entries of a given job.
  2. DataStage job(AuditDataStageJobLogs.dsx) to process the output of AuditDataStageJobLogs.sh and to create a user friendly output file.
Source code for the .sh and .dsx can be downloaded by clicking the respective links.

Now lets take a deep dive into each of this

Script(AuditDataStageJobLogs.sh)
  • Script uses IBM DataStage command "dsjob -logsum" to pull the log details.
  • Script needs input directory in the same level as script is, with the input file in it. This is one of the prerequisite to run this script, The temp and logs directory will be auto created in same level as script is.
  • Script needs 2 parameters as input - ENVIRONMENT and INPUTFILE.
  • Environment will be the environment or region(could be DEV, TEST, PROD as per your shop) and this will be the dynamic part of project name.
  • INPUTFILE will have the list of all jobs and respective project names in below format( Please ignore the first two fields named UNKNOWN. We have been using it to hold the system name and scheduler job name so that the output can be summarized in various format).
  • Output and Logs files of this script will be captured in temp and logs dir respectively at same level, Below is the snippet of a sample output file.

DataStage job(AuditDataStageJobLogs.dsx) 
  • Processes the output from AuditDataStageJobLogs.sh. The prerequisite is script should be complete with RC 0.
  • Requires a cutoff time stamp. This cutoff time stamp will be used to compare against job start time and end time to drive the cycle status.
  • Below is a sample output file which has the derived JOB_STATUS based on the cutoff time stamp and output from the script.
Hope this helps !

Error 404

The page you were looking for, could not be found. You may have typed the address incorrectly or you may have used an outdated link.

Go to Homepage