Job Management and Catalog-Based Data Access for Embarrassingly Parallel Applications

From EGEE-see WIki

Jump to: navigation, search

This Wiki page is a part of SEE-GRID Gridification Guide. It is contributed by Belgrade University Computer Centre.

Contents

[edit]

Introduction

This document is written with the purpose to serve as a tutorial on how to automatize an arbitrary application to the grid. All the scripts used in this document have been written in such manner, so that with some modifications, other applications can use them with minimal changes. It is especially useful for Embarrassingly Parallel Applications.

What are the steps that would require automation of application usage in the grid environment? Some of the most common operations include:

  • Creation of a number of similar jobs with different arguments and input files
  • Keeping track and checking status of many similar jobs
  • Manipulating many output files of different jobs in an organized manner

Workflow of an automated application in the grid environment
Figure 1.1 – Workflow of an automated application in the grid environment

[edit]

PROPEL Application

As an example, in this article you can find instructions on how to automatize the usage of PROPEL application in the grid environment. PROPEL (Asteroid Proper Elements) is an application designed to calculate proper elements for asteroids in the Asteroid belt of the Solar system. It is developed by the Astronomical Observatory of Belgrade, Serbia. Gridification of the application was done in collaboration between Astronomical Observatory of Belgrade and Belgrade University Computer Centre.

Two scripts will be introduced, submit_jobs that will handle creation and submission of jobs to the grid and check_jobs that will handle checking jobs status and management of output (result) files. Another common operation would be automated benchmarking of jobs. This is described in document Template Based Submission and Benchmarking of MPI Jobs, Varying Jobs and Job Sets.

Detailed workflow of automated PROPEL application in the grid environment
Figure 1.2 – Detailed workflow of automated PROPEL application in the grid environment

PROPEL application has the following input files:

  • barsunxx.inc
  • planxx.inc
  • *.cat – represents a cumulative asteroids catalogue file since the certain date (for instance, current one is juno.cat – jun 2007)
  • orb9.opt – represents an option file where asteroids listed are going to be processed

Application is run using the executable propel.x. Example run script can be found in Software_Installation_Management_for_PROPEL_Application. Output files are vast.fil and vpla.fil. This simple run will represent one run of a partial job.

[edit]

Proxy Certificate Creation

First thing that you need to do is create proxy certificate by logging in as a regular user. Example for AEGIS VO: 

$voms-proxy-init -voms aegis
[edit]

Creation of Partial Jobs

Before proceeding, you should read the following documents:

The idea behind automation of PROPEL application is to divide asteroids into segments of optional size, and then process one segment in each partial (sub) job. Partial jobs are identical except for a few different arguments which include: asteroid segment to be processed and CE where job will be executed. Because the partial jobs executions are independent, partial jobs are organized in the form of a job collection. This segments are then used to create orb9.opt option file that will contain the asteroids chosen for processing.

Similar idea can be used to create partial jobs for an arbitrary application. Above mentioned segments can be generalized into any sort of a string argument that will define a partial job. This argument can represent an argument of some executable file, or can represent a name of some file that will be used by a partial job.

Script submit_jobs does the following:

  • Reads the input where job options and partial jobs parameters are stored
  • Creates temporary JDL file containing the collection of partial jobs
  • Submits JDL file to the grid
  • Creates LFC directories where output files will be stored
  • Creates run scripts for each partial job
  • Creates an output file which will be used for check_jobs script

Example usage of submit_jobs script would be: 

$./submit_jobs submit_output < job-options.txt

job-options.txt is the input file containing arguments for partial jobs and submit_output is file that will be generated by submit_jobs and used by check_jobs script. Besides job-options.txt, submit_jobs also needs input.tar.gz file where the rest of the input files needed for PROPEL are located. This file is identical for all the partial jobs.

[edit]

Input for Partial Jobs

Input for creation of partial jobs is a file that can be redirected to the check_jobs script. Here is an example of this file: 

[options]
# Job options file example for submit_jobs script
# File must contain two sections
# First section is [options] section where run options are set, most of the
# options are mandatory for the script to work properly
# Second section is [subjob-parameters] section where actual job parameters
# are set. Format is ’CE SegmentID’
# To query available SEs use lcg-infosites –-vo aegis SE
# To query available CEs use lcg-infosites –-vo aegis CE
# and lcg-infosites -–vo aegis tag
VO aegis
APP_NAME propel
APP_VERSION 1.1
CATALOG_NAME juno.cat
LFC_HOST grid02.rcub.bg.ac.yu
DESTINATION_SE grid15.rcub.bg.ac.yu
SEGMENT_SIZE 200 # to be implemented
LCG_GFAL_VO aegis
[/options]
[subjob-parameters]
rti29.etf.bg.ac.yu:2119/jobmanager-pbs-aegis 000020
grid01.rcub.bg.ac.yu:2119/jobmanager-pbs-aegis 000450
cluster1.csk.kg.ac.yu:2119/jobmanager-pbs-aegis 000067
[/subjob-parameters]
[edit]

LFC Directories and Output Files

sumbit_jobs script creates LFC directory for collections of jobs depending on their asteroids catalogue name. Advantage of using LFC instead of SE directly is because LFC gives a much better control over a number of files that need to be organized in some sort of hierarchy, which is the case with the PROPEL application. Partial jobs are identified by their SegmentID, so output file names are prefixed by SegmentID. Partial jobs run scripts store (upload) files in the LFC and SE using the following naming convention: LFC directory example: 

/grid/aegis/propel-1.1/output/juno.cat/:
Contents of the directory:
-rw-rw-r--   1 211      101                  129504 Nov 24 03:55 
000020-vast.fil.gz
-rw-rw-r--   1 211      101                   23394 Nov 24 03:56 
000020-vpla.fil.gz
-rw-rw-r--   1 211      101                  129526 Nov 24 11:05 
000067-vast.fil.gz
-rw-rw-r--   1 211      101                   23394 Nov 24 11:05 
000067-vpla.fil.gz
-rw-rw-r--   1 211      101                  129504 Nov 26 04:08 
000300-vast.fil.gz
-rw-rw-r--   1 211      101                   23394 Nov 26 04:08 
000300-vpla.fil.gz
-rw-rw-r--   1 211      101                  129504 Nov 26 04:08 
000450-vast.fil.gz
-rw-rw-r--   1 211      101                   23394 Nov 26 04:08 
000450-vpla.fil.gz
[edit]

Checking Jobs Status and Manipulation of Output Files

check_jobs script offers following functionalities:

  • Checking for job results – script checks to see which of the partial jobs have submitted output (result) files to the LFC and reports the missing output files; this way user can easily find out which partial jobs need to be resubmitted
  • Retrieval of job results – script allows the user to download partial job results to the UI from LFC and SE, user can specify whether he/she wants to download all of the output files or just some
  • Checking of job status – script checks the status of partial jobs to see if jobs are still working or if some are aborted or have failed
  • Checking of job output information – script allows the user to download output information for partial jobs (console outputs); this can be useful for debugging in case some of the jobs have failed
  • Removal of output files – script allows the user to delete all or some of the output files from the remote LFC directory and SE

Input file for the check_jobs script is a file generated by the submit_jobs script which contains information about submitted partial jobs.

Example usage of check_jobs would be: 

$ ./check_jobs submit_output
1) Check job results
2) Get job results
3) Check job status (useful for debug info)
4) Get job output information (useful for debugging)
5) Remove job results from remote LFC catalog (careful!)
6) Quit
Select an option and press enter:
[edit]

Script Sources

Source of submit_jobs script is given below: 

#!/bin/bash

# VERSION 1.1
#
# Script for submitting multiply propel jobs to the grid
#
# Takes one optional argument (default value submit_output)- output
# filename where information about submitted jobs is stored needed
# for later analysis with "check_jobs" script
#
# example usage ./submit_jobs submit_output < job-options.txt

# Reading and checking number of parameters

test $# -gt 1 && \
echo "Usage: ${0##*/} [filename]" && \
echo " - [filename] is an optional file name (default is submit_output)" && exit 1

if [ $# -eq 0 ]; then
	SUBMIT_OUTPUT_FILENAME="submit_output"
else
	SUBMIT_OUTPUT_FILENAME=$1
fi

# checking if file [filename] already exists
test -f $SUBMIT_OUTPUT_FILENAME && \
echo "Warning! Filename $SUBMIT_OUTPUT_FILENAME already exists! It will be overwritten."

# Reading job options - [options] [/options] section, must exist in input
# file for script to work properly

read INPUT_LINE
if [[ -z $INPUT_LINE || $INPUT_LINE != "[options]" ]]; then
	echo "Error! Input must start with the [options] section."
	exit 1
fi

echo "Reading job options:"
while :
do
	read INPUT_LINE
	OPTION_VALUE=`echo $INPUT_LINE | gawk '{ print $2 }'`
	case `echo $INPUT_LINE | gawk '{ print $1 }'` in	# parse options from input
		"#"*)						# ignore comments
			;;
		"VO")
			VO=$OPTION_VALUE
			#echo $VO
			;;
		"APP_NAME")
			APP_NAME=$OPTION_VALUE
			#echo $APP_NAME
			;;
		"APP_VERSION")
			APP_VERSION=$OPTION_VALUE
			#echo $APP_VERSION
			;;
		"CATALOG_NAME")
			CATALOG_NAME=$OPTION_VALUE
			#echo $CATALOG_NAME
			;;
		"LFC_HOST")
			export LFC_HOST=$OPTION_VALUE
			#echo $LFC_HOST
			;;
		"DESTINATION_SE")
			DESTINATION_SE=$OPTION_VALUE
			#echo $DESTINATION_SE
			;;
		"LCG_GFAL_VO")
			export LCG_GFAL_VO=$OPTION_VALUE
			#echo $LCG_GFAL_VO
			;;
		"[/options]")				# options section finished reading
			echo "End of options."
			break
			;;
		*)
			echo "Unknown option (`echo $INPUT_LINE | gawk '{ print $1 }'`) ... Ignored." #unknown input, ignored...
			;;
	esac
done
# finished reading job options section

# Testing if all mandatory job options are set properly
# TODO

# finished testing job options

# Reading subjob parameters from [subjob-parameters] section and creating a
# temporary job collection jdl file, must exist in input for script to work

read INPUT_LINE
if [[ -z $INPUT_LINE || $INPUT_LINE != "[subjob-parameters]" ]]; then
	echo "Error! Mandatory [subjob-parameters] section not found."
	exit 1
fi

cat > propel_job_collection_tmp.jdl <<EOF	#Creating temporary job-collection file
[
 Type = "Collection";
 InputSandbox = {"run-subjob", "input.tar.gz"};
 DefaultNodeShallowRetryCount = 10;
 Requirements = Member("VO-$VO-$APP_NAME-$APP_VERSION", other.GlueHostApplicationSoftwareRunTimeEnvironment);
 Nodes = {
EOF

echo "Reading subjob parameters:"


cat > output2_tmp <<EOF		# Creating temp output file
LFC_HOST $LFC_HOST
LCG_GFAL_VO $LCG_GFAL_VO
CATALOG_NAME $CATALOG_NAME
APP_NAME $APP_NAME
APP_VERSION $APP_VERSION
VO $VO
[subjob-parameters]
EOF

COMMA_AFTER_FIRST="";
while :
do
	read INPUT_LINE
	case `echo $INPUT_LINE | gawk '{ print $1 }'` in 	# parse subjob parameters 								# from input
		"[/subjob-parameters]")
			echo "Finished reading subjob-parameters section."
			break
			;;
		*)	#Actual parameters, format DesiredCE DesiredSegment
			SELECTED_CE=`echo $INPUT_LINE | gawk '{ print $1 }'`
			SELECTED_OPTION_SEGMENT=`echo $INPUT_LINE | gawk '{ print $2 }'`

			#Adding option segment to temp output file
			cat >> output2_tmp << EOF
$SELECTED_CE $SELECTED_OPTION_SEGMENT
EOF
			#Adding subjob to temporary job-collection file
			cat >> propel_job_collection_tmp.jdl <<EOF
$COMMA_AFTER_FIRST
  [
    Executable = "run-subjob";
    StdOutput = "std.out";
    StdError = "std.err";
    OutputSandbox = {"std.out", "std.err"};
    Arguments = "$SELECTED_OPTION_SEGMENT";
    Requirements = Member("VO-$VO-$APP_NAME-$APP_VERSION", other.GlueHostApplicationSoftwareRunTimeEnvironment) && other.GlueCEUniqueID == "$SELECTED_CE";
  ]
EOF
			COMMA_AFTER_FIRST="  ,"
			;;
	esac
done

cat >> propel_job_collection_tmp.jdl <<EOF	#Closing up temporary job-collection file
 };
]
EOF

cat >> output2_tmp <<EOF			#Closing up temporary output file
[/subjob-parameters]
EOF

if [ -z $COMMA_AFTER_FIRST ]; then
	echo "Error! No subjob segments found in input."
	exit 1
fi

# finished reading subjob parameters


# creating output directory in LFC
lfc-mkdir /grid/$VO/$APP_NAME-$APP_VERSION/
if [ ! $? = 0 ]; then
	echo "Warning! Directory /grid/$VO/$APP_NAME-$APP_VERSION/ not created (it may already exist). Ignoring..."
fi
lfc-mkdir /grid/$VO/$APP_NAME-$APP_VERSION/output/
if [ ! $? = 0 ]; then
	echo "Warning! Directory /grid/$VO/$APP_NAME-$APP_VERSION/output/ not created (it may already exist). Ignoring..."
fi
lfc-mkdir /grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/
if [ ! $? = 0 ]; then
	echo "Warning! Directory /grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/ not created (it may already exist). Ignoring..."
fi
# finished creating LFC directory

# Creating job run script (run-subjob)
cat > run-subjob <<EOF
#!/bin/bash

SELECTED_OPTION_SEGMENT=\$1
export LFC_HOST=$LFC_HOST
export LCG_GFAL_VO=$LCG_GFAL_VO
DESTINATION_SE=$DESTINATION_SE
CATALOG_NAME=$CATALOG_NAME
APP_NAME=$APP_NAME
APP_VERSION=$APP_VERSION
VO=$VO

tar -xzf input.tar.gz

# Generate options file using Segment argument - TODO

# This is needed for application to see the required libraries
export PROPEL_LIB_DIR=\$VO_AEGIS_SW_DIR/$APP_NAME-$APP_VERSION/
# Run application
\$VO_AEGIS_SW_DIR/$APP_NAME-$APP_VERSION/propel.x

gzip vast.fil
gzip vpla.fil

# Store output in LFC catalog
# output is stored in LFC in following example format
# /grid/aegis/propel-1.0/output/juno.cat/
lcg-cr -v -d $DESTINATION_SE -P $APP_NAME-$APP_VERSION/output/$CATALOG_NAME/\$SELECTED_OPTION_SEGMENT-vast.fil.gz -l lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/\$SELECTED_OPTION_SEGMENT-vast.fil.gz file:\`pwd\`/vast.fil.gz
if [ ! \$? = 0 ]; then
	echo "Error sending output file (vast.fil.gz) to SE."
fi
lcg-cr -v -d $DESTINATION_SE -P $APP_NAME-$APP_VERSION/output/$CATALOG_NAME/\$SELECTED_OPTION_SEGMENT-vpla.fil.gz -l lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/\$SELECTED_OPTION_SEGMENT-vpla.fil.gz file:\`pwd\`/vpla.fil.gz
if [ ! \$? = 0 ]; then
	echo "Error sending output file (vpla.fil.gz) to SE."
fi

EOF
chmod +x run-subjob	# make sure its executable
echo "Job run script created (run-subjob). Do not delete until jobs are actually submitted."

# finished creating run script

rm -f output1_tmp
#submitting job collection to the grid
glite-wms-job-submit -a -o output1_tmp propel_job_collection_tmp.jdl
if [ ! $? = 0 ]; then
	echo "Error submitting job collection."
	exit 1
else 
	echo "Jobs successfully submitted!"
fi

# Adding jobid to output file
cat output1_tmp output2_tmp > $SUBMIT_OUTPUT_FILENAME
if [ ! $? = 0 ]; then
	echo "Error! Output data not saved."
	rm -f output1_tmp
	rm -f output2_tmp
	exit 1
else
	echo "Output data saved to $SUBMIT_OUTPUT_FILENAME."
	rm -f output1_tmp
	rm -f output2_tmp
fi

#echo "Debug mode, listing created jdl file..."
#cat propel_job_collection_tmp.jdl
echo "Deleting created jdl file..."
rm -f propel_job_collection_tmp.jdl

Source of check_jobs script is given below: 

#!/bin/bash

# VERSION 1.1
# Script for checking status and results of propel jobs submitted to the grid
#
# Takes one optional argument (default value submit_output) - input file name where
# information about submitted jobs is stored
#
# example usage ./check_jobs submit_output



# Function declarations

# Checks to see if all output files (identified by segments in the input file)
# are stored in LFC directory
checkJobResults(){
	local i=0
	echo "Files found in the remote LFC directory:"
	lfc-ls -l /grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/
	if [ ! $? = 0 ]; then
		return 1
	fi
	
	local file1Miss=""
	local file2Miss=""
	local missing=0
	echo "Checking to see if some result files are missing:"
	while [ $i -lt $N ]
	do
		file1Miss=""
		file2Miss=""
		lfc-ls /grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/${SEGMENTS[$i]}-vast.fil.gz &> /dev/null
		if [ ! $? = 0  ]; then # File not found!
			file1Miss="vast.fil.gz"
		fi

		lfc-ls /grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/${SEGMENTS[$i]}-vpla.fil.gz &> /dev/null
		if [ ! $? = 0  ]; then # File not found!
			file2Miss="vpla.fil.gz"
		fi
		
		if [[ -n $file1Miss || -n $file2Miss ]]; then # At least one file is missing
			echo "${SEGMENTS[$i]} segment missing (file: $file1Miss $file2Miss)"
			missing=1
		fi
		let "i += 1"
	done
	test $missing -eq 0 && echo "All result files are in LFC directory!"

	return 0
}

# Downloads result files out of the LFC directory for the jobs identified
# by the input file. Job results are downloaded into the directory 'output'
getJobResults(){
	local i=0
	local answer=""
	echo "Following segments where found in the input file:"
	while [ $i -lt $N ]
	do
		printf "${SEGMENTS[$i]} "
		let "i += 1"
	done
	echo ""
	
	echo "Choose which results you wish to download [(A)ll / SegmentID / (Q)uit]:"
	read answer
	case $answer in
		a|A)		# Downloading all segments defined in the input file
			mkdir output
			test ! -d output && echo "Error! 'output' exists and is not a directory." && return 1
			i=0
			while [ $i -lt $N ]
			do
				lcg-cp -v lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/${SEGMENTS[$i]}-vast.fil.gz file:`pwd`/output/${SEGMENTS[$i]}-vast.fil.gz
				lcg-cp -v lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/${SEGMENTS[$i]}-vpla.fil.gz file:`pwd`/output/${SEGMENTS[$i]}-vpla.fil.gz
				let "i += 1"
			done
			echo "Files saved in directory 'output'."
			;;
		q|Q)		# Quit, dont save anything
			;;
		*)		# Download results for segment specified by user
				# Possible use case problem is when user types SegmentID
				# not listed in prompt, for now this is allowed (user
				# can download some results present in the LFC
				# directory from some previous jobs)
			mkdir output
			test ! -d output && echo "Error! 'output' exists and is not a directory." && return 1
			local saveFailed=0
			lcg-cp -v lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/$answer-vast.fil.gz file:`pwd`/output/$answer-vast.fil.gz
			test ! $? = 0 && saveFailed=1
			lcg-cp -v lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/$answer-vpla.fil.gz file:`pwd`/output/$answer-vpla.fil.gz
			test ! $? = 0 && saveFailed=1
			if [ $saveFailed -ne 0 ]; then
				echo "Warning! Some (or all) result files were not successfully saved to directory 'output'."
			else
				echo "Files saved in directory 'output'."
			fi
			;;			
	esac
	return 0
}

# Deletes job results from the remote LFC directory
# jobs are identified by data in input file
removeJobResults()
{
	local i=0
	local answer=""
	local deleteFailed=0
	echo "Following segments where found in the input file:"
	while [ $i -lt $N ]
	do
		printf "${SEGMENTS[$i]} "
		let "i += 1"
	done
	echo ""

	echo "Choose which results you wish to delete from the remote LFC directory [(A)ll / SegmentID / (Q)uit]:"
	read answer
	case $answer in
		a|A)		# Deleting all segments defined in the input file
			i=0
			local ans=""
			echo "Are you sure you want to delete ALL result files (specified in the input file) from the remote LFC directory? [y/n]n"
			read ans
			if [[ "$ans" = "Y" || "$ans" = "y"  ]]; then
				while [ $i -lt $N ]
				do
					lcg-del -v -a lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/${SEGMENTS[$i]}-vast.fil.gz
					test ! $? = 0 && deleteFailed=1
					lcg-del -v -a lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/${SEGMENTS[$i]}-vpla.fil.gz
					test ! $? = 0 && deleteFailed=1
					let "i += 1"
				done
				if [ $deleteFailed -ne 0 ]; then
					echo "Warning! Some (or all) result files were not successfully deleted."
				else
					echo "Finished deleting result files."
				fi
			fi
			;;
		q|Q)		# Quit, dont delete anything
			;;
		*)		# Delete results for segment specified by user
				# Possible use case problem is when user types SegmentID
				# not listed in prompt, for now this is allowed (user
				# can delete some results present in the LFC
				# directory from some previous jobs)
				# Much more serious then download use case problem!
			local ans=""
			echo "Are you sure you want to delete result files (SegmentID = $answer) from the remote LFC directory? [y/n]n"
			read ans
			if [[ "$ans" = "Y" || "$ans" = "y"  ]]; then
				lcg-del -v -a lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/$answer-vast.fil.gz
				test ! $? = 0 && deleteFailed=1
				lcg-del -v -a lfn:/grid/$VO/$APP_NAME-$APP_VERSION/output/$CATALOG_NAME/$answer-vpla.fil.gz
				test ! $? = 0 && deleteFailed=1
				if [ $deleteFailed -ne 0 ]; then
					echo "Warning! Some (or all) result files were not successfully deleted."
				else
					echo "Finished deleting result files."
				fi
			fi
			;;			
	esac
	return 0

}

# end of function declarations

# Reading and checking number of parameters

test $# -gt 1 && \
echo "Usage: ${0##*/} [filename]" && \
echo " - [filename] is an optional file name (default is submit_output)" && exit 1

if [ $# -eq 0 ]; then
	SUBMIT_OUTPUT_FILENAME="submit_output"
else
	SUBMIT_OUTPUT_FILENAME=$1
fi

# checking if file [filename] exists
test ! -f $SUBMIT_OUTPUT_FILENAME && \
echo "Error! Filename $SUBMIT_OUTPUT_FILENAME does not exist." && exit 1

# Reading input file

N=-1	# Number of subjobs
while read INPUT_LINE; do
	OPTION_VALUE=`echo $INPUT_LINE | gawk '{ print $2 }'`
	# parse subjob parameters from input
	case `echo $INPUT_LINE | gawk '{ print $1 }'` in
		"#"*)			# ignore comments
			;;
		https*)			# JobID for the collection
			JOB_ID=`echo $INPUT_LINE | gawk '{ print $1 }'`
			#echo $JOB_ID
			;;
		"VO")
			VO=$OPTION_VALUE
			#echo $VO
			;;
		"APP_NAME")
			APP_NAME=$OPTION_VALUE
			#echo $APP_NAME
			;;
		"APP_VERSION")
			APP_VERSION=$OPTION_VALUE
			#echo $APP_VERSION
			;;
		"CATALOG_NAME")
			CATALOG_NAME=$OPTION_VALUE
			#echo $CATALOG_NAME
			;;
		"LFC_HOST")
			export LFC_HOST=$OPTION_VALUE
			#echo $LFC_HOST
			;;
		"LCG_GFAL_VO")
			export LCG_GFAL_VO=$OPTION_VALUE
			#echo $LCG_GFAL_VO
			;;
		"[subjob-parameters]")		# Reading subjob parameters
			N=0
			;;
		"[/subjob-parameters]")	# Finished reading subjob-parameters 
						# section
			break
			;;
		*)				# Actual subjob parameters but
						# only if N != -1
			if [ $N -ne -1 ]; then
				SEGMENTS[$N]=$OPTION_VALUE
				let "N += 1"
			fi
			;;
	esac
done < $SUBMIT_OUTPUT_FILENAME

# Test if all mandatory parameters are present
# TODO

# check if at least one subjob is found
test $N -lt 1 && \
echo "Error! No subjob parameters were found in the input file." && exit 1

# Create user menu with options

PS3="Select an option and press enter:"
select MENU_OPTION in "Check job results" "Get job results" "Check job status (useful for debug info)" "Get job output information (useful for debugging)" "Remove job results from remote LFC catalog (careful!)" "Quit"
do
	case $MENU_OPTION in
		"Check job results")
			checkJobResults
			if [ ! $? = 0 ]; then
				echo "Error! LFC directory not found."
				exit 1
			fi
			exit 0
			;;
		"Get job results")
			getJobResults
			exit $?
			;;
		"Check job status (useful for debug info)")
			glite-wms-job-status $JOB_ID
			exit 0
			;;
		"Get job output information (useful for debugging)")
			glite-wms-job-output --dir output-info $JOB_ID
			if [ ! $? = 0 ]; then
				echo "Error! Output info not saved."
				exit 1
			else
				echo "Output info saved in directory 'output-info'."
				exit 0
			fi
			;;
		"Remove job results from remote LFC catalog (careful!)")
			removeJobResults
			exit $?
			;;
		"Quit")
			exit 0
			;;
		*)
			echo "Error! Invalid option."
			exit 1
			;;
	esac
done
Retrieved from "http://wiki.egee-see.org/index.php/Job_Management_and_Catalog-Based_Data_Access_for_Embarrassingly_Parallel_Applications"
Personal tools