Job output monitoring using job perusal
From EGEE-see WIki
gLite WMS has the option (called job perusal) to monitor output files while the job is still running. To enable perusal, two options need to be added to the standard job JDL:
Executable = "perusal.sh";
StdOutput = "stdout.log";
StdError = "stderr.log";
InputSandbox = "perusal.sh";
OutputSandbox = {"stdout.log", "stderr.log", "results.txt"};
PerusalFileEnable = true;
PerusalTimeInterval = 60;
PerusalFileEnable enables perusal altogether and PerusalTimeInterval defines how ofter are the output files copied back to the WMS. A third option not used in the example is also available - PerusalFilesDestURI allows an upload of files to some file server other than the WMS.
Example script used to test perusal periodically writes to standard output, standard error and a file called results.txt:
#!/bin/sh while true; do date voms-proxy-info >&2 df >> results.txt sleep 30 done
The job can now be submitted as usual using glite-wms-job-submit. When the job lands on the WN, WMS job wrapper starts a perusal process in parallel with the user job. Perusal process periodically (as defined by PerusalTimeInterval) downloads a list of files to be tracked from the WMS (/<JobSandbox>/peek/files2peek) and uploads the listed files to the WMS (/<JobSandbox/peek directory). Note that files are tracked by size and uploaded in chunks - only the file part that was appended since the last upload is sent.
By default the list of files to be tracked by the perusal process on the WN is empty. The names of tracked files are set using glite-wms-job-perusal with the --set parameter:
$ glite-wms-job-perusal --set -f stdout.log -f stderr.log -f results.txt https://wms.phy.bg.ac.yu:9000/LX93Pkurwtm-hCx5e9TTbA Connecting to the service https://147.91.84.25:7443/glite_wms_wmproxy_server Connecting to the service https://147.91.84.25:7443/glite_wms_wmproxy_server ====================== glite-wms-job-perusal Success ====================== Files perusal has been successfully enabled for the job: https://wms.phy.bg.ac.yu:9000/LX93Pkurwtm-hCx5e9TTbA ==========================================================================
This command creates a files2peek file on the WMS that is subsequently downloaded by the perusal process running on the WN.
File parts are downloaded from the WMS using glite-wms-job-perusal with the --get parameter.
$ glite-wms-job-perusal --get -f stdout.log https://wms.phy.bg.ac.yu:9000/LX93Pkurwtm-hCx5e9TTbA Connecting to the service https://147.91.84.25:7443/glite_wms_wmproxy_server Connecting to the service https://147.91.84.25:7443/glite_wms_wmproxy_server ====================== glite-wms-job-perusal Success ====================== The retrieved files have been successfully stored in: /tmp/vvidic_LX93Pkurwtm-hCx5e9TTbA ========================================================================== -------------------------------------------------------------------------- file 1/1: stdout.log-20071102144924_3-20071102160452_73 -------------------------------------------------------------------------- Fri Nov 2 14:48:34 CET 2007 Fri Nov 2 14:49:04 CET 2007 Fri Nov 2 14:49:35 CET 2007 Fri Nov 2 14:50:05 CET 2007 ...
The command will instruct WMS to assemble all parts not seen since the last download (parts 3-73 in this case) and return the URL to the assembled file. The assembled file is then downloaded into a local directory and displayed on standard output. All file parts (seen and new) can be downloaded by adding --all parameter to the previous command.
In order to disable perusal use the --unset parameter of the glite-wms-job-perusal.
$ glite-wms-job-perusal --unset https://wms.phy.bg.ac.yu:9000/LX93Pkurwtm-hCx5e9TTbA Connecting to the service https://147.91.84.25:7443/glite_wms_wmproxy_server Connecting to the service https://147.91.84.25:7443/glite_wms_wmproxy_server ====================== glite-wms-job-perusal Success ====================== File(s) perusal has been successfully disabled for the job: https://wms.phy.bg.ac.yu:9000/LX93Pkurwtm-hCx5e9TTbA ==========================================================================
This command disables the perusal altogether by clearing the list of files to be tracked (files2peek). As a result, perusal process on the WN will stop uploading file parts to the WMS. Perusal can be enabled again using the --set parameter as shown before.
