SG Job submission using Python WMProxy

From EGEE-see WIki

Our python submission codes were developed following the recommendations and examples from:

  http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=WMProxyAPI.APIDocumentation

In our job submission script, we used:

   from wmproxymethods import Wmproxy
   from wmproxymethods import JobIdStruct

in the beginning. In order for this to work you need to have some version of

  glite-wms-wmproxy-api-python 

installed.

Due to problem with the wms-proxy-delegation API, we had to issue:

    os.system("/opt/glite/bin/glite-wms-job-delegate-proxy  --endpoint 
    https://WMSHOSTNAME:7443/glite_wms_wmproxy_server -o wms_proxy --noint -d NAMEOFDELEGATIONID")

in order to delegate a proxy. Then we created a client object via:

    url="https://WMPSHOSTNAME:7443/glite_wms_wmproxy_server"
    client=Wmproxy(url)

A job is registered as

    delegationId="NAMEOFDELEGATIONID"
    (name taken from the os.system commandline).

A job jdl is similar to a jdl for commandline job submission. Example for our jdls (which start parametric jobs):

 jobjdl="[ requirements = other.GlueCEUniqueID == \""+ii+"\";
           RetryCount = 0; 
           JobType = \"parametric\"; 
           Parameters="+str(bulkjobs)+";
           ParameterStart=1;
           ParameterStep=1;
           Executable=\"salute_field.sh\"; 
           InputSandbox={\"gsiftp://se001.ipp.acad.bg:2811/
           pnfs/ipp.acad.bg/seegrid/salute_field.sh\"};
           OutputSandbox  = {\"std.out\",\"std.err\"}; 
           StdOutput=\"std.out\"; 
           StdError=\"std.err\"; 
           Environment ={\"SUPERBATCH_ID="+str(self.form.superbatchid.text())+"\",\"FPOLE=15\",
           \"LFC_HOST=grid02.rcub.bg.ac.yu\"}; 
           VirtualOrganisation = \"seegrid\";
           rank = 1;
           Type = \"job\"; 
          ]"
          
 jobId =client.jobRegister(jobjdl,delegationId)

You can obtain the jobId string as:

   jobIdstringform=jobIds.getJobId()
   print jobIdstringform

In the case of parametric jobs, there are child jobs, that can be obtained via:

    children=jobIds.getChildren()
    childrenjobs=[]
       for i in  children:
           childrenjobs.append(i.jobid)
           print i.jobid

We found useful to wrap the job submission with:

   try:
       jobsubmissioncode
       
   except:
       import traceback
       traceback.print_exc(file=sys.stderr)
       sys.stderr.flush()

Caveats:

At time of writing our code it was not possible to renew proxies via myproxy for job collections (like parametric or dag jobs). If this is important for the application, only single jobs must be used.