Submit massive and complicated serial jobs on cluster
######################################################################################
Example A: You need to allocate many computing cores to execute the same program, but each process will use different
input files or different input arguments. We use JOB_Array to accomplish this task.
Notes:
1. It will execute 20 times. It could be using 20 different processores, or it could just be using less number of processors repeatly.
2. In order to show flexibility as an example, the first ten job elements will run the program called "getargument", whereas the later ten job elements will run another program called "getargument2" ( see source of getargument.c and getargument2.c )
3. The output will be 20 files named with your $LSB_JOBID and $LSB_JOBINDEX and suffixed with ".out". The output will be generated on the local computing nodes /tmp directory, and will be copied back when the computing node finish its running
4. Three very important environmental variables $LSB_JOBID, $LSB_JOBINDEX, $LSB_HOSTS give you ability to control the workflow of the job execution.
5. Your job will be dispatched in order as long as resources for each single job element are available.
Now, create a new file called "script.lsf" with the following codes and then submit it by using "bsub < script.lsf"
#!/bin/bash
# Here is to define the job array, here I have 20 jobs
#BSUB -J testArray[1-20]
#BSUB -q normal
#BSUB -o "%J.%I.out"
#BSUB -e "%J.%I.err"
# how many CPU for each job element
#BSUB -n 1
#BSUB -u youremail@partners.org
#BSUB -N
# getting to the working dir
workdir=/shr/home/$USER/iotest
cd $workdir
# you can use $LSB_JOBINDEX to control how to use proprams, input file or output
# file in the array. The following example shows the first ten jobs will execute
# getargument and the next ten job will execute getargument2, each processor will
# get each different input argument. In your real application, you can use $LSB_JOBINDEX
# to identify your different arguments or input files
if [ $LSB_JOBINDEX -le 10 ] ; then
./getargument $LSB_JOBINDEX
else
./getargument2 $LSB_JOBINDEX
fi
# end of job script
|
