pHPC account registration	pHPC user guide	pHPC services	pHPC web protal

OpenMP on pHPC cluster

Intro: OpenMP is an implementation of multithreading, a method of parallelization whereby the master "thread" (a series of instructions executed consecutively) "forks" a specified number of slave "threads" and a task is divided among them. The threads then run concurrently, with the runtime environment allocating threads to different processors.

The section of code that is meant to run in parallel is marked accordingly, with a preprocessor directive that will cause the threads to form before the section is executed. Each thread has an "id" attached to it which can be obtained using a function (called omp_get_thread_num() in C/C++ and OMP_GET_THREAD_NUM() in FORTRAN). The thread id is an integer, and the master thread has an id of "0". After the execution of the parallelized code, the threads "join" back into the master thread, which continues onward to the end of the program.

We encourage our user to explore more openMP in our RCCLU cluster, it contains mixture of 4-core and 8-core computing nodes. Parallelizing the loops in your applications by utilizing all the computing cores in these nodes can decrease your computing time almost linearly.

Here we give a quick example about how to write and submit your OpenMP jobs in RCCLU cluster, for more information of openMP program, you can visit

OpenMP Tutorial at Lawrence Livermore National Lab
OpenMP Tutorial at National Energy Research Scientific Computing Center (NERSC)


How to run OpenMP job on cluster

Step 1: Please read "how to setup your working environment" and do the following:

a. Include the sentence of "module load gcc/4.2/default" in the module-loading blocks in your .bashrc file (If you decide to use gcc with openMP, you have to make sure use gcc4.2 in the cluster. The default gcc in the cluster is gcc 3.4, which does NOT support OpenMP )

b. Type ". .bashrc" in your terminal, and then type "module list or type "which gcc" in your terminal again, you shall be able to see:

[testy@n137 ~]$ . .bashrc
[testy@n137 ~]$ module list
Currently Loaded Modulefiles:
   1) gcc/4.2/default

[testy@n137 ]$ which gcc
/source/gnu_4.2/bin/gcc
[testy@n137 ]$ which g++
/source/gnu_4.2/bin/g++

[testy@n137 ]$ echo $LD_LIBRARY_PATH
/source/gnu_4.2/lib64/:/source/gnu_4.2/lib/:/source/gnu_4.2/lib/gcc/x86_64-unknown-linux-gnu/4.2.2:/source/gnu_4.2/lib/:

[testy@n137 ~]$ bsub -I -n1 /bin/bash
Job <668932> is submitted to default queue .
Waiting for dispatch ...
Starting on < n60 >

[testy@n137 ~]ssh n60
[testy@n60 ~]make -f testOpenMP.mak
[testy@n60 ~]./testOpenMP
D: checking arguments: N_args=1
D: arg[0] = ./testOpenMP.exe
Allocating memory ...
array size in MB = 7.629 MB
 (remember, you have 2 of these)normalization a: 0.05510,  b: 0.00173
0.0334, 0.0339, 0.0328, 0.0331, 0.0341, 0.0334, 0.0342, 0.0328, 0.0325,
0.0335, 0.0341, 0.0335, 0.0335, 0.0330, 0.0339, 0.0335, 0.0336, 0.0323,
0.0331, 0.0328, 0.0332, 0.0336, 05
 33.386699438
milestone 0 to 1 time=0.021 seconds
milestone 1 to 2 time=0.040 seconds
milestone 2 to 3 time=0.048 seconds
milestone 3 to 4 time=2.595 seconds
milestone 4 to 5 time=0.000 seconds

#!/bin/bash

# enable your environment, which will use .bashrc configuration in your home directory
#BSUB -L /bin/bash

# the name of your job showing on the queue system
#BSUB -J myjob

# the following specify the queue that it will use, "normal" queue will dispatch your job onto 4-core nodes
# whereas queue "8core" will dispatch your job only to 8-core nodes

#BSUB -q normal


#the computing core number for the 4-way multithread job
#BSUB -n 4

#this following line will specify all the cores will land on one host
#BSUB -R "span[hosts=1]"

# the system output and error message output, %J will show as your jobID
#BSUB -o %J.out
#BSUB -e %J.err




#when job finish that you will get email notification
#BSUB -u YourEmail@partners.org
#BSUB -N


#enter your working directory
cd /shr/home/$USER/


# Finally, Start the program
./testopenMP.exe

[testy@n137 ~]$ bsub < testopenMP.lsf