PBS Resource Specification Options

Most typically the resources being requested in a PBS batch script will be given in the format of #PBS -l nodes=2:ppn=16 . IN this case the job is requesting two nodes that have 16 cores each. The ppn value must be a match to the number of physical core on a node in order for the job to be able to run on a given node; in this case only the nodes with 16 physical cores will be considered in finding nodes on which the job will run. However, as we now have clusters that have nodes with differing core counts, there are other options that one can use in order to increase the options in cases where the core count is not important; the use of the other options can often increase your workflow by opening up additional resources for a given job. Note that if the number of cores being used is fixed elsewhere, such as in the input file for your application, the additional resource specifications methods may not be an option.

Resource specification options

Using '#PBS -l nodes=X:ppn=y' will give you X nodes, each having Y cores. In this case the ppn specification must match the number of physical cores on the node(s) you are targeting. This is the best option if you want all the same type and a set number of cores per node. In this case the $PBS_NODEFILE has one entry per core.
A second option is that you can use '#PBS -l procs=X' where you specify the total number of cores you want. Again, in this case the $PBS_NODEFILE has one entry per core.
Yet another option that can be used on any cluster if you do not want to use all of the cores (for example, if memory is a limitation or you are doing bench marking on scaling within a node), then you have two options to use Y cores of each node, where Y can be any number from 1 to the number of cores available on a node. In both cases, the $PBS_NODEFILE is correct will have Y entries of each node received. Note that in this case your job is still charged as if it was using the entire node, as we do not allow node sharing.
1. 1. '#PBS -l nodes=X:ppn=Y -W x=nmatchpolicy:exactnode'
  2. '#PBS -l procs=X,tpn=Y'
If you want a set number of nodes, X, but you do not care if they are X 16 core nodes, X 20 core nodes, or a mixture, then you can use the " task request list" or trl option instead of requesting nodes. This option also works if you have a narrow range on the number of cores. In this case, as in case 1, the $PBS_NODEFILE has one entry per core. In order to get the number of cores for your mpirun -np flag you can query the number of lines in the $PBS_NODEFILE, e.g., set NCORE= `more $PBS_NODEFILE | wc -l`
For X nodes of any size you can use '#PBS -l nodes=X -W x=nmatchpolicy:exactnode'. Note that there is no ppn being set. This will give you X nodes and your $PBS_NODEFILE will have one entry per node. If you need this file to have one entry per core, as needed for mpi, then you can use the following script (first entry is for tcsh, second for bash) to create this file from the existing $PBS_NODEFILE. As above, once you have this file you can look at the line count to determine the number of cores.

# Description: This script takes the existing PBS_NODEFILE
# and converts it to a form that will allow mpirun to use all
# cores on all nodes allocated for the job. 
# The new name of the nodefile is $pbs_new.
#
# Usage: In your PBS batch script add the line
#        source <path-to-script>/pbsmangle.sh
#      OR add the following directly in your script
#
#################################################################
#inform the user that the script is in effect
echo -ne "Running the mixed node mangling script...."
#setup the new PBS_NODEFILE
export PBS_OLDNODEFILE=$PBS_NODEFILE
pbs_new=$HOME/$PBS_JOBID.mangled
rm -f $pbs_new
touch $pbs_new

#play it safe; assume that the PBS_NODEFILE is not formed properly
cat $PBS_NODEFILE | sort | uniq > /tmp/$PBS_JOBID.old
#for each line in $PBS_NODEFILE
for i in $(cat /tmp/$PBS_JOBID.old)
  do
    #get the info on the node, find the np line, then strip 'np =' and whitespace
    npcount=$(pbsnodes $i | grep np | sed 's/np =//' | sed 's/^[ \t]*//')
    for j in $(seq 1 $npcount)
      do
        echo $i >> $pbs_new
      done
  done
# do some cleanup
rm -f /tmp/$PBS_JOBID.old

#output useful things to the user for debugging
count=$(cat $HOME/$PBS_JOBID.mangled | wc -l)
ncount=$(cat $PBS_OLDNODEFILE | wc -l)
echo -e "Script done!"
echo "The file $HOME/$PBS_JOBID.mangled now contains references to $count cores on $ncount nodes and
should be used as your machinefile."
echo -e "If this is not what you expect, either remove the reference to this"
echo -e "script in your batch script or adjust your PBS paramters."
echo -e "The old PBS_NODEFILE is now assigned to PBS_OLDNODEFILE."

CHPC - Research Computing and Data Support for the University

University Information Technology