Updraft "Tricks and Tips"

By Martin Cuma

Last fall, CHPC took delivery of a new cluster called Updraft. It has 256 nodes connected with Qlogic InfiniBand network. Each node has two quad-core Intel Xeon 2.8 GHz processors and 16 GB of RAM. There is also a dedicated NFS file server with a total of 16 TB of disk space. While similar to our other clusters setups, there are some important specifics that we detail in this article.

Updraft has been bought partly with funds from the Institute for Clean and Secure Energy (ICSE), and as such, the cluster CPU allocation and disk space are divided between the ICSE group (uintah allocation) and general use (general allocation). The Uintah group gets 2/3 of the nodes and 12 TB of disk space at /scratch/uintah, the general allocation gets 1/3 of the nodes and 4 TB in /scratch/general. The Uintah group members get assigned to Quality of Service (QOS) qos=uintah, other users get qos=general. Priorities in these two qos's are the same, except that one qos can't run on the nodes allocated to the other qos. To facilitate possibility of running larger jobs we implemented special qos called "bigrun", which allows to use up to 256 nodes and is specified in the PBS job script as:

#PBS -l nodes=256:ppn=8,walltime=12:00:00,qos=bigrun

Users need to contact CHPC to be added to qos=bigrun.

Users can also request Dedicated Access Time (DAT) that grants exclusive use of the cluster. DATs are set for 48 hours (Monday 12pm to Wednesday 12pm), 2 times per month (the 1st and the 3rd weeks of each month) for Uintah users and the last week of the month for General users. Those interested in DAT need to contact CHPC in advance at issues@chpc.utah.edu.

A new feature that we implemented on the Updraft cluster is preemption of jobs. That is, a running job can be terminated if a job with higher priority is queued up. All freecycle jobs are preemptable, that is jobs that don't have any allocation (recall that CHPC allows users without CPU allocation to run at a very low priority in the freecycle mode). Note that a preempted job is not requeued – that is, user must manually resubmit a job that was preempted. Users with allocation can also specify their job to be preemptable. This will keep the job's priority high (so it'll run sooner), but, it can be preempted by another job. To specify job as preemptable, add qos=preemptable as:

#PBS -l nodes=16:ppn=8,walltime=12:00:00,qos=preemptable

The advantage of this setup is that preemptable job will get charged only ¼ of the allocation that it uses. This option is beneficial for users whose jobs are restartable and want to use as little allocation as possible. Some of our users are successfully using this option, having scripts or cron jobs automatically monitoring progress of their jobs and resubmitting them as necessary.

From the hardware standpoint, Updraft uses Intel Xeon CPUs, as opposed to AMD Opterons on our older clusters. From our extensive testing, Intel compilers produce the fastest code on the Intel CPUs, thus, we recommend using Intel compilers to build your codes. For details, see Updraft user's guide at http://www.chpc.utah.edu/docs/manuals/user_guides/updraft/.

While most of the application stack on Updraft is the same as on all other CHPC clusters, one difference is the MPI.
(continued on Page 6)
We use the Qlogic InfiniPath MPI binaries, that are installed as a part of the system, so, they reside in the standard paths (/usr/bin, /usr/lib64, etc.). InfiniPath was historically associated with Pathscale, which also makes compilers, so, by default, InfiniPath MPI tries to use Pathscale compilers.

To use Intel compilers instead, there are flags for each MPI compiler wrapper that let one specify preferred compiler. For example, mpicc -cc=icc. See Updraft user's guide for more details on this.

We are excited to have this new computational resource available for users and hope that this article both helps newcomers to come up to speed faster and experienced users to become more familiar with the specifics of the Updraft. If you have any questions on Updraft or anything else related to CHPC, please, contact us at issues@chpc.utah.edu.