GPUs and Accelerators at CHPC

• P40 device of the Pascal generation. It contains 24 GB of global memory (GDDR5X) with a memory bandwidth of 346 GB/s. It has a single-precision performance of 12 TFlops; the double-precision performance is 0.35 TFlops. 
•  Titan V devices (Volta generation). Each device has 5120 CUDA cores and a global memory of 12 GB (HBM2). Its memory bandwidth is 652.8GB/s. Each device has the following performance specifics: 6.9 TFlops (Double Precision), 13.8 TFlops (Single Precision), 27.6 (Half Precision) & 110 TFlops (Tensor Performance - Deep Learning).
•  GTX 1080 Ti devices (Pascal generation). Each device contains 11 GB of global memory (GDDR5X) with a memory bandwidth of 484 GB/s. It also contains 3584 CUDA cores. Each GPU Card has a single-precision performance of 10.6 TFlops. The double-precision performance is only 0.33 TFlops.
•  RTX 2080 Ti devices (Turing generation). Each GPU device has 11 GB (GDDR6) global memory with a memory bandwidth of 616 GB/s. It also contains 4352 CUDA cores. The single-precision performance of each GPU device is 11.75 TFlops. Its double-precision performance is 0.37 TFlops.
• GeForce TitanX  devices (Maxwell generation). Each of these GPU devices has 12 GB of global memory. Each device has great single-precison performance (~7 TFlops) but does rather poor with double-precision (max ca. 200 GFlops).  Therefore, the TitanX nodes should be used for either single-precision, or mixed single-double precision GPU codes. 
• Tesla P100 devices (Pascal generation). Each device has 16 GB global memory. Each GPU card contains 56 multiprocessors with each 64 cores (3584 CUDA cores in total). The ECE support is (currently) disabled.The system interface is a PCIe Gen3 Bus. The double-precision performance per card is 4.7 TFlops. The single-precision is 9.3 TFlops anf the half-precision performance is 18.7 TFlops. 
• Tesla V100 (Volta generation). Each GPU has 16 GB of global memory.  Peak double-precision performance of a Tesla V100  is 7 TFlops; its peak single-precision performance is 14 TFlops. Its memory bandwidth is 900 GB/s. It also contains 5120 CUDA cores.
• Tesla A100 (Ampere generation). Each GPU has either 40 or 80 GB of global memory and a peak double-precision performance of 9.7 TFlops. The total memory bandwidth is 1555 Gb/s. There are 6912 CUDA Cores. 
• RTX 3090 (Ampere generation). Each GPU has 24 GB global memory and 10,496 CUDA cores and 328 Tensor cores. The peak single precision performace is 35.7 TFlops. The memory bandwidth is 936 Gb/s.
• Tesla T4 (Turing generation). Each GPU has 16 GB global memory, with 2,560 CUDA cores and 320 Tensor cores. The peak single precision performace is 8.1 TFlops, with 65 TFlops peak mixed precision performance. The memory bandwidth is 320 Gb/s.
• Tesla A40 (Ampere generation):  Each GPU has 48 GB global memory. The memory bandwith is 696 GB/s. There are 10,752 CUDA cores, 84 2nd generation Ray Tracing cores, and 336  3rd genereration Tensor cores.
• RTX 3090  (Ampere generation): Each GPU has 24 GB memory, 10,496 CUDA cores.
• Tesla A30 (Ampere generation):  Each GPU has 24 GB global memory and a peak double-precision performance of 5.2 TFlops. The memory bandwidth is 933 GB/s.
• RTX A6000 (Ampere generation):  Each GPU has  48 GB memory, 10,752 CUDA cores, memory bandwidth 768 GB/s.

The node notch060 has 16 physical CPU cores (dual Intel Xeon Silver 4110 CPU @ 2.10 GHz) and 96 GB memory.

The nodes notch[001-004, 055, 083-084] each have 32 physical CPUS cores (dual Intel Xeon Gold 6130 CPU @ 2.10 GHz) and 192 GB memory.

The nodes notch{081-082] each have 64 CPU physical cores (dual AMD EPYC 7601@ 2.2 GHz) and 512 GB memory.

The node notch085 has 32 physical CPU cores (dual Intel Xeon Silver 4216 CPU @ 2.10GHz) and 384 GB memory.

The nodes notch[086-089, 136, 204, 215, 271, 243} each have 40 physical cores (dual Intel Xeon Gold 6230 CPU @ 2.10GHz) and 192 GB memory.

The node notch102 has 40 physical CPU cores (dual Intel Xeon Gold 6230 CPU @ 2.10GHz) and 192 GB memory.

The node notch103 has 40 physical CPU cores (dual Intel Xeon Gold 6230 CPU @ 2.10GHz) and 384 GB memory.

The nodes notch[168, 169] each have 40 physical CPU cores (dual Intel Xeon Gold 6230 CPU @ 2.10 GHz) and 384 GB memory.

The node notch293 has 64 physical cores (dual AMD 7502 CPU @ 2.5 GHz) and 512 GB memory.

The node notch299 has 32 physical cores (single AMD 7502P 32-Core Processor) and 256 GB memory.

The node notch300 has 52 physical cores (dual Intel Xeon Gold 6230R CPU @ 2.10 GHz) and 768 GB memory.

The nodes notch{[308-309] each have 52 physical cores (dual Intel Xeon Gold 6230R CPU @ 2.10 GHz) and 512 GB memory.

The node notch328 has 48 physical cores (dual AMD 7413 CPU @ 2.65 GHz) and 256 GB memory.

The node notch329 has 64 physical cores (dual AMD 7543 CPU @ 2.8 GHz) and 512 GB memory.

The node notch330 has 128 physical cores (dual AMD 7713 CPU @ 2.0 GHz ) and 1024 GB memory.

The node notch 343 has 56 physical cores (dual Intel Xeon Gold 6330 CPU @ 2.0 GHz) and 256 GB memory.

The nodes notch[347-348] each have 56 physical cores (dual Intel Xeon Gold 6330 CPU @ 2.0 GHz) and 1024 GB memory.

The nodes notch[367-370] each have 64 physical cores (dual AMD 7513 @ 2.6 GHz) and 256 GB memory.

Note that the owner GPU nodes are accessible to all CHPC users via the  if and only if they use the SLURM partition notchpeak-gpu-guest if the owner groups leaves the nodes idle; the guest jobs are preemptible by slurm jobs from the owner group.

Also note that the nodes notch{081-082,308-309} are part of the notchpeak-shared-short partition open to all users, but with constraints on wall time, cores, memory and gpus allowed per user. 

The GPU nodes kp297 and kp298 each have two 6-core Intel Haswell generation CPUs (2.4 GHz) and 64 GB memory.

The remaining GPU nodes i.e., kp359, kp360, kp361 & kp362 have each 2 14-core Intel Broadwell processors (E5-2680 v4 running @ 2.4 GHz) and 256 GB RAM. These nodes are general nodes (i.e. owned by the CHPC). The GPU nodes kp{359-362} are owned by the School of Computing (SOC). Therefore, CHPC users outside SOC need to the use the SLURM kingspeak-gpu-guest partition if they want to get access the GPUS of kp{359-362} nodes.

Redwood contains two general compute nodes with GPU devices. Both compute nodes have 32 physical CPUS cores (Intel Xeon Gold 6130 CPU @ 2.10 GHz) and 192 GB memory.

There are also owner GPU nodes:

rw183 has 128 physical cores (dual AMD 7713 CPU @ 2.0 GHz ) and 1024 GB memory.

rw188 has 56 physical cores (dual Intel Xeon Gold 6330 CPU @ 2.0 GHz) and 512 GB memory.

rw[189-190] each have 56 physical cores (dual Intel Xeon Gold 6330 CPU @ 2.0 GHz) and 1024 GB memory.

There are 2 categories of GPU nodes:

• General nodes ( labeled Y in the last column of Table 1) i.e. nodes owned by the CHPC. The corresponding SLURM partition and account settings are:
  • --account=$(clustername)-gpu
  • --partition=$(clustername)-gpu 
  where $(clustername) stands for eithernotchpeak, kingspeak, or redwood.
• Owner nodes (non-general nodes) i.e. nodes owned by a research group (labeled N in the last column of Table 1).
  • For members of the groups who own GPU nodes a specific instance of a SLURM account & partition settings will be given to get access to these nodes. For example, the members of the group who own the nodes kp{359-362}, need to use the following settings:
    • --account=soc-gpu-kp
    • --partition=soc-gpu-kp
  • Users outside the group can also use these devices. The corresponding account & partition names are:
    • --account=owner-gpu-guest
    • --partition=$(clustername)-gpu-guest
    where $(clustername) stands for either notchpeak or kingspeak. Note that the jobs by users outside the group may be subjected to preemption. 

If one wants to access the GPU devices on a node one must specify the generic consumable resources flag (a.k.a. gres flag). The gres flag has the following syntax:

--gres=$(resource_type)[:$(resource_name):$(resource_count)]
where:

• $(resource_type) is always equal to gpu string for the GPU devices.
• $(resource_name) is a string which describes the type of the requested gpu(s) e.g. 1080ti, titanv, 2080ti, etc
• $(resource_count) is the number of gpu devices that are requested of the type $(resource_name)$.
  Its value is an integer in the closed interval: {1,max. number of devices on one node}
• the [ ] means optional parameters, that is, to request any single GPU (the default for the count is 1), regardless of a type, --gres=gpu will work. To request more than one GPU of any type, one can add the $(resource_count), e.g. --gres=gpu:2

The gres flag attached to each type of node can be found in the second-to-last column of Table 1.
For example, the flag --gres=gpu:titanx:5 must be used to request 5 GTX Titan X devices that can only be satisfied by the nodes kp297 and kp298. 

Note that if you do not specify the gres flag, your job will run on a GPU node (presuming you use the correct combination of the --partition and --account flag), but it will not have access the node's GPUs.

Some programs are serial, or able to run only on a single GPU; other jobs perform better on a single or small number of GPUs and therefore cannot efficiently make use of all of the GPUs on a single node. Therefore, in order to better utilize our GPU nodes,  node sharing has been enabled for the GPU partitions. This allows multiple jobs to run on the same node, each job being assigned specific resources (number of cores, amout of memory, number of accelerators). The node resources are managed by the scheduler up to the maximum available on each node. It should be noted that while efforts are made to isolate jobs running on the same node, there are still many shared components in the system.  Therefore a job's performace can be affected by the other job(s) running on the node at the same time and if you are doing benchmarking you will want to request the entire node even if your job will only make use of part of the node.

Node sharing can be accessed by requesting less than the full number of gpus, core and/or memory. Note that node sharing can also be done on the basis of the number of cores and/or memory, or all three. By default, each job gets  2 GB of memory per core requested (the lowest common denominator among our cluster nodes), therefore to request a different amount than the default amount of memory, you must use --mem flag . To request exclusive use of the node, use --mem=0.

When node sharing is on (default unless asking full number of GPUs, cores or memory), the SLURM scheduler automatically sets task to core affinity, mapping one task per physical core. To find what cores are bound to the job's tasks, run:

cat /cgroup/cpuset/slurm/uid_$SLURM_JOB_UID/job_$SLURM_JOB_ID/cpuset.cpus

Below is a list of useful job modifiers for use:

An example script that would request two Notchpeak nodes with 2xM2090 GPUs, including all cores and all memory, running one GPU per MPI task, would look like this:

#SBATCH --nodes=2
#SBATCH --ntasks=4
#SBATCH --mem=0
#SBATCH --partition=notchpeak-gpu
#SBATCH --account=notchpeak-gpu
#SBATCH --gres=gpu:m2090:2
#SBATCH --time=1:00:00
... prepare scratch directory, etc
mpirun -np $SLURM_NTASKS myprogram.exe

 To request all 8 3090 GPUs on notch328again using one GPU per MPI task, we would do:

#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --mem=0
#SBATCH --partition=notchpeak-gpu
#SBATCH --account=notchpeak-gpu
#SBATCH --gres=gpu:3090:8
#SBATCH --time=1:00:00
... prepare scratch directory, etc
mpirun -np $SLURM_NTASKS myprogram.exe

As an example, using the script below  will get four GPUs,  four CPU cores, and 8GB of memory.  The remaining GPUs, CPUs, and memory will then be accessible for other jobs.

#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --gres=gpu:titanx:4
#SBATCH --account=kingspeak-gpu
#SBATCH --partition=kingspeak-gpu

The script below will ask for 14 CPU cores, 100 GB of memory and 1 GPU card on one of the P100 nodes.

#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=14
#SBATCH --gres=gpu:p100:1
#SBATCH --mem=100GB
#SBATCH --account=owner-gpu-guest
#SBATCH --partition=kingspeak-gpu-guest

To run an interactive job, add the --gres=gpu option to the salloc command, e.g.

salloc -n 2 -N 1 -t 1:00:00 -p kingspeak-gpu -A kingspeak-gpu --gres=gpu:titanx:2

This will allocate the resources to the job, namely two tasks and two GPUs. To run parallel job, use srun or mpirun commands to launch the calculation on the allocated compute node resources. To specify more memory than the default 2GB per task, use the --mem option.

For serial, non-MPI jobs, utilizing one or more GPUs, ask for one node, e.g.

salloc -n 1 -N 1 -A owner-gpu-guest -p kingspeak-gpu-guest --gres=gpu:p100:1

This will allocate the resources to the job, namely one core (task) and one GPUs. To run the job, use the  srun command to launch the calculation on the allocated compute node resources. 

Note - April 2021: Due to a change made in slurm with regards to the allocation of gpus to a job: the nvidia-smi command now shows you only the gpus which have been assigned to your job instead of all of the gpus on the node.

Note - April 2022:  You can also check on the utilization of  the GPUs assigned to a batch job from an interactive node with the following command and the job number:

srun --pty --jobid XXXX nvidia-smi

This will return the nvidia-smi results for the GPU(s) assigned to your job.

Nvidia HPC SDK bundles Math libraries which can be used to offload computation to the GPUs and Communication libraries which can be used for fast communication between multiple GPUs. The Math libraries are located at $NVROOT/math_libs and the communication libraries are at $NVROOT/comm_libs.

To compile and link e.g. the cuBLAS, include the following parameters in the compilation line: -I$NVROOT/math_libs/include , and these parameters in the link line: -L$NVROOT/math_libs/lib64 -Wl,-rpath=$NVROOT/math_libs/lib64 -lcublas.

Nvidia HPC SDK and CUDA distributions include a terminal debugger named cuda-gdb. Its operation is similar to the GNU gdb debugger. For details, see the cuda-gdb documentation.

For out of bounds and misaligned memory access errors, there is the cuda-gdbmemcheck tool. For details, see the cuda-memcheck documentation.

The Totalview debugger that we license used to license and DDT debugger that we currently license also support CUDA and OpenACC debugging. Due to its user friendly graphical interface we recommend them for GPU debugging. For information on how to use DDT or Totalview, see our debugging page.

Profiling can be very useful in finding GPU code performance problems, for example inefficient GPU utilization, use of shared memory, etc. Nvidia CUDA provides a visual profiler called Nsight Systems(nsight-sys). There are also older, deprecated, command line profiler (nprof) and visual profiler (nvvp). More information is in the CUDA profilers documentation.

NOTE that the usage of GPU hardware counters is restricted due to a security problem in the Nvidia driver, as detailed in this Nvidia post. This error will be demonstrated by the following:

$ nvprof -m all ./my-gpu-program
==4983== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link
for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM

In this case please open a support ticket asking for a reservation on a GPU node. Our admins will enable the hardware counters profiling for all users inside of this reservation and instruct how to use it.