- 144 Dual Socket-Six Core Nodes (1728 total cores)
- 2.8 GHz Intel Xeon (Westmere X5660) processors
- 24 Gbytes memory per node (2 Gbytes per processor core) on most nodes
- Larger memory nodes available. See CHPC Policy 2.1.3 Ember Job Scheduling Policy for details.
- Mellanox QDR Infiniband interconnect
- Gigabit Ethernet interconnect for management
- There are also 11 GPU nodes. If you have a need for the GPU resources, please contact us.
CHPC resources are available to qualified faculty, students (under faculty supervision), and researchers from any Utah institution of higher education. Users can request accounts for CHPC computer systems by filling out an account request form. This can be found by following this link: account request form.
The general nodes of the Ember cluster are run in an unallocated manner. See https://www.chpc.utah.edu/documentation/policies/2.1GeneralHPCClusterPolicies.php#Pol2.1.3 for scheduling policies.
The Ember cluster can be accessed via ssh (secure shell) at the following address:
All CHPC machines mount the same user home directories. This means that the user files on Ember will be exactly the same as the ones on other CHPC clusters. The advantage is obvious: users do not need to copy files between machines.
Ember compute nodes mount the following scratch file systems:
- /scratch/ucgd/serial (restricted access)
- /scratch/ash/lustre (restricted access)
- /scratch/ucgd/lustre (restricted access)
As a reminder, the non-restricted scratch file systems are automatically scrubbed of files that have not been accessed for 60 days.
Your environment is setup through the use of modules. Please see the User Environment section of the General Cluster Information page for details in setting up your environment for batch and other applications.
The batch implementation on Ember is Slurm.
The creation of a batch script on the Ember cluster
A shell script is a bundle of shell commands which are fed one after another to a
tcsh,..). As soon as the first command has successfully finished, the second command is
executed. This process continues until either an error occurs or the complete array
of individual shell commands has been executed. A batch script is a shell script which
defines the tasks a particular job has to execute on a cluster.
Below this paragraph a batch script example for running in Slurm on the Ember cluster is shown. The lines at top of the file all begin with #SBATCH which are interpreted by the shell as comments, but give options to Slurm.
Example Slurm Script for Ember:
#SBATCH --time=1:00:00 # walltime, abbreviated by -t
#SBATCH --nodes=2 # number of cluster nodes, abbreviated by -N
#SBATCH -o slurm-%j.out-%N # name of the stdout, using the job number (%j) and the first node (%N)
#SBATCH --ntasks=16 # number of MPI tasks, abbreviated by -n # additional information for allocated clusters
#SBATCH --account=baggins # account - abbreviated by -A
#SBATCH --partition=ember # partition, abbreviated by -p # # set data and working directories
setenv WORKDIR $HOME/mydata
setenv SCRDIR /scratch/kingspeak/serial/UNID/myscratch
mkdir -p $SCRDIR
cp -r $WORKDIR/* $SCRDIR
# load appropriate modules, in this case Intel compilers, MPICH2
module load intel mpich2
# for MPICH2 over Ethernet, set communication method to TCP - for general lonepeak nodes
# see above for network interface selection options for other MPI distributions
setenv MPICH_NEMESIS_NETMOD tcp
# run the program
# see above for other MPI distributions
mpirun -np $SLURM_NTASKS my_mpi_program > my_program.out
For more details and example scripts please see our Slurm documentation. Also, to help with specifying your job and instructions in your slurm script, please review CHPC Policy 2.1.3 Ember Job Scheduling Policy.
Job Submission on Ember
In order to submit a job on ember one has to login first into a ember interactive node. Note that this is a change from the way job submission has worked in the past where you could submit from any interactive node to any cluster.
To submit a script named slurmjob.ember, just type:
To check the status of your job, use the "sinfo" command
For information on compiling on the clusters at CHPC, please see our Programming Guide.