Policies: CHPC Data Services
Many of the home directory file systems are based on NFS (Network File System), and proper management of files is critical to the performance of applications and the performance of the entire network. All files in home directories are NFS mounted from a fileserver, and a request for data must go over the network. Therefore, it is advised that all executables and input files be copied to a scratch directory before running a job on the clusters.
Home Directory Policies
- The general home directory file system (CHPC_HPC) is available to users who have CHPC account and do not have a department file system maintained by CHPC. This filesystem enforces quotas set at 50 GB per user. If you need a temporary increase on this limit, please let us know (issues@chpc.utah.edu) and we can increase it based on your needs.This file system is not backed up and users are encouraged to move important data back to a file system which is backed up, such as a department file server.
- The policies for quotas and backups of department file servers maintained by CHPC are worked out between CHPC and each department. Any backups run regularly by chpc have a two week retention period.
CHPC does NOT perform regular backups on the default HPC home directory space (CHPC_HPC).
Users are responsible for moving important data to a more permanent location such as their department fileserver.
In certain cases where CHPC supports a research group file server, backups are run on certain directories. These backups are performed as determined within each research group.
Scratch file systems (those with a path beginning /scratch or /tmp) are never backed up and files in these spaces are scrubbed periodically (see below).
While CHPC does NOT perform regular backups on the default HPC home directory space, we have recognized the needs of some groups to protect their data. CHPC has the ability to make periodic archive backups to tape of data for research groups. These archives can be no more frequent than once per quarter. Each research group is responsible for the cost of the tapes. Future archive backups can be made to the original tapes (more tapes may be needed if data set has grown), or a new tape purchase can be done with CHPC's assistance.
To schedule this service, please:
- send email to issues@chpc.utah.edu
- purchase tapes (CHPC will assist you with tape requirements).
- CHPC will perform the archive backup to tape.
- tapes can be stored at CHPC or can be stored by the PI.
CHPC suggests that group have 2 sets of tapes so that any time a full backup is being done to one set we still have the copy on the other set to protect us if the disaster were to happen mid archive.
***DISCLAIMER ON ARCHIVE BACKUPS***
These backups are made onto a high-density tape medium and are either retained in the CHPC backup library or can be delivered to the user for long-term storage at the user's discretion.
CHPC wishes to make a note here that high-density tapes do have a limited life span and so should not be regarded as a reliable medium for archive recovery after a certain reasonable length of time. For this reason we are going to specify that after two years, any archive tapes which the user wishes to still maintain for possible recovery should be replaced with new media.
CHPC can at that time furnish the user with a quote for the number of tapes which need to be updated, at which time we can make new duplicate tapes of the archive. which again may be stored for up to two years.
Scratch space for each HPC system is architected differently:
- NFS Scratch(/scratch/serial): This space is visible to all arches nodes and can be access with the path /scratch/serial. Each user will be responsible for creating directories and cleaning up after their jobs. This filesystem is not backed up. There are also /scratch/da, /scratch/mm, (/scratch/tr, /scratch/tr2) and /scratch/general file systems available on delicatearch, marchingmen, telluride and updraft respectively.
- Local Scratch space is the storage space unique to each individual node. Local Scratch is cleaned aggressively and is not supported by CHPC. It can be accessed on each node through "/tmp". This space will be the fastest, but not necessarily the largest. Users should use this space at their own risk.
Files on /scratch can be stored up to 30 days. All files unaccessed within 30 days on local scratch will be removed. Files in local, global or shared scratch space can be deleted at any time. Files on scratch filesystems are not backed up.
CHPC offers no guarantee on the amount of available /scratch disk space.
