Data Transfer Services

Data Transfer

There are a number of options for the movement of data to and from CHPC storage, including:

  • direct mounts of home and group space on your local machine
    • For home directories:  We include the information about the location of your home directory space in the Account Creation Notification email sent to new users.    In all of the following you must replace <uNID> with your unid.  You can also get this information when on a CHPC resources. For home directories you do a df or df | grep <uNID>. As an example with a home directory if this returns  drycreek-vg5-0-lv1.chpc.utah.edu:/uufs/drycreek/common/drycreek-vg6-0-lv1/XX/<uNID> , then:
      • On Windows, you need to map a network drive using the path:  \\drycreek-vg6-0-lv1.chpc.utah.edu\XX-home\<uNID> .  NOTE - your username is ad\<uNID>.
      • On a MAC:  smb://ad;<uNID>@drycreek-vg6-0-lv1.chpc.utah.edu/XX-home/<uNID> 
      • On Linux: If you have root, you can cifs mount CHPC file spaces by (you will be promted for your password): 

        mount.cifs //drycreek-vg6-0-lv1.chpc.utah.edu/XX-home/<uNID>  /mnt -o user=<uNID>

    • For group directories: The information on group spaces is sent to the PI when a given group space is provisioned. You can also get this information by doing a cd to that space followed by  df -h | grep -B 1 groupspacename .  As an example, for a space called name-group1 if this command gives drycreek-vg4-0-lv1.chpc.utah.edu:/uufs/drycreek/common/drycreek-vg4-0-lv1/name then:
      • On Windows, you need to map a network drive using the path:  \\drycreek-vg4-0-lv1.chpc.utah.edu\name-group1
      • On a MAC:  smb://ad;<uNID>@drycreek-vg6-0-lv1.chpc.utah.edu/name-group1 
      • On Linux: If you have root, you can cifs mount CHPC file spaces by (you will be promted for your password): 

        mount.cifs //drycreek-vg6-0-lv1.chpc.utah.edu/name-group1  /mnt -o user=<uNID>

  • the use of simple tools such as WinSCP on Windows machines, CyberDuck on Macs to move data to/from a local machine
  • the use of linux commands and tools (scp, rsnyc over ssh, sftp, wget, curl are examples) on CHPC interative nodes or on the specialized Data Transfer Nodes (DTNs)
  • the use of specialized data transfer servers (DTNs) with specific high performance transfer tools such as Globus. Other high performance data transfer tools installed at CHPC include bbcp and fdt (see the application database for more information)
  • the use of CHPC's  Guest Transfer Service (for cases where an indivudual on one end of the data transfer does not have a CHPC account)

When working with small data transfers, the first  three of the options listed above are sufficient.  However, when moving larger amounts of data it is advantageous to move to the use of the specialized DTNs as well as to the use of the high performance data transfer tools.

A good resource for information on data transfer considerations are the Esnet Faster Data site; pages of specific interest are the ones with information on expected time to transfer data as well as on issues with using tcp for long range data transfers.

Data Transfer Nodes

Network traffic from most systems located at CHPC and at other locations on campus has traditionally passed through the campus firewall when communicating  with resources off campus. For many small data transfer usage cases, this traffic flow is acceptable.  However, large research computing workflows require more bandwidth requirements and more connections/sessions than the campus firewall can reasonably handle well.  These characteristics of research computing workflows can easily overwhelm the capacity of the campus firewall, impacting much of the day to day usage for the rest of campus.  To help address these research computing workflow needs, the campus has created a different network segment, which allows for high performance, low latency, and other special network and security characteristics.   CHPC offers a number of dedicated Data Transfer Nodes (DTNs) that utilize the Science DMZ for data transfers.  

The general DTNs are:

  • dtn01-dmz.chpc.utah.edu (connected at 40gbs)
  • dtn04-dmz.chpc.utah.edu (connected at 10gbs)
  • airplane01-dmz.chpc.utah.edu (connected at 40gbs)
  • airplane02-dmz.chpc.utah.edu (connected at 40gbs)
  • airplane03-dmz.chpc.utah.edu (connected at 40gbs)
  • airplane04-dmz.chpc.utah.edu (connected at 40gbs)

Any CHPC user can ssh to any of the above DTNs and use any of the command line options mentioned at the top of this page to move data to/from CHPC storage. In addition, users can utilize tools such as Globus (described below), Aspera, FDT, and other high performance tools. CHPC also supports additional specialized  tools for moving data to/from cloud storage.   Some of these tools are specific to a single cloud storage provider (such as s3cmd for Amazon cloud services), whereas others such as rclone work with different cloud storage providers.

CHPC also supports a number of group owned DTNs.  If you need any information about an existing group owned DTN or are interested in having a group dedicated DTN, please contact us.

NOTE: Large transfers are very dependent on the characteristics of the resources on both ends of the transfer.  If you need assistance in initiating, monitoring, or troubleshooting large transfers, you can reach out to CHPC via issues@chpc.utah.edu.

Data Transfer Services

Globus Online

The Globus service is comprised of a set of tools developed to facilitate parallel, load-balanced, fault tolerant data transfers. There are a few steps involved in getting set up to use this service.

Instructions:

  • Create a Globus account at: https://www.globus.org/ or you can use the "alternate login" and use your institutes identity provider & authenticate appropriately. 
  • Log in to your newly created Globus account.
  • Under "Quick Links" select "Transfer Files." You will be directed to a page with a look similar to the WinSCP interface.
  • Log in to an endpoint on each side of the interface: one side being the source and the other the destination.  
  • Next to the "Endpoint" box, click the "..." botton for a list of all possible endpoints. CHPC currently offers several public endpoints:

                         uofuchpc#dtn01 (single server attached at 40gbps)

                         uofuchpc#dtn02 (clustered endpoint consisting of 6 servers each attached at 10gbps)

                         uofuchpc#dtn04 (single server connected at 10gbps)

  • In most cases, users must authenticate to activate an endpoint. There are a few different possible authentication methods, but the most commonly used are either the organization authentication method or a federated authentication. The endpoints offered by CHPC use either uNID and campus password directly for authentication, or the InCommon federated authentication, which merely redirects users through the InCommon webpage to the U's CAS page to enter username and password.
  • Once authenticated, choose the desired files and/or directories and click the appropriate directional arrow button to start the transfer.
  • As the transfer is proceeding, click the "Activity" button and select your transfer and monitor progress. When the transfer completes, you will receive an email notifying you it is done and giving you statistics of the transfer.

 For further information please visit: https://www.globus.org/quickstart 

Guest Transfer

CHPC provides a mechanism for our users to transfer files to and from individuals without CHPC accounts. This service is called guest-transfer.

What is it for?

  • At times, CHPC users need to transfer files to or from individuals that don't have CHPC accounts. These files are often quite large (many gigabytes), and thus unsuitable for other transport mechanisms (email, DVD).
  • These file transfers often need to happen with little or no warning. They may also need to occur outside CHPC's support hours. Thus, the guest-transfer service must function without intervention or assistance from CHPC staff.

What is it not for?

  • The guest transfer service is not for repeated events.
  • The guest transfer service is not for long-term data storage.
  • The guest transfer service is not for restricted (PHI/HIPAA/HITECH/FERPA) or sensitive data.
  • The guest transfer service is not for huge data transfers (it's currently restricted to approximately 5 terabytes).

How do I get a guest account?

  • When you need to use the guest transfer service, visit  https://guest-transfer.chpc.utah.edu/ and fill out the form. This form creates a guest transfer account. You then gives the guest account username and password to your colleague. You and your colleague can now share files.

How do I use the service?

  • Once you have created a guest-transfer account and given it to your colleague, you and he copy your files to guest-transfer.chpc.utah.edu with your scp/sftp client (scp, sftp, WinSCP, etc).

Things to remember:

  • The process is completely automatic. You fill out the form, it immediately gives you a guest account.
  • Only CHPC users can create accounts.
  • The person who creates the guest account is responsible for all activity of the guest account.
  • This guest account is only usable for the guest transfer service. It provides no access to any other CHPC or University resources.
  • Files are transferred via scp/sftp. Interactive logins are disabled.
  • Files are automatically removed after 90 days (based on last-access time).
  • Files in the guest-transfer service can be read or changed by any other user.
  • Consider using encryption to protect and verify your files.
  • DO NOT USE THIS SERVICE FOR SENSITIVE OR RESTRICTED DATA!