Data Transfer¶
One influential part of HPC is data, Big Data. Large datasets and tons of calculations against those datasets was one of the major reasons HPC was born. This is even more prevalent in a world full of sensors and datapoints.
There are several ways you can transfer data to Muscadine. These methods include:
Note
All forms of data transfer must be done on the head node. These methods will not work on any other node.
Each of these methods have their own specific usecases. The way they move data will heavily determine which one you’ll use for your use case. If you need help moving data, please refer to Getting Help to contact our admins.
Curl
Curl is a free and open-source computer program for transferring data to and from Internet servers. For single files from a http/https source, curl is the best.
Example use:
cd /work/users/$USER && curl -LO <insertLinkHere>
Git
Git also uses http/https, but does so for entire repositories. This works for any publicly hosted git repository including services like Github, Gitlab, and Gitea.
Example use:
cd /work/users/$USER && git clone <insertRepositoryHTTPLinkHere>
Open OnDemand
Open OnDemand, or just OOD, is a web-based interface for HPC clusters. OOD is the only supported graphical interface for transfering files to Muscadine.
First navigate to Muscadine-OOD and login using your HPC2 account.
After you are logged in, now you’ll hover over the Files tab in the navbar in the top right of the screen. Select the “Home Directory” option. You should now see an interactive page allowing you to upload to or download from your home directory on Muscadine.
SCP
SCP is pretty cool
Rsync
Drew really likes rsync…