Data transfer

Data transfer¶

All users have the option to transfer files to and from the systems. Instructions on how to do that can be found below.

Linux/Mac¶

The two most common commands you can use for data transfers over SSH:

• scp: for the full transfer of files and directories (only works fine for single files or directories of small/trivial size).
• rsync: a software application which synchronizes files and directories from one location to another while minimizing data transfer as only the outdated or inexistent elements are transfered (practically required for lengthy complex transfers, which are more likely to be interrupted in the middle). Of both, normally the second approach should be preferred, as more generic; note that, both ensure a secure transfer of the data, within an encrypted tunnel.

Using scp¶

scp or secure copy is probably the easiest of all the methods. The basic syntax is as follows:

$scp [-r] source_path destination_path

where the -r option copies the directories recursively.

Transfer from your local machine to the remote cluster home directory

Let’s assume you have a local directory ~/code/myfiles you want to transfer to the cluster, in your remote home directory:

$scp -r ~/code/myfiles username@hostname:

To transfer the local directory ~/code/myfiles to the directory ~/project:

$scp -r ~/code/myfiles username@hostname:project/.

Transfer from the remote cluster to your local machine

Let’s assume you want to transfer back the directory ~/project/code/myfiles to your local machine:

$scp -r username@hostname:project/code/myfiles /path/to/local/directory

See the scp(1) man page for more details.

Using rsync¶

An alternative to scp is rsync, which has the advantage of transferring only the files which differ between the source and the destination. This feature is often referred to as fast incremental file transfer. The typical syntax of rsync is similar to the one of scp:

$rsync -rv source_path destination_path

where the -v option enables the verbose mode and the -r option copies the directories recursively.

Transfer from your local machine to the remote cluster home directory

Coming back to the previous examples, let’s assume you have a local directory ~/code/myfiles you want to transfer to the cluster, in your remote home directory:

$rsync -rv ~/code/myfiles username@hostname:

To transfer the local directory ~/code/myfiles to the directory ~/project:

$rsync -rv ~/code/myfiles username@hostname:project/.

Transfer from the remote cluster to your local machine

Let’s assume you want to transfer back the directory ~/project/code/myfiles to your local machine:

$rsync -rv username@hostname:project/code/myfiles /path/to/local/directory

See the rsync(1) man page for more details.

Windows¶

WinSCP is a popular Secure File Transfer application (SFTP) for Windows computers which also includes support for SCP file transfers, and the ability to create secure tunnels via SSH.

The software can be downloaded here.

The installation of WinSCP is quite simple. Basically you need to click on the Next button several times leaving the default options.

In order to customize it to your likings you can choose between the Norton-Commander interface and the Windows Explorer-like interface. The main difference is that the first interface provides 2 panels (one with the local computer directory structure and the other with the remote) while the second shows only the remote.

Once you are ready with the installation, you can establish a secure connection from your local computer to the remote server as seen below. You just need to give the hostname of the cluster, your username and browse to find your private key:

During the establishment of the connection you will be prompted for the pass phrase of your key.

Transfer from your local machine to the remote cluster

1. In the right panel, open the directory on the remote computer to which you want to upload the file.

2. In the left panel, click once on a local file name to select it. Note: It is best to remove spaces, capitalization, and extensions from filenames before uploading them.

3. Upload the file or folder by using the mouse to drag the file to the remote server panel, by pressing the F5 function key on your keyboard, or by right clicking the file you want to transfer and selecting Copy from the pop-up menu.

Transfer from the remote cluster to your local machine

1. In the left panel, open a directory folder or location on your computer to which you want to download the file.

2. In the right panel, click once on a file or directory folder name to select it.

3. Download the file or folder by using the mouse to drag the remote file to your local directory, by pressing the F5 function key on your keyboard, or by right clicking the file you want to transfer and selecting Copy from the pop-up menu.