Contents
Skip to end of metadata
Go to start of metadata

Overview

The data transfer between login.osgconnet.net and desktop (or laptop or some backup machine) may be performed to backup data or move data base for calculations.  We use scp, sftp or rsync for transferring data of less than 10 GB.  Globus transfer is highly recommended for transferring data of more than ten gigabytes. 

SCP

We can transfer files  using scp which  is a counterpart to ssh that allows for secure, encrypted file transfers between systems using your ssh credentials.

To transfer a file using scp, you'll need to run scp with the source and destination. Files on remote systems are indicated using user@machine:/path/to/file .

 

First, log in to OSG Connect:

ssh username@login.osgconnect.net #Connect to the login node with your username
passwd:       # your password

Once done, you can change to the data directory in your home area:

$ cd ~/data   

This directory is an area on Stash that you can use to store files and directories. It functions just like any other UNIX directory although it has additional functions that we'll go over shortly. Let's create a file :

$ echo "Hello world" > my_hello_world

 

Let's copy the file we just created from Stash to our local system (Desktop or laptop):

$ scp username@login.osgconnect.net:~/data/my_hello_world .

As you can see, scp uses similar syntax to the cp command. To copy directories using scp, you'll just pass the -roption to it. E.g:

$ scp -r username@login.osgconnect.net:~/data/my-directory .

 

Globus

 

Transfer between existing servers

From the portal.osgconnect.net web site, click the Transfer -> Start Transfer menu.

On one side, enter "osgconnect#stash" into the endpoint field, and press "Return" or the "Go" button.  

On the other side, enter the other endpoint from which you want to transfer data.

Browse the files and folders on the two endpoints to configure you transfer.

Optionally modify the advanced options below the two file browser panes.

Click on one of the arrows at the top to submit the transfer request: 

Go to the View Transfer window to see your transfer progress.  Click on the progress bar to drill in on transfer details.


Transferring between your laptop and OSG Connect Stash

To add your own laptop as an endpoint, so that you can transfer data to/from stash, you need to install Globus Connect, by clicking on the "Get Globus Connect" in the upper right corner of the Start Transfer page. More detailed instructions for installing Globus Connect can be found here: https://www.globusonline.org/globus_connect/.

 

For servers

Icon

To add your own server as an endpoint, so that any user on that server can transfer data to/from stash, you need to install Globus Connect Multiuser. Detailed instructions can be found here: https://www.globusonline.org/gcmu/. 

Using the Globus Online Command Line Interface

You can also use the Globus Online CLI via ssh <your-user-name>@cli.globusonline.org. More instructions on using the Globus Online CLI are here: https://www.globusonline.org/usingcli/.  

Using the Globus CLI from login.osgconnect.net

Icon

When logging into login.osgconnect.net, if you use the ssh -A flag, then you will be able to login from login.osgconnect.net to cli.globusonline.org.  This would be useful to run transfer scripts on login.osgconnect.net. For example:


More Information

There are a few other quick start guides here: https://www.globusonline.org/quickstart/.  

There's lots of good Globus Online material, including documentation, faqs, etc here: https://support.globusonline.org/forums

HTTP 

 

Making data accessible over HTTP

All user accounts on OSG-Connect have a directory that is automatically web accessible.  This directory is located at ~/data/public.  To make a file or directory accessible, copy it to this directory or a subdirectory of this directory and give files permissions of 644 and directories permissions of 755.  E.g. :

Making file accessible on HTTP

Manually Accessing Stash using HTTP

All the contents of the public directory are made available over HTTP.  Go to http://stash.osgconnect.net/+username to view the files and directory that you just made available in the previous section.  You can also use wget to retrieve the files, e.g:

Accessing files using wget
  • No labels