This demo provides specific examples of how to use the new tool, Motuz, to transfer data between Fred Hutch data storage locations.
Motuz is available at https://motuz.fredhutch.org. This link is only accessible from the campus network. If you are off campus you will need to use VPN.
Log in with your HutchNet ID and password.
At this time, Motuz allows copying of files/objects between the following file storage systems (with more to come):
- Shared file systems on campus (your home directory,
- Amazon S3
- Azure Blob Storage
- Google Cloud Bucket
- Swift (Economy Local Storage)
- Dropbox (Beta)
- Microsoft OneDrive (Beta)
This demo will focus on moving files between shared file systems on campus and Amazon S3.
Add a new Cloud Connection to Motuz
Amazon S3 Bucket Connection
Before we can use Motuz to move data, we need to add a Cloud Connection which will tell Motuz about our S3 account and bucket.
Every researcher at Fred Hutch has access to
an S3 bucket belonging to their lab. If your PI’s name is Jane Doe, your lab’s bucket would be called
Each researcher also has credentials to access this bucket. See the documentation for how to obtain these credentials.
In Motuz, click the
Cloud Connections link at the top of the browser window. Then click the
New Connections button.
Basic section, set
Amazon Simple Storage Service (s3). Give the connection a name such as
lab s3 bucket. Optionally enter the bucket name (
fh-pi-doe-j here is just an example).
Credentials section, paste your access key ID and secret access key from the credentials you obtained above. You can optionally click
Verify Connection to make sure the connection works. Then click
Create Cloud Connection.
To create a connection to a Swift location, you will need your Fred Hutch ID, your password and the
lastname_f for the PI whose Swift storage location that you have access to and the following information:
- Connection Name: Anything you’d like to show up in the lists
- Bucket Name: Unnecessary
- Auth URL: “https://tin.fhcrc.org/auth/v2.0”
- Tenant: Of the format: “AUTH_Swift_lastname_f” with lastname_f being the last name and first initial of the PI
- User: Your Fred Hutch ID
- Password/Key: Your Fred Hutch password (Note: you will need to update the connection when your password changes)
Copying Files to Cloud Locations
You can copy a single file or the contents of a directory.
The Motuz UI consists of two panes. You can actually copy in either direction, but in this case we will copy from left to right, and therefore use the button that points to the right.
First, select the files you want to copy. In this case I want to copy a directory. I navigate from my home directory by clicking the Path widget in the right-hand pane and navigating to the top level directory (
From there I can navigate to the location
of the files I want to copy, under
Note that each user has different permissions and you may not be able to access the files that I can access. Try to copy files in your lab’s subdirectory under
Then, in the right-hand pane, choose the cloud connection that you set up earlier:
If you want to copy to a sub-folder (prefix) of the S3 bucket, navigate there (this example uses a bucket you may not have access to; you will be seeing different files/folders in your PI’s bucket).
In the left-hand pane, single-click the folder you want to copy to S3:
Then click the blue arrow that points to the right:
You will see a confirmation screen like this that allows you to enter an optional description of the copy job:
Submit Copy Job button.
You’ll see the copy job’s progress shown in the bottom pane:
If all goes well, the copy will complete with the status
At any point during or after the copy, you can click this line in the status bar to see more information about the job status:
If you change your mind, you can cancel the copy job by clicking the
Stop Job button.
Copying from S3 to a shared directory on campus
This is similar to the previous steps; just find the directory you want to copy in the right hand pane, navigate to the place you want to copy to in the left hand pane, and click the blue arrow that points to the left:
Motuz also has an API; consult its documentation. Note that the API endpoint to use with Fred Hutch’s instance of Motuz is
scicomp or file an issue in the Motuz repository.
Updated: December 31, 2020Edit this Page via GitHub Comment by Filing an Issue Have Questions? Ask them here.