Frequently Asked Questions

Getting an Account

Accessing the System

Using the iRods System

Backup and Restoration

Getting an Account

How do I get an account on the iCat server?

Send a request to oit_datascience@help.ncsu.edu, with the subject line “Data Science”

How do I get an account at the HPC?

Go to HPC’s website and click on the big red “Get Access” button.

How do I get access to iRods from an HPC login node? How do I access the HPC-VCL image?

Send a request to oit_datascience@help.ncsu.edu, with the subject line “HPC VCL”. When access has been granted, make a reservation for the HPC-VCL image (HPC (CentOS 7.1 64 bit VM)) at VCL’s website, vcl.ncsu.edu. When logging in for the first time, initialize iRods using the instructions below; “How to initialize the HPC-VCL login node for iRods use?”

How do I initialize the HPC-VCL login node for iRods use?

In your home directory, create a file called $HOME/.irods/irods_environment.json. The content of this file should be:
{
“irods_authentication_scheme”: “PAM”,
“irods_host”: “irods01.hpc.ncsu.edu”,
“irods_port”: 1247,
“irods_zone_name”: “dsai_ncsu”
}
Then finally issue the command “iinit”. You should then enter your unity username and password.

How do I get access to the HPC/ROI storage space? (100 TB /roi_share/username)

Send a request to oit_datascience@help.ncsu.edu, with the subject line “Data Science”

Accessing the System

How do I login to the HPC?

Windows

Use PuTTY to connect to the HPC system via SSH. PuTTY can be downloaded from here.

Once downloaded, create a connection with the username being your Unity account name and the address being login.hpc.ncsu.edu. When prompted for a password, use your Unity account password.

MAC/Linux

Use the Mac or Linux terminal to connect to the HPC system via SSH. Your username and password are your Unity account credentials.

To connect, open a terminal and type: ssh unityid@login.hpc.ncsu.edu

How do I access my data on the iRods datagrid?

From any browser, use your Shibboleth credentials to log in at https://irods-cloudbrowser.hpc.ncsu.edu This web server will allow you limited control of your data: Upload, download, manually create metadata etc. For more detailed control, like using command line controls to run iRules, log into the the iRods client.

How do I log in to the iRods client, how do I use iCommands client?

  1. Go to vcl.ncsu.edu and click on Make a Reservation
  2. Use your Shibboleth credentials to login
  3. Click on the Reservations tab
  4. Click on the New Reservations button
    1. From the environments dropdown, select Centos 6 with iRods Client
    2. Choose a time for which you want to schedule the reservation
  5. Wait for the reservation to be created. Once it is ready, you will see a button to connect. Click this and will see a popup. At the bottom of the popup, you will see a button for RDP File. Click this and open it within Windows.
  6. Login with you Shibboleth credentials.

You are now connected to the data grid. “ils” will list all of your files on the data grid.

More Information about using the data grid and icommands can be found here (page 18 onwards.)

How do I log in to the iRods client (newer image)?

    1. Go to vcl.ncsu.edu and click on Make a Reservation
    2. Use your Shibboleth credentials to login
    3. Click on the Reservations tab
    4. Click on the New Reservations button
      1. From the environments dropdown, select iRODS Client, Centos 7.3
      2. Choose a time for which you want to schedule the reservation
    5. Wait for the reservation to be created. Once it is ready, you will see a button to connect. Click this and you will be able to ssh in.

Login with your Shibboleth credentials.

  • Initialize the client by creating the file $HOME/.irods/irods_environment.json, with the content:
    {
    “irods_authentication_scheme”: “PAM”,
    “irods_host”: “irods01.hpc.ncsu.edu”,
    “irods_port”: 1247,
    “irods_zone_name”: “dsai_ncsu”
    }

Type “iinit”.

You are now connected to the data grid. “ils” will list all of your files on the data grid.

More Information about using the data grid and icommands can be found here (page 18 onwards.)

How do I get files from the iCat data grid to my iRods client reservation?

On the iRods client, execute: iget filename or iget -f filename to force overwrite.

More Information about using the data grid and icommands can be found here (page 18 onwards.)

How do I put files onto the iCat data grid from my iRods client reservation?

On the iRods client, execute: iput filename or iput -f filename to force overwrite.

More Information about using the data grid and icommands can be found here (page 18 onwards.)

How do I transfer files from an anonymous ftp server (which as a specific URL) to my HPC:/roi_share/username space?

Log into the HPC and go to /roi_share/username. Then use wget to transfer files, like:

wget ftp://path/to/data.file

How do I install software on the iRods client? How do I use “yum install?”

After an irods-client vcl image is reserved and iRods is initialized (see above): Open a terminal window by going to top left menu and selecting Applications > System Tools > Terminal. Then use sudo yum install package to install package.

How do I transfer files from my HPC:/roi_share/username to my iRods data grid space?

Start up the HPC-vcl image at vcl.ncsu.edu and use it to log into the HPC. Then start Firefox browser and go to url https://irods-cloudbrowser.hpc.ncsu.edu. Files can now the transferred between the iRods zone and any HPC directory.

How to allow scp on an iCat development server (or any vcl reservation)?

Update iptables to accept files sent via SCP from a particular iRods client.

Here is an example for the iRods client irods01.hpc.ncsu.edu:

sudo /sbin/iptables -I INPUT -s irods01.hpc.ncsu.edu -p tcp --dport 22 -j ACCEPT

Using the System

Where is the iRods user documentation?

https://irods.org/uploads/2015/06/GettingStartedwiRODS4.1.pdf
.

How do I upload files to the iRods data-grid (iCat server)?

From any browser, use your Shibboleth credentials to log in at https://irods-cloudbrowser.hpc.ncsu.edu This web server will allow you limited control of your data: Upload, download, manually create metadata etc. For more detailed control, like using command line controls to run iRules, log into the the iRods client.

How do I share an iRods directory of data with people in another zone?

From the VCL iRods Client, do (Renci is used in this example):
ichmod -r read username-at-renci#dsai_renci directory-to-be-shared
.

How to create/debug/experiment/play around with workflow rule automation without risk of damaging the iCat server, and without bugging the System Administrator?

Users have complete control of their iRods Client reservation, including root access.  They can be as carefree as they want with it.  They can install any software they want on it (sudo yum install anything).  They can create, experiment with and run any rules they want from the client without risk of hurting the iCat server (the shared resource).

However, implementing rule automation via Policy Enforcement Points (triggering a workflow automatically, like executing a workflow whenever a file is uploaded to the data grid) requires administrator access to the iCat data grid server.  Users can use VCL to spin up their own copy of an iCat server, and there they can experiment with PEP and rules as much as they want.  When the rules, automation and triggers are debugged on this iCat server copy, the system administrator can then copy the rules, automation and trigger scripts to the “real” shared iCat data-grid server.

How to spin up my own iCat server?

Go to the VCL Reservation web page, and select iCat Development Server to reserve.  It takes about 4 mins to spin up.

How do I create a new user/account on my iCat development server?

To add an irods user:
Log in to iCat development server (with putty, for instance)
sudo bash
useradd unityId
su – irods
iadmin mkuser unityId rodsuser
mkdir -p /var/lib/irods/iRODS/Vault/ncsuResc/home/unityId

How do I prepare a new iRODS development server?

Tips on starting the irods server image on a different host (different from the host of the original iCat server).

The image “iCAT federation dsai_ncsu server” is configured to work on the host it was originally installed on. So an iCat development reservation (which runs on a different host) needs some minor changes to run. After the iCat development image is loaded onto a new (different) host; do the following
sudo bash
get hostname with command “hostname”
update the icat_host variable in /etc/irods/server_config.json to the FQDN of your machine – NOTE this is the one with a value of irods01.hpc.ncsu.edu, not one of the ones in the federation block. Replace irods01… with the hostname.
Turn on postgresql if not already running
service postgresql start
switch to postgresql user, start psql client, connect to ICAT database
su – postgres
-bash-4.1$ psql
postgres=# \connect ICAT
(note; \l list the databases, \dt list the tables of the connected database, \quit exits)
view the hostname defined in the r_resc_main table
Select * from r_resc_main;
resc_net column needs to be updated to list the correct hostname – update the records
do:
update r_resc_main SET resc_net = ‘bn??-??.dcs.mcnc.org’ where resc_net = ‘irods01.hpc.ncsu.edu’;
(copy and paste of above will not work, because the ‘ characters will not
transfer properly; fix them manually)
\quit to exit
exit ( back to root user)
As root update the irods’ environment file
edit /var/lib/irods/.irods/irods_environment.json
update irods_host variable with new hostname
Try to restart irods
/etc/init.d/irods restart
su – irods
start iadmin tool
iadmin

Backup and Restoration

What are the backup policies on the iCat server?

Backup of ROI space is being done using the HPC backup server
and TSM software (IBM Tivoli Storage Manager). Backups occur daily. TSM does all incremental backups. One consistent backup is maintained by daily updating any new or
changed files to match what is on the disk.

Files that are deleted are retained for several days to allow
accidentally deleted files to be recovered.

Recovery of files is not self service. It has to be done by TSM
administrator.

How do I retrieve a file from backup, and restore it to the iRods system?

Suppose /dsai_ncsu/home/username/filename was accidentally deleted, and a request is made within a few days to restore this file (the backup will keep copies of deleted files around for a few days).
The physical location of this file (before deletion), on the iCat server is, for example:
/var/lib/irods/iRODS/Vault/ncsuResc/home/username/filename

Since backups are made daily, and filename is older than a few days, there will be a copy of it in the backup system. Recovery of files is not self-service; it has to be done by the TSM administrator (Mikhail Bugaev). So a user would have to contact the iRods administrator (A. Petersen) and or Mikhail via email. If the iRods administrator was contacted, he would then contact Mikhail, who will recover the file. Files will be recovered typically within 24 hours.

It will be restored to the same position it was in:
/var/lib/irods/iRODS/Vault/ncsuResc/home/username/filename
However it needs to be registered in order for iRods to recognize it.
To register it, log in as irods user (the iRods Administrator can do this):
sudo bash
su – irods
Then register it:
ireg /var/lib/irods/iRODS/Vault/ncsuResc/home/username/filename /dsai_ncsu/home/username/filename
But this registers it as being owned by irods user.

To restore ownership to username:
ichmod own username /dsai_ncsu/home/username/filename

Now user should check that they can really see it:
Exit irods user status
exit
Exit sudo status
exit
Now, in user status, “ils filename” should show that the file is there.
.