I am going to share some aliases that I use and have added in the .bashrc file (both at my local computer and the HiPerGator). If you are using the newer versions of mac, the default shell is Z shell, so you should add the aliases in .zshrc

Accessing the .bashrc file ( or .zshrc )

The .bashrc file is located in the home directory on your local machine. The location for home directory on my mac is: /Users/nhans,

On HiPerGator, the .bashrc file is located in the home directory, which in my case is as follow: /ufrc/burleigh/nhans

After you open, edit and close the file make sure you source the .bashrc

source .bashrc

My favorite Aliases

Alias command in linux can be used to create shortcuts for commands you mostly use. Make sure that you start with alias command and then add a space between the word/combination you would like to use as command. Use a “=” sign and double quotes of the command you want to set up as alias. Make sure there is no space between the your substitute word,= sign and quotes.

1) This is a great way for finding out what is taking so much space on your drives!

alias diskspace="du -S | sort -n -r |more"

2) Show me the size (sorted) of only the folders in this directory

alias folders="find . -maxdepth 1 -type d -print | xargs du -sk | sort -rn"

3) List by line in reverse time order

alias lsr="ls -ltr";

4) Naming working directories

alias gbird="cd /ufrc/burleigh/nhans/Cloning_Github/BigBird";

5) Git related alias added

alias gs='git status';
alias gc='git commit';
alias ga='git add';
alias gd='git diff';

alias gNH='git@github.com:NatyaHans';

Don’t forget to source the .bashrc after you have, copied and pasted the above Aliases

6) Function to extract compressed files

Copy the following to the .bashrc file. This should extract any file with the following extensions.

 # NH function to extract any file with following extensions
extract () {
   if [ -f $1 ] ; then
       case $1 in
           *.tar.bz2)   tar xvjf $1    ;;
           *.tar.gz)    tar xvzf $1    ;;
           *.bz2)       bunzip2 $1     ;;
           *.rar)       unrar x $1     ;;
           *.gz)        gunzip $1      ;;
           *.tar)       tar xvf $1     ;;
           *.tbz2)      tar xvjf $1    ;;
           *.tgz)       tar xvzf $1    ;;
           *.zip)       unzip $1       ;;
           *.Z)         uncompress $1  ;;
           *.7z)        7z x $1        ;;
           *)           echo "don't know how to extract '$1'..." ;;
       esac
   else
       echo "'$1' is not a valid file!"
   fi
}

USAGE: the file samplecompressedfolder.tar.gz given as first argument with the command should extract the folder

extract samplecompressedfolder.tar.gz

7) Function to compress a folder or files to tar.gz

Copy the following to the .bashrc file. This should compress any file or folder to tar.gz format.

# NH function to compress a file/directory to tar.gz
compress() {
   if [ -f $1 ] ; then
       echo "$1 is a file";
       case $1 in
           *)   tar czvf $1".tar.gz" $1 ;;
       esac
   elif [ -d $1 ]; then
       echo "$1 is a directory";
       case $1 in
           *)   tar czvf $1".tar.gz" $1 ;;
       esac
   else
      echo "'$1' is not a valid file!"
   fi
}

USAGE:

 compress samplefoldertobecompressed

should give you a compressed folder samplefoldertobecompressed.tar.gz

8) Function to go up when you have multiple subdirectories

 # NH function to go up in a directory 
up(){
    local d=""
    limit=$1
    for ((i=1 ; i <= limit ; i++))
       do
         d=$d/..
       done
    d=$(echo $d | sed 's/^\///')
    if [ -z "$d" ]; then
       d=..
    fi
    cd $d
    }

USAGE: to go up 4 directories

 up 4 

9) To find empty files

findzero() {
      echo " Files with zero size with extension ------ $1 ------ are : ";
      find . -type f -name "*.$1"  -size 0;
      echo " Total files: ";
      find . -type f -name "*.$1"  -size 0 |wc;
}

USAGE:

 findzero txt

Here txt can be replaced by the file extension of your choice.

10) To make a new directory and change into the directory

# NH function to mkdir and cd into it
function mc() {
    mkdir -p "$*" && cd "$*" && pwd
}

USAGE:

mc madeNewDirAndNowIamInIt

So now if you check your current dir it should be madeNewDirAndNowIamInIt

pwd

Breakdown of Slurm Script

#!/bin/sh: Important first line that informs it is a shell script.

#SBATCH --job-name=JOBNAME: Name of the job (can be changed by the user).

#SBATCH --mail-type=ALL : Mail events that the user chooses to receive. Options include: (NONE, BEGIN, END, FAIL, ALL).

#SBATCH --mail-user=USERNAME@ufl.edu : Where to send email

#SBATCH --cpus-per-task=4 : Number of cores to be used for this job

#SBATCH --mem-per-cpu=1gb : Per processor memory and can be increased to 8gb

#SBATCH -t 18:00:00 : Time required to run a job. The format is hours:minutes:seconds.
If you want to add day, for example using #SBATCH -t 1-4:20:50 will make the job scheduled to be run for 1 day 4 hrs 20mins and 50 seconds (unless it finishes first)

#SBATCH -o OUTPUTNAME.%j.out : # Name of the output file (can be changed by the user).

#SBATCH --account=GROUPNAME : # Name of the group to be specified by the user).

pwd; hostname; date : Prints out the current working directory, hostname and date respectively

echo Working directory is $SLURM_SUBMIT_DIR : The working directory is printed in the output file, and it is the directory from which the slurm command was sent.

cd $SLURM_SUBMIT_DIR : Changes directory from where the slurm command was sent.

echo There are $SLURM_CPUS_ON_NODE cores available. : prints out the number of core available.

The script then follows the list of modules you want to load and then software specific command requirements. The sample slurm script, shows how to run RAxML on HiPerGator.

Running Jobs on HiPerGator

To submit a job,

sbatch nameofthejob.sh

For example,

sbatch slurm_4_tania_raxml.sh

This will print something like: Submitted batch job 35808155, if your job gets submitted to the cluster. The number is the jobid.

To check the status of the job,

squeue -u username

For example,

squeue -u nhans


JOBID PARTITION     NAME        USER ST       TIME  NODES NODELIST(REASON)
35808155 hpg2-comp nameofthejob  nhans PD       0:00      1 (None)

PARTITION - Which partition on cluster
NAME - the script you submitted
USER - your username
TIME - time it has been running
ST- status, R means running PD means pending
NODES- how many nodes assigned to the job

To cancel a job,

scancel jobid

For example,

scancel 35808155

module save

This will show the following message: Saved current collection of modules to: "default"

If you quit and login to the HiPerGator again, typing the following should restore your modules loaded from last session:

module restore 

Personal Modules

If you want to create a personalized list of modules you can follow this tutorial here. Don’t forget to add the module list name to .bashrc file for this to work. The .bashrc file is located in your home directory. Be careful with this file, it is a super useful file and can make your life easier as we can create multiple aliases.

Navigating your way on the cluster

Now that you are on the cluster, let us learn some basic commands to navigate your way. First you need to know your location on the cluster. Type:

pwd

This should print out your “present working directory”. If you are logging in for the very first time, HiPerGator should default to the home directory.

/home/USERNAME

To do any analysis, or run jobs, it is recommended that you work in the /ufrc directory. Since all the modules are loaded and compatible in the dev version. To change directory, use command:

cd /ufrc/GROUPNAME/USERNAME

Note that here groupname is your group account created by hipergator. ( If you are a grad student. this should be your PI’s account). Replace groupname and username with your own information.

Creating new directory

Now to create a new directory, type:

mkdir NAME_OF_DIR

Replace NAME_OF_DIR with whatever you want to name your directory. For example:

mkdir 01_test

Now if you want to see the content of this directory, you need to go into the directory and use list command as follows:

cd 01_test
ls

Because you just created this directory, it should be empty.

Copying and moving files within cluster

To copy a file(already present on the cluster) into this directory, use the command cp(copy) as follows:

cp PathWhereTheFileIs/filename.txt PathWhereYouWantTheFileToBe

For example, let’s assume your file ‘scripting.txt’ is in directory path /ufrc/burleigh/nhans/oldfiles and you want to copy it to newly created directory with path /ufrc/burleigh/nhans/01_test

cp /ufrc/burleigh/nhans/oldfiles/scripting.txt  /ufrc/burleigh/nhans/01_test

Now you should have a copy of scripting.txt in 01_test directory. If you want to move the files, use the move command (mv):

mv /ufrc/burleigh/nhans/oldfiles/scripting.txt  /ufrc/burleigh/nhans/01_test

Now if you change directory by:

cd /ufrc/burleigh/nhans/01_test 

And type:

ls

You should have scripting.txt in your directory.

These are some basic maneuvering on the cluster. Note that moving and copying files work only when the files are already on the cluster.

Moving Files from your local machine to HiPerGator

Big files: Try Globus, if your data files are large (hundreds of megabytes or gigabytes).

Small files: For Windows, use SFTP by connecting to the ‘sftp.rc.ufl.edu’. SFTP uses port 22 if you have to specify it.

For Mac, you can use either rsync or scp.
Using scp to transfer files from local machine let’s say file.txt to HiPerGator, first cd to the location. Let’s assume /home/newfiles is the path where file.txt exists.

cd /home/newfiles 

You can type ls to check if the file exists there. Then type the following in terminal:

scp file.txt USERNAME@hpg.rc.ufl.edu:/ufrc/GROUPNAME/USERNAME

Replace GROUPNAME and USERNAME with your account information. For transfering ALL files with extension .txt, use the wildcard * as follows:

scp *.txt USERNAME@hpg.rc.ufl.edu:/ufrc/GROUPNAME/USERNAME

You can learn some basic BASH commands using this Cheat Sheet.

Creating Account on Hipergator

To use the HiPerGator, you first need to have an account with UF research computing. To request an account with Library HiPerGator Sponsorship Program, submit the request here.

You can also request an account with research computing if/when you have funding by submitting request here

Login to the HiPerGator

Depending on the operating system, logging in into the cluster can be easy or a bit tedious.

For windows, download an SSH client such as PuTTy or FireZilla. Set up the SSH client. For PuTTy,

1. Download PuTTY to your local machine and start the program .
2. Next, connect to hpg.rc.ufl.edu.
3. When asked for the login prompt, type your username (which should be your GatorLink username)
4. Enter your password when prompted. You should be connected to the HiPerGator.

If on Mac, open Terminal window, type

ssh USERNAME@hpg.rc.ufl.edu

Replace USERNAME with your username. Your USERNAME is your GATORLINK ID. Enter the password when prompted. You should be connected to the HiPerGator.

Logging off from the HiPerGator

To log off from the HiPerGator, type in the terminal,

exit