Getting Started

This is intended to be a primer for Charlie, assuming little to no Linux experience. If you do have experience with Linux systems, you can skip the first section "Linux Basics"

Connecting to Charlie

Charlie is accessed using a terminal program such as Terminal.app (macOS), iTerm (macOS 3rd party), or Command Prompt / PowerShell (Windows).

While connected to the Bigelow network (at Bigelow or over the VPN), open a terminal application. We will use the Secure Shell (ssh) command to remotely connect to the charlie front end (cfe). Replace username with your charlie username.

$ ssh username@cfe.bigelow.org

After pressing enter/return, You will be prompted for your charlie password (provided in your welcome email).

If this is your first time logging in, please use the cpasswd command to change your password.

Navigating Linux

You should see a "Welcome to Charlie" message when you log in with some helpful information and updates.

Since Charlie doesn't have a graphical user interface, you will use a few simple commands to navigate the file system.

You are currently in your home directory where you can save your personal files, data, and scripts.

The cd command is used to change directories. Typing cd without a file path after it will bring you home.

$ cd

There is a mounted file system that utilizes our on-premise NetApp storage system. Let's change directories to the database directory, where common datasets are stored.

$ cd /mnt/databases

Now that we are in the /mnt/databases directory, let's take a look at what's there. The ls command is used to list files in a directory.

$ ls
alphafold  cat  interproscan  kaiju  NCBI  pfam  vog

Typing ls without anything after it will show the files in the current directory. In this case, there are seven folders, starting with alphafold. Peraps we want to peek inside of the alphafold folder without changing directories. To do this, use the ls command again and add the folder that you want to list files/folders inside of.

$ ls alphafold/
bfd     params  pdb_mmcif   uniclust30  uniref90
mgnify  pdb70   pdb_seqres  uniprot

At any time, you can use the pwd command to print the current working directory.

$ pwd
/mnt/databases

Making and editing files

You can create and edit files on Charlie using the text editor nano.

Let's navigate back to your home directory using the cd command. To create a new file called "hello-world.txt" and open it in nano for editing, use the following command:

nano hello-world.txt

You will see a window that looks like this:

  GNU nano 2.9.8              hello-world.txt

















^G Get Help  ^O Write Out ^W Where Is  ^K Cut Text  ^J Justify
^X Exit      ^R Read File ^\ Replace   ^U Uncut Text^T To Spell

You can start typing like in a normal text editor. Note that your mouse will not work in the terminal, so use the arrow keys to move the cursor around.

Common nano commands can be found at the bottom of the window. For example, to exit and save, use the ^X command (where ^ is CTRL).

It will ask whether you want to save the file. Press Y for yes or N for no. Then, it confirms that you want to save to the same file name (i.e. save vs save as). Press enter to save to the same file name or change the file name and press enter for "save as".

Run ls and you should see your new hello-world.txt file. If you want to edit it again, just type nano hello-world.txt

To make a new directory to put your files into, use the mkdir command followed by the directory name. For example, to make a directory called scripts:

mkdir scripts

Modules

Charlie uses a system called modules to load different versions of software. New software installs can be requested via the helpdesk. Use modules avail to list the available software modules.

$ module avail
----------------------------- /mod/bigelow ------------------------------
ALE/1.0                      diamond/2.1.8(default)      R/4.1.3(default)
alphafold/2.2.4(default)     gcc/6.2.0                   R/4.2.0
anaconda3/4.0.0              gcc/9.0.0                   R/4.2.1
anaconda3/4.8.4(default)     gcc/11.2                    R/4.3.0
anaconda_individual/2020.02  hmmer/3.3.2(default)        R/dada2
anvio/5.0.0                  interproscan/5.60(default)  R/ecocast
anvio/7.0.0                  java/1.8.0_131              ragoo/1.11
anvio/7.1                    jre/1.8.0_201               raxml/8.2.12(default)
bfc/r181(default)            julia/1.0.2                 rclone/1.46
bitsanity/4.0.0              kaiju/1.9.0(default)        rstudio/1.1
blast/2.8.1                  mafft/7.487(default)        rstudio/1.1a
blast/2.13.0(default)        megacc/10.2.4(default)      rstudio/1.3
...

As you can see, there are many applications installed. Let's say we want to load the default version of R (4.1.3). We will use the module load command to load the module into our session.

$ module load R

Now you can run R by typing R as you normally would. You can use the module unload R command to unload the software when you're done (signing out of an ssh session will automatically unload the software).

Running a Job

So far, we have been working on a charlie front end (cfe) node. These nodes have limited resources and are for programming and job submission, not for computing. Computing is done on the compute nodes (e.g. c4-1, c4-2, etc.) To organize the cluster, we use the PBS job scheduler, which was developed by NASA for their supercomputing system.

To use a compute node, you will need to submit a job to PBS using the qsub command. PBS will then queue your job and run it as soon as there are enough resources available. Let's try running a simple interactive job.

$ qsub -I -q devel -l ncpus=2,mem=8gb,walltime=01:00:00
qsub: waiting for job 2670650.cfe to start
qsub: job 2670650.cfe ready


kguay at c4-3 in ~
$

Let's break this command down:

-I is used to specify that it is an interactive job. That is, when there are resources available, a new session will be started on a compute node.

-q devel specifies the devel queue. The devel queue has access to a development-only node (currently c4-1) that usually has space available even when there are many production jobs queued.

-l ncpus=2,mem=8gb,walltime=1:00:00 requests 2 cpu cores, 8gb of memory (RAM), and 1 hour of walltime. Wall time means the amount of time that your job takes to run - different from CPU time.

As you can see in your terminal, a session was opened on one of the cx-x compute nodes. You can also see your job ID (2670650.cfe for the example above).

When you are done with the session, use the exit command to end it.

$ exit
logout

qsub: job 2670650.cfe completed

You can also write scripts that run jobs - see the detailed Submit a Job page for more information.

Viewing Jobs and Resources

There are several custom tools and "wrappers" that we have written specifically for charlie.

The most useful is qstat (or the cstat wrapper)

$ cstat

cfe:
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2670594.cfe     hbigelow normal   Late_Ram_* 40943*   1  16  200gb 200:0 R 03:43 c4-3/2*16
2670620.cfe     lsmith   high     r          40684*   1  50  300gb 24:10 R 05:27 c4-3/1*50

As you can see, there are currently two jobs running. The first job, for example, uses 16 cores (TSK column) and 200gb of memory. It is scheduled to run for 200 hours (8 days and 8 hours) and has so far run for 3 hours and 43 minutes.

Some useful variations on the cstat command are:

cstat -u username which shows only jobs for the specified user

cstat -r which only shows running jobs

You can view the list of online nodes and the used & available resources for each node using the ctop command.

$ ctop
                                                Num
Node   CPU             Memory (gb)       GPU    Jobs
----   --------------  ----------------  -----  ---
c4-1     0/192 (0%)       0/1006 (0%)    -/-    0
c4-2     0/192 (0%)       0/1006 (0%)    -/-    0
c4-3    66/192 (34%)    500/1006 (49%)   -/-    2
c4-4     0/192 (0%)       0/1006 (0%)    -/-    0
c4-5     0/192 (0%)       0/1006 (0%)    -/-    0
c4-6     0/192 (0%)       0/1006 (0%)    -/-    0
c5-1     0/128 (0%)        0/250 (0%)    0/1    0
----   --------------  ----------------  -----  ----
Total  66/1280          500/6286         0/1    2
Queue  0                0                       0

Here you can see the two processes are both running on the c4-3 node and using a combined 34% of c4-3's CPU capacity and almost half of its memory. This is a great way to see a quick snapshot of the current load on Charlie.

Documentation & Support

In addition to the documentation on this support site, you can use the chelp command to quickly see some key information, including qsub examples and parameters, qstat job codes, queue information and more.

You can always reach out to the IT team via a helpdesk request at support.bigelow.org/help.