Submit a job

Updated

Charlie uses a job submission engine called PBS Pro. To submit jobs to a node on Charlie, use the qsub command (/opt/pbs/bin/qsub). Upon successful submission of a job, you will receive a job id, a unique 6 or 7 digit number. Job id's are issued in sequential order and can be used to look up your job using qstat.

Submit a job from the command line

To submit a job from the command line, use the qsub command followed by the arguments/options (below).

Start an interactive session (i.e. log into a node through PBS):

qsub -I -N jobname -l ncpus=12,mem=20GB

Start a script:

qsub -N jobname -l ncpus=12,mem=20GB -- /bin/sh /mnt/dir/script.sh

Submit a job from a script

To submit a job using a shell script, include the arguments/options (below) at the beginning of the script, using the #PBS prefix. Normally in a shell script, anything after a hash (#) is a comment, however #PBS is a special operator that specifies PBS options. To run your script (example below) use qsub /path/to/script.sh.

#!/opt/pbs/bin/qsub

#PBS -N jobname               # name
#PBS -q route                 # use the default route queue
#PBS -V                       # copy environment variables

#PBS -l ncpus=1,mem=1gb       # request 1 cpu and 1GB of memory
#PBS -l walltime=00:10:00     # request 10 minutes of runtime (HH:MM:SS)

# Load modules
module use /mod/bigelow       # use the default bigelow modules
module load R                 # load the R module
# etc...

Options

Use the following options when using qsub to customize the job parameters and select resources to request.

Option

Description

Example

-a

Start time ([[[[CC]YY]MM]DD]hhmm[.SS] where CC=century, YY=year, MM=month, DD=day of the month, hh=hour, mm=minute, SS=seconds)

qsub -a 201901201000

-e

Error directories

qsub -e /home/<username>/error

-I

Interactive

qsub -I

-J

Range of job array (start-end[:step])

qsub -J 2-8:2

-l

Request options (more below)

qsub -l ncpus=2,mem=4GB

-m

Email job status (b=begin, e=end, a=abort)

qsub -m bea

-n

Name of job

qsub -N Example

-o

Output directories

qsub -o /home/<username>/out

-q

Queue (see queue list)

qsub -q low

-V

Pass environment variables with job

qsub -V

-X

Use X-Window for GUI applications (should be used with interactive jobs; -I)

qsub -X -I

List of job requirements (-l)

Option

Description

Example

ncpus

Number of CPU nodes

qsub -l ncpus=2

ngpus

Number of GPUs

qsub -l ngpus=1

mem

Amount of memory (RAM)

qsub -l mem-4GB

model

Model (e.g. c1, c2, c3)

qsub -l model=c1

vnode

Specific node (e.g. c3-20)

qsub -l vnode=c4-2

walltime

Walltime (HH:MM:SS)

qsub -l walltime=05:00:00

select

Nodes (see below)

qsub -l select=1

Resource requests can be combined using commas between. You can use any combination as long as it starts with a select.

E.g. Start a job with 2 cpus and 4GB memory (RAM) that will run on a node in the c3 cluster in under 5 hours:

qsub -l ncpus=2,mem=4GB,walltime=05:00:00,model=c3

GPU

c5-1 is currently the only node with a GPU (NVIDIA A100). To use c5-1 with the gpu:

qsub -l select=1:ngpus=1 -q gpu

You can also specify cpus, memory, etc. as usual.

A note on job arrays (-J)

Job arrays are a way to efficiently organize and submit multiple similar jobs. The PBS scheduler treats each sub-job in the job array as an individual job in the queue specified by the job array. Since job arrays are often large and can take up a significant amount of resources, we recommend submitting them to the low priority queue, so that other individual jobs can still be scheduled in the meantime.

Examples

# Request interactive job
qsub -I -l ncpus=1,mem=1gb

# Request interactive job on c4
qsub -I -l ncpus=1,mem=1gb,model=c4

# Request interactive job on c4-2
qsub -I -l ncpus=1,mem=1gb,model=c4-2

# Request interactive job with a GPU
qsub -I -q gpu -l ncpus=1,ngpus=1,mem=1gb

# Request interactive job with X Window (GUI) support
qsub -I -X -l ncpus=1,mem=1gb

# Run command using high priority queue
qsub -q high -l ncpus=1,mem=1gb -- /mnt/dir/command

# Run PBS submission script
qsub /path/to/script.sh

Still need help? Submit a ticket.

How did we do?

Job status (qstat)

Powered by HelpDocs (opens in a new tab)