Submit a job
Updated
Charlie uses a job submission engine called PBS Pro. To submit jobs to a node on Charlie, use the qsub command (/opt/pbs/bin/qsub). Upon successful submission of a job, you will receive a job id, a unique 6 or 7 digit number. Job id's are issued in sequential order and can be used to look up your job using qstat.
Submit a job from the command line
To submit a job from the command line, use the qsub command followed by the arguments/options (below).
Start an interactive session (i.e. log into a node through PBS):
qsub -I -N jobname -l ncpus=12,mem=20GB
Start a script:
qsub -N jobname -l ncpus=12,mem=20GB -- /bin/sh /mnt/dir/script.sh
Submit a job from a script
To submit a job using a shell script, include the arguments/options (below) at the beginning of the script, using the #PBS prefix. Normally in a shell script, anything after a hash (#) is a comment, however #PBS is a special operator that specifies PBS options. To run your script (example below) use qsub /path/to/script.sh.
#!/opt/pbs/bin/qsub
#PBS -N jobname # name
#PBS -q route # use the default route queue
#PBS -V # copy environment variables
#PBS -l ncpus=1,mem=1gb # request 1 cpu and 1GB of memory
#PBS -l walltime=00:10:00 # request 10 minutes of runtime (HH:MM:SS)
# Load modules
module use /mod/bigelow # use the default bigelow modules
module load R # load the R module
# etc...
Options
Use the following options when using qsub to customize the job parameters and select resources to request.
Option | Description | Example |
-a | Start time ( |
|
-e | Error directories |
|
-I | Interactive |
|
-J | Range of job array ( |
|
-l | Request options (more below) |
|
-m | Email job status ( |
|
-n | Name of job |
|
-o | Output directories |
|
-q | Queue (see queue list) |
|
-V | Pass environment variables with job |
|
-X | Use X-Window for GUI applications (should be used with interactive jobs; |
|
List of job requirements (-l)
Option | Description | Example |
ncpus | Number of CPU nodes |
|
ngpus | Number of GPUs |
|
mem | Amount of memory (RAM) |
|
model | Model (e.g. c1, c2, c3) |
|
vnode | Specific node (e.g. c3-20) |
|
walltime | Walltime (HH:MM:SS) |
|
select | Nodes (see below) |
|
Resource requests can be combined using commas between. You can use any combination as long as it starts with a select.
E.g. Start a job with 2 cpus and 4GB memory (RAM) that will run on a node in the c3 cluster in under 5 hours:
qsub -l ncpus=2,mem=4GB,walltime=05:00:00,model=c3
GPU
c5-1 is currently the only node with a GPU (NVIDIA A100). To use c5-1 with the gpu:
qsub -l select=1:ngpus=1 -q gpu
You can also specify cpus, memory, etc. as usual.
A note on job arrays (-J)
Job arrays are a way to efficiently organize and submit multiple similar jobs. The PBS scheduler treats each sub-job in the job array as an individual job in the queue specified by the job array. Since job arrays are often large and can take up a significant amount of resources, we recommend submitting them to the low priority queue, so that other individual jobs can still be scheduled in the meantime.
A note about output files
If you don't need your output files (stdout and stderr), please use the following syntax:
-o /dev/null -e /dev/null
Examples
# Request interactive job
qsub -I -l ncpus=1,mem=1gb
# Request interactive job on c4
qsub -I -l ncpus=1,mem=1gb,model=c4
# Request interactive job on c4-2
qsub -I -l ncpus=1,mem=1gb,model=c4-2
# Request interactive job with a GPU
qsub -I -q gpu -l ncpus=1,ngpus=1,mem=1gb
# Request interactive job with X Window (GUI) support
qsub -I -X -l ncpus=1,mem=1gb
# Run command using high priority queue
qsub -q high -l ncpus=1,mem=1gb -- /mnt/dir/command
# Run PBS submission script
qsub /path/to/script.sh