Job Submission

Job scripts are submitted to the batch system using the qsub command.  Be sure to submit your job on the system you want your job to run on.

The batch systems on the two clusters are entirely separate; neither knows anything about the jobs queued or running on the other system.  You may edit your batch scripts anywhere, but you must submit and monitor your jobs from a login node on the system where you want to run.

Standard batch job

Most jobs on our system are submitted as scripts with no command-line options. If your script is in a file named “myscript”:

qsub myscript

In response to this command you’ll see a line with your job ID:

123456.oak-batch.osc.edu

You’ll use this job ID (numeric part only) in monitoring your job. You can find it again using the “qstat -u userid” command described elsewhere.

When you submit a job, the script is copied by the batch system. Any changes you make subsequently to the script file will not affect the job. Your input files and executables, on the other hand, are not picked up until the job starts running.

Interactive batch

The batch system supports an interactive batch mode. This mode is useful for debugging parallel programs or running a GUI program that’s too large for the login node. The resource limits (memory, CPU) for an interactive batch job are the same as the standard batch limits.

Interactive batch jobs are generally invoked without a script file, for example:

qsub -I -X -l nodes=2:ppn=12 -l walltime=1:00:00

The -I flag indicates that the job is interactive. The -X flag enables X11 forwarding, which is necessary with a GUI. You will need to have a X11 server running on your computer to use X11 forwarding [see more]. The remaining flags in this example are resource requests with the same meaning as the corresponding header lines in a batch file.

After you enter this line, you’ll see something like the following:

qsub: waiting for job 123456.opt-batch.osc.edu to start

Your job will be queued just like any job. When the job runs, you’ll see the following line:

qsub: job 123456.opt-batch.osc.edu ready

At this point, you have an interactive login shell on one of the compute nodes, which you can treat like any other login shell.

It is important to remember that OSC systems are optimized for batch processing, not interactive computing. If the system load is high, your job may wait for hours in the queue, making interactive batch impractical. Requesting a walltime limit of one hour or less is recommended because your job can run on nodes reserved for debugging.

Job arrays

If you submit many similar jobs at the same time, you should consider using a job array. With a single qsub command, you can submit multiple jobs that will use the same script. Each job has a unique identifier, $PBS_ARRAYID, which can be used to parameterize its behavior.

Individual jobs in a job array are scheduled independently, but some job management tasks can be performed on the entire array.

To submit an array of jobs numbered from 1 to 100, all using the script sim.job:

qsub -t 1-100 sim.job

The script would use the environment variable $PBS_ARRAYID, possibly as an input argument to an application or as part of a file name.

Job dependencies

It is possible to set conditions on when a job can start. The most common of these is a dependency relationship between jobs.

For example, to ensure that the job being submitted (with script sim.job) does not start until after job 123456 has finished:

qsub -W depend=afterany:123456 sim.job

Many other options are available, some quite complicated; for more information, see the qsub online manual by using the command:

man qsub