Scheduling Policies and Limits

The batch scheduler is configured with a number of scheduling policies to keep in mind. The policies attempt to balance the competing objectives of reasonable queue wait times and efficient system utilization. The details of these policies differ slightly on Oakley and Glenn. Exceptions to the limits can be made under certain circumstances; contact oschelp@osc.edu for details.

Hardware limits

Oakley and Glenn differ in the number of processors (cores) and the amount of memory and disk they have per node. We commonly find jobs waiting in the queue that cannot be run on the system where they were submitted because their resource requests exceed the limits of the available hardware. Jobs never migrate between systems, so please pay attention to these limits.

Notice in particular the large number of standard nodes and the small number of large-memory nodes. Your jobs are likely to wait in the queue much longer for a large-memory node than for a standard node. Users often inadvertently request slightly more memory than is available on a standard node and end up waiting for one of the scarce large-memory nodes, so check your requests carefully.

This is a brief summary. Details about the available hardware can be found elsewhere in the documentation.

 

# of Nodes # of cores per node (ppn) Memory in GB (*approximate)

Temporary File Space (GB)

Oakley (standard) 685 12 48 812
Oakley (bigmem) 8 12 192 812
Glenn  635 8 24 392

On Oakley, 64 of the standard nodes have 2 gpus each. On Glenn, 32 of the available newdual nodes have 2 gpus each.

* The actual amount of memory you can request in GB is slightly less than the nominal amount shown. It is safest to request memory in MB, for example, 8000MB instead of 8GB. (1GB is interpreted as 1024MB.)

Walltime limits per job

Serial jobs (that is, jobs which request only one node) can run for up to 168 hours, while parallel jobs may run for up to 96 hours.

Users who can demonstrate a need for longer serial job time may request access to the longserial queue, which allows single-node jobs of up to 336 hours. Longserial access is not automatic. Factors that will be considered include how efficiently the jobs use OSC resources and whether they can be broken into smaller tasks that can be run separately.

Limits per user and group

These limits are applied separately on Oakley and Glenn.

An individual user can have up to 128 concurrently running jobs and/or up to 2048 (2040 for Oakley) processor cores in use. All the users in a particular group/project can among them have up to 192 concurrently running jobs and/or up to 2048 (2040 for Oakley) processor cores in use. Jobs submitted in excess of these limits are queued but blocked by the scheduler until other jobs exit and free up resources.

A user may have no more than 1000 jobs submitted to the batch system at once. Jobs submitted in excess of this limit will be rejected.

Fair-share limits

To keep any one user or group/project from monopolizing the system when others need the same resources, the scheduler imposes what are known as fair-share limits. If a user or group/project uses large amounts of computing resources over a period of a few days, any new jobs they submit during that period will have reduced priority.

Projects with large negative RU balances

Projects that have used all their allocated resources may have further restrictions placed on their ability to run jobs. The project PI will receive multiple emails prior to having restrictions put in place. Restrictions may be lifted by submitting a proposal for additional HPC resources.

Priority

The priority of a job is influenced by a large number of factors, including the processor count requested, the length of time the job has been waiting, and how much other computing has been done by the user and their group over the last several days. However, having the highest priority does not necessarily mean that a job will run immediately, as there must also be enough processors and memory available to run it.

Short jobs for debugging

A small number of nodes are set aside during the day for jobs with a walltime limit of 1 hour or less. Please remember to exit debug jobs when you are done using the resources, to free them up for other users.

GPU Jobs

All GPU nodes are reserved for jobs that request gpus. Short non-GPU jobs are allowed to backfill on these nodes to allow for better utilization of cluster resources.