In late 2018, OSC installed 260 Intel® Xeon® 'Skylake' processor-based nodes as the original Pitzer cluster. In September 2020, OSC installed additional 398 Intel® Xeon® 'Cascade Lake' processor-based nodes as part of a Pitzer Expansion cluster. This expansion makes Pitzer a heterogeneous cluster, which means that the jobs may land on different types of CPU and behaves differently if the user submits the same job script repeatedly to Pitzer but does not request the resources properly. This document provides you some general guidance on how to request resources on Pitzer due to this heterogeneous nature.
Step 1: Identify your job type
| Nodes the job may be allocated on | # of cores per node | Usable Memory | GPU | |
|---|---|---|---|---|
| Jobs requesting standard compute node(s) | Dual Intel Xeon 6148s Skylake @2.4GHz | 40 |
178 GB memory/node 4556 MB memory/core |
N/A |
| Dual Intel Xeon 8268s Cascade Lakes @2.9GHz | 48 |
178 GB memory/node 3797 MB memory/core |
N/A | |
| Jobs requesting dual GPU node(s) |
Dual Intel Xeon 6148s Skylake @2.4GHz |
40 |
363 GB memory/node 9292 MB memory/core |
2 NVIDIA Volta V100 w/ 16GB GPU memory |
| Dual Intel Xeon 8268s Cascade Lakes @2.9GHz | 48 |
363 GB memory/node 7744 MB memory/core |
2 NVIDIA Volta V100 w/32GB GPU memory | |
| Jobs requesting quad GPU node(s) | Dual Intel Xeon 8260s Cascade Lakes @2.4GHz | 48 |
744 GB memory/node 15872 MB memory/core |
4 NVIDIA Volta V100s w/32GB GPU memory and NVLink |
| Jobs requesting large memory node(s) | Dual Intel Xeon 8268s Cascade Lakes @2.9GHz | 48 |
744 GB memory/node 15872 MB memory/core |
N/A |
| Jobs requesting huge memory node(s) | Quad Processor Intel Xeon 6148 Skylakes @2.4GHz | 80 |
2989 GB memory/node 38259 MB memory/core |
N/A |
According to this table,
- If your job requests standard compute node(s) or dual GPU node(s), it can potentially land on different types of nodes and may result in different job performance. Please follow the steps below to determine whether you would like to restrain your job to a certain type of node(s).
- If your job requests quad GPU node(s), large memory node(s), or huge memory node(s), please check pitzer batch limit rules on how to request these special types of resources properly.
Step 2: Perform test
This step is to submit your jobs requesting the same resources to different types of nodes on Pitzer. For your job script is prepared with either PBS syntax or Slurm syntax:
Request 40 or 48 core nodes
#SBATCH --constraint=40core #SBATCH --constraint=48core
Request 16gb, 32gb gpu
#SBATCH --constraint=v100 #SBATCH --constraint=v100-32g --partition=gpuserial-48core
Once the script is ready, submit your jobs to Pitzer and wait till the jobs are completed.
Step 3: Compare the results
Once the jobs are completed, you can compare the job performances in terms of core-hours, gpu-hours, walltime, etc. to determine how your job is sensitive to the type of the nodes. If you would like to restrain your job to land on a certain type of nodes based on the testing, you can add #SBATCH --constraint=. The disadvantage of this is that you may have a longer queue wait time on the system. If you would like to have your jobs scheduled as fast as possible and do not care which type of nodes your job will land on, do not include the constraint in the job request.
