Computing with GPUs
There are currently two capabilities available for GPUs in the gizmo. The J and K class nodes have consumer-grade RTX cards. We have recently (fall 2024) added more capable harmony nodes with NVIDIA L40s cards. These are significantly more capable systems.
GPU Nodes
| Location | Partition | Node Name | GPU |
|---|---|---|---|
| FHCRC | campus, short, new | j | NVIDIA GTX 1080ti |
| FHCRC | campus, short, new | k | NVIDIA RTX 2080ti |
| FHCRC | chorus | harmony | NVIDIA L40S |
| FHCRC | none (interactive use) | rhino | NVIDIA RTX1080ti |
Accessing GPUs
GPUs are available in multiple partitions- the newest GPUs are in the chorus partition. Refer to this page for limits and other details about the partitions.
In campus, short, and restart-new partitions
GPUs are requested via the --gpus option:
sbatch --gpus=1 ...
Specific GPU models can be requested by indicating model and count:
sbatch --gpus=rtx2080ti:1 ...
In the chorus partition
The new GPU systems have a different processor and a newer version of operating system- this requires a new set of environment modules specific to this new architecture. Because of this, these nodes are separated into new partition, chorus:
sbatch --partition=chorus --gpus=1 ...
There are currently only L40S GPUs, but for the sake of precision and future-proofing, you can specify the GPU as well:
sbatch --partition=chorus --gpus=l40s:1 ...
Since these nodes have more GPUs available per node, you can request more just by increasing the count:
sbatch --partition=chorus --gpus=3 ...
# or:
sbatch --partition=chorus --gpus=l40s:3 ...
Please make sure your code is capable of using multiple GPUs before requesting more than one,
When submitting jobs make sure that your current environment does not have modules loaded (i.e. module purge) and that you are loading the new modules in your script. You may run into conflicts with modules built for rhino/gizmo compute platforms.
Using GPUs
When your job is assigned a GPU, Slurm sets the environment variable CUDA_VISIBLE_DEVICES. This environment variable indicates the assigned GPU- most CUDA tools (e.g. tensorflow) use this to restrict execution to that device.