GPUs are requested via the GRES option:
sbatch --gres=gpu ...
Request multiple GPUs with:
At this time we have one GPU per node.
Requesting GPUs on Gizmo
The GPUs are currently installed only on nodes in the largenode partition, thus it is necessary to request the minimums required for largenode jobs:
sbatch -p largenode --mem=21500 -c 6 --gres=gpu ...
Requesting GPUs on Koshu
The GPU configured nodes on koshu are in the partition “gpu”. Jobs requiring GPUs need to request this partition and the GPUs:
sbatch -p gpu --gres=gpu ...
When your job is assigned a GPU, Slurm sets the environment variable CUDA_VISIBLE_DEVICES. This environment variable indicates the assigned GPU- most CUDA tools (e.g. tensorflow) use this to restrict execution to that device.