GPU Jobs

GPU Hardware

Default GPU Limit

By default, every user can request up to 1 GPU per job. This is enforced by the normal QOS. You must explicitly request a higher QOS to access more GPUs.

Single GPU Job (default)

Available to all users under the default normal QOS:

#!/bin/bash
#SBATCH --job-name=train_model
#SBATCH --partition=standard
#SBATCH --time=24:00:00
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --gres=gpu:1                  # Request 1 GPU (default QOS supports this)
#SBATCH --output=%x_%j.out

source /opt/miniforge3/etc/profile.d/conda.sh
conda activate /data/apps/$(id -gn)/$USER/my-env

# Verify GPU is visible
nvidia-smi

python train.py --device cudagpu_job.sh

Multi-GPU Jobs

To request more than 1 GPU, you need the corresponding QOS. Contact an admin to have it added to your account. Then use both --gres and --qos:

Example — 4-GPU PyTorch DDP job

#!/bin/bash
#SBATCH --job-name=ddp_train
#SBATCH --partition=standard
#SBATCH --time=48:00:00
#SBATCH --cpus-per-task=32
#SBATCH --mem=256G
#SBATCH --gres=gpu:4                  # Request 4 GPUs
#SBATCH --qos=high                    # QOS must permit N GPUs
#SBATCH --output=%x_%j.out

source /opt/miniforge3/etc/profile.d/conda.sh
conda activate /data/apps/$(id -gn)/$USER/torch-env

# PyTorch DDP launch
torchrun \
    --standalone \
    --nproc_per_node=4 \
    train_ddp.py \
        --batch-size=512 \
        --epochs=100ddp_job.sh

Example — 6-GPU job with multigpu QOS

#!/bin/bash
#SBATCH --gres=gpu:6
#SBATCH --qos=multigpu
# ... other SBATCH options ...

torchrun --nproc_per_node=6 train.pymultigpu_job.sh

GPU Isolation

Slurm uses cgroups to enforce device-level GPU isolation. When your job is allocated GPU 2, only GPU 2 appears in nvidia-smi inside your job — you cannot accidentally use another user's GPU.

# Inside a job with --gres=gpu:1, only your GPU is visible
$ nvidia-smi
+-------------------------------------------------------------------------------+
| NVIDIA-SMI ...    Driver Version: ...    CUDA Version: ...                    |
|------------------------------|----------------------|------------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC  |
|   0  NVIDIA H200         Off  | 00000000:XX:00.0 Off |                 0     |
+------------------------------+----------------------+------------------------+
# Only 1 GPU shown, even though the node has 8 — cgroup isolation works.

# CUDA_VISIBLE_DEVICES is also set automatically
$ echo $CUDA_VISIBLE_DEVICES
2   # (whichever physical GPU was allocated)pasteur

Monitoring GPU Jobs

# Check GPU utilization while job is running
# (from pasteur-login, watching your job)
$ srun --jobid=101 --pty nvidia-smi dmon -s u

# After job completes, see GPU memory used
$ sacct -j 101 --format=JobID,MaxRSS,ReqGRES,AllocGRESpasteur-login

Framework Tips

PyTorch

import torch

# Verify GPU is available
print(torch.cuda.is_available())        # True
print(torch.cuda.device_count())        # = number of GPUs allocated
print(torch.cuda.get_device_name(0))    # NVIDIA H200

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = MyModel().to(device)python

TensorFlow

import tensorflow as tf

# List available GPUs
gpus = tf.config.list_physical_devices("GPU")
print(f"GPUs available: {len(gpus)}")  # = your allocated count

# Enable memory growth (recommended)
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)python

Requesting Elevated QOS