Slurm & Scheduling

Partitions

Choose your partition based on how long your job needs to run. The standard partition is the default for most workloads.

Partition	Time Limit	Default?	Use Case
`debug`	1 hour	No	Quick tests, interactive sessions, script debugging
`standard`	48 hours	Yes	Most research workloads — model training, experiments
`extended`	7 days	No	Long-running jobs requiring more than 48 hours

ℹ

Always use the debug partition for testing Use --partition=debug when you're testing a new job script. It has a short time limit which makes it a quick feedback loop. Move to standard only when you're confident the script is correct.

QOS Levels

Quality of Service (QOS) controls what resources your job can use. Your default QOS is normal, which limits you to 1 GPU per job. Contact an admin to request elevated QOS.

QOS	Max GPUs	Description	Who gets it
`normal`	1 GPU	Default for all users	Everyone
`elevated`	2 GPUs	Elevated single-user priority	On request
`high`	4 GPUs	High priority workloads	On request
`multigpu`	6 GPUs	Multi-GPU training runs	On request
`multigpu_full`	8 GPUs	Full-node GPU allocation	Admin approval only

Submitting Jobs

Batch job — sbatch

The most common way to run jobs. Write a script with #SBATCH directives, then submit:

#!/bin/bash
#============================================================
# Slurm job script template for pasteur
#============================================================
#SBATCH --job-name=my_experiment      # Job name
#SBATCH --partition=standard          # Partition (debug/standard/extended)
#SBATCH --time=12:00:00              # Wall time (HH:MM:SS)
#SBATCH --ntasks=1                   # Number of tasks
#SBATCH --cpus-per-task=8           # CPUs per task
#SBATCH --mem=64G                    # RAM
#SBATCH --output=%x_%j.out          # Stdout log (%x=name, %j=job ID)
#SBATCH --error=%x_%j.err           # Stderr log
#SBATCH --mail-type=END,FAIL        # Email on completion/failure
#SBATCH --mail-user=you@university.edu

# ── Environment ──────────────────────────────
GROUP=$(id -gn)
source /opt/miniforge3/etc/profile.d/conda.sh
conda activate /data/apps/$GROUP/$USER/my-env

# ── Work ─────────────────────────────────────
echo "Starting job $SLURM_JOB_ID at $(date)"
echo "Running on: $(hostname)"

python my_script.py \
    --input /data/scratch/$GROUP/$USER/input_data \
    --output $HOME/results

echo "Done at $(date)"job.sh

# Submit
$ sbatch job.sh
Submitted batch job 101pasteur-login

Interactive job — srun

For interactive debugging or exploration:

# Basic interactive shell on the compute node
$ srun --partition=debug --time=01:00:00 --cpus-per-task=4 --mem=16G --pty bash

# Interactive job with a GPU (requires appropriate QOS)
$ srun --partition=debug --gres=gpu:1 --time=01:00:00 --pty bashpasteur-login

Monitoring Jobs

# View your jobs in the queue
$ squeue -u $(whoami)
  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST
    101  standard   my_exp   jsmith  R    1:23:45      1 pasteur
    102     debug  test_run  jsmith PD       0:00      1 (None)

# View all jobs (all users)
$ squeue

# Detailed info about a specific job
$ scontrol show job 101

# View cluster status
$ sinfo

# Cancel a job
$ scancel 101

# Cancel all your jobs
$ scancel -u $(whoami)pasteur-login

Job states

State	Code	Meaning
Running	R	Job is actively running on the compute node
Pending	PD	Waiting for resources to become available
Completing	CG	Job finishing up, cleaning resources
Failed	F	Job ended with a non-zero exit code
Cancelled	CA	Job was manually cancelled
Timeout	TO	Job exceeded its wall time limit

Job History & Accounting

# View completed jobs
$ sacct -u $(whoami) --format=JobID,JobName,Partition,State,Elapsed,CPUTime,MaxRSS

# View jobs from the last 7 days
$ sacct -u $(whoami) --starttime=now-7days --format=JobID,JobName,State,Elapsed

# Check your current resource usage / QOS
$ sacctmgr show assoc user=$(whoami) format=user,qos,maxgres -ppasteur-login

Tips & Best Practices

Test first in debug: Always run a shortened version of your job in --partition=debug before submitting a long job.
Request only what you need: Asking for 384 CPUs and all 8 GPUs when you need 8 CPUs and 1 GPU wastes cluster resources and delays your job.
Use --mail-type=END,FAIL: You'll get an email when your job finishes or fails — no need to keep checking.
Check memory usage: After a job completes, review MaxRSS in sacct to calibrate your memory requests for future jobs.
Name your outputs: Use %x (job name) and %j (job ID) in output file names so they don't overwrite each other.

⚠

Time limit = hard wall time When your job's wall time runs out, Slurm kills it immediately — no cleanup, no save. If your experiment is long, implement checkpointing in your code, or request the extended partition.