Conda / Miniforge

pasteur uses Miniforge — a minimal conda installer with the conda-forge channel pre-configured. The faster mamba solver is included and is the recommended way to create and install packages.

Miniforge is installed at /opt/miniforge3 and is available on both pasteur-login and the compute node. It is mounted read-only on the login VM via NFS.

Activation

Conda is activated automatically when you log in. If for any reason it isn't, initialize it manually:

# Conda is auto-loaded via /etc/profile.d/miniforge.sh
# Verify it's available:
$ conda --version
conda 26.1.0

$ mamba --version
mamba 1.x.x

# If not available, source manually:
$ source /opt/miniforge3/etc/profile.d/conda.shpasteur-login

Creating Environments

Always create personal environments under /data/apps/<group>/<username>/ — not in /opt/miniforge3 (which is read-only) and not in your home directory (counts against quota).

# Create a new environment (use mamba, not conda — it's faster)
$ mamba create --prefix /data/apps/$(id -gn)/$USER/my-env python=3.11
  ... solving environment ...
  ... downloading packages ...

# Activate it
$ conda activate /data/apps/$(id -gn)/$USER/my-env
(/data/apps/casl/jsmith/my-env) $ 

# Install packages
$ mamba install numpy pandas scikit-learn matplotlib jupyter

# Install PyTorch with CUDA (H200 is CUDA 12.x compatible)
$ mamba install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

# Install from pip inside an active mamba env
$ pip install transformers accelerate datasetspasteur-login

Using Environments in Job Scripts

#!/bin/bash
#SBATCH --job-name=ml_train
#SBATCH --partition=standard
#SBATCH --gres=gpu:1
# ... other SBATCH options ...

GROUP=$(id -gn)

# Always source conda before activating in a batch job
source /opt/miniforge3/etc/profile.d/conda.sh
conda activate /data/apps/$GROUP/$USER/torch-env

# Confirm environment
echo "Python: $(which python)"
echo "PyTorch: $(python -c 'import torch; print(torch.__version__)')"

python train.pyjob.sh

Shared Lab Environments

Admins may maintain shared conda environments under /data/apps/<group>/common/ for commonly used packages. These are read-only and available to all users in the group without any setup.

# List shared environments
$ ls /data/apps/$(id -gn)/common/

# Activate a shared env
$ conda activate /data/apps/$(id -gn)/common/<env-name>

# Do NOT install packages into shared envs — they are read-only.
# Clone one to your space if you need to modify it:
$ mamba create --prefix /data/apps/$(id -gn)/$USER/my-clone \
           --clone /data/apps/$(id -gn)/common/<env-name>pasteur-login

Useful Conda Commands

Apptainer Containers

Apptainer (formerly Singularity) v1.4.5 is installed on pasteur. Containers give you a completely reproducible software environment — useful for workflows that depend on specific system libraries, CUDA versions, or external software stacks (e.g., GROMACS, OpenFOAM, pre-built NGC images).

Shared Container Library

Pre-pulled containers are stored at /data/shared/containers/. Check there before pulling your own — common ML and scientific containers may already be available.

# List shared containers
$ ls /data/shared/containers/pasteur-login

Pulling Containers

# Pull from Docker Hub — store in your apps dir
$ apptainer pull /data/apps/$(id -gn)/$USER/containers/ubuntu.sif \
    docker://ubuntu:24.04

# Pull from NVIDIA NGC (recommended for GPU workloads)
$ apptainer pull /data/apps/$(id -gn)/$USER/containers/pytorch.sif \
    docker://nvcr.io/nvidia/pytorch:24.01-py3pasteur-login

Running Containers

# Run a command inside the container
$ apptainer exec /data/apps/$(id -gn)/$USER/containers/pytorch.sif \
    python -c "import torch; print(torch.cuda.is_available())"

# Interactive shell inside container
$ apptainer shell /data/apps/$(id -gn)/$USER/containers/pytorch.sifpasteur-login

Using Containers in Slurm Jobs

#!/bin/bash
#SBATCH --job-name=container_job
#SBATCH --partition=standard
#SBATCH --time=12:00:00
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --output=%x_%j.out

# Paths on the compute node use /data/ prefix
GROUP=$(id -gn)
SIF=/data/apps/$GROUP/$USER/containers/pytorch-24.01.sif

# --nv passes GPU access into the container
apptainer exec --nv \
    --bind /data/scratch/$GROUP/$USER:/workspace \
    $SIF \
    python /workspace/train.py \
        --data /workspace/dataset \
        --output /workspace/checkpointscontainer_job.sh

Building Custom Containers

If you need a custom container, write a definition file (.def) and build it. Building requires root or a sandbox approach — contact an admin if you need a custom image built:

# Example definition file (my_env.def)
Bootstrap: docker
From: nvcr.io/nvidia/pytorch:24.01-py3

%post
    pip install transformers accelerate datasets peft

%environment
    export HF_HOME=/workspace/.cache/huggingface

%labels
    Author jsmith@university.edu
    Version 1.0my_env.def

When to Use What