Docs
Enroot Guide
How to set up and run Enroot containers on BigPurple with Slurm, plus tips for sharing images and GPU access.
Quick setup
Minimal steps to pull and run an Enroot container with GPU access.
1) Load modules
module add enroot parallel jq2) Personal images
export XDG_RUNTIME_DIR=$HOME
export XDG_CONFIG_HOME=$HOME/.config
export XDG_CACHE_HOME=$HOME/.cache
export XDG_DATA_HOME=$HOME/.local/share2) Lab repository (shared)
export XDG_RUNTIME_DIR=/gpfs/data/oermannlab/public_data/enroot/
export XDG_CONFIG_HOME=/gpfs/data/oermannlab/public_data/enroot/.config
export XDG_CACHE_HOME=/gpfs/data/oermannlab/public_data/enroot/.cache
export XDG_DATA_HOME=/gpfs/data/oermannlab/public_data/enroot/.local/share3) Request a GPU session (srun)
srun -p a100_short --nodes=1 -t 4:00:00 --tasks=1 --cpus-per-task=4 --tasks-per-node=1 --gres=gpu:a100:1 --mem=32G --pty bashor submit a batch job with
sbatch my_slurm_batch_file.sbatch4) Import base image
enroot import --output ubuntu_2004.sqsh docker://ubuntu:20.04or a blank CUDA container:
enroot import --output cuda_base.sqsh docker://nvidia/cuda5) Create and run
enroot create --name my_enroot_image ./ubuntu_2004.sqsh
enroot list | grep my_enroot_image
enroot start -r -w -m .:/mnt my_enroot_image-rroot / sudo-wwritable rootfs-mhost:container bind mount
Advanced setup (Slurm + Conda)
Run a simple Torch program on BigPurple GPUs inside Enroot with a Conda env.
1) Test script
echo "
import torch
device = torch.device('gpu' if torch.cuda.is_available() else 'cpu')
print(f'using device: {device}')
a = torch.rand(10).to(device)
b = torch.rand(10).to(device)
c = torch.matmul(a, b)
print(a)
print(b)
print(c)
" > test_torch.py2) Enroot config (my_enroot_conf.batch)
#ENROOT_ROOTFS=ubuntu
#ENROOT_REMAP_ROOT=y
#ENROOT_ROOTFS_WRITABLE=y
echo "hello in home ${HOME}"
export NVIDIA_VISIBLE_DEVICES=all
export THIS_IS_A_TEST=test_variable
environ() { env }
mounts() {
echo "${PWD} /mnt none bind"
echo "/usr/bin/ /mnt/import_execs/ none bind"
}3) Request GPU + launch
srun -p a100_short --nodes=1 -t 4:00:00 --tasks=1 --cpus-per-task=4 --gres=gpu:a100:1 --mem=32G --pty bash
enroot start --conf my_enroot_conf.batch my_enroot_image4) Install Conda in the container
apt-get update && apt-get install -y wget
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh --no-check-certificate -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
export PATH="$PATH:~/miniconda3/bin"
conda init && source ~/.bashrc
pip3 install torch numpy5) Run the program
python /mnt/test_torch.pySLURM: fully isolated runtimes
Use sbatch to launch Enroot with a command.
#!/bin/bash
# torch_run.sbatch
#SBATCH -p a100_short
#SBATCH --nodes=1
#SBATCH -t 4:00:00
#SBATCH --tasks=1
#SBATCH --cpus-per-task=4
#SBATCH --tasks-per-node=4
#SBATCH --gres=gpu:a100:1
#SBATCH --mem=32G
cd /gpfs/data/oermannlab/users/<netid>/ENROOT_EXAMPLES
module add enroot jq parallel
enroot start --conf my_enroot_conf.batch my_enroot_image -- bash -c 'python /mnt/test'Submit with
sbatch torch_run.sbatchNotes
- Permanent state: once an image is created, changes persist across sessions. Install conda once per image.
- GPU visibility: set
NVIDIA_VISIBLE_DEVICES=allinside the container or via--conf. Test withnvidia-smi.
If GPU isn’t available, you may see errors from NVIDIA hooks (e.g., libnvidia-container). Ensure your Slurm session requested a GPU.
Sample nvidia-smi output
Tue Jul 16 14:47:19 2024
NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0
...