HPC documentation

HPC Workshop

If this is the first time you're using an HPC system, then please go through the HPC workshop content.

General information

To access the HPC, start a terminal session and ssh into:

• wolffe.cdms.westernsydney.edu.au

The current set up has the following partitions:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
cpu*         up 21-00:00:0      3    mix compute-[001-003]
ampere24     up 7-00:00:00      4  alloc a30-[002-005]
ampere40     up 7-00:00:00      3  alloc a100-[000-002]
ampere80     up 7-00:00:00      2   idle a100-[101-1]

The partitiions with name starting with ampere are the GPU nodes, the number refers to the RAM size of the GPU. ampere24 are the A30 chips, and ampere40 and 80 are A100 chips. Each GPU node has 8 CPUs.

So for the A100 nodes:

• a100-100[0,1] nodes: 12 CPUs, 128GB RAM, 80GB of GPU RAM.

Copying files to and from the HPC

Use scp to copy files to and from the HPC.

To copy a local file to the cluster (do this from your local machine):

scp ./<filename> <user>@wolffe.cdms.westernsydney.edu.au:<path>

If you don't use the path, then it should appear under your home directory.

To copy a file from the cluster:

scp wolffe.cdms.westernsydney.edu.au:<path>/file .

This will copy the file under path to your current directory.

Note about the HDD

There are no backups. If a file or files go missing, that is the end of them. The HDD for the HPC is not meant to store valuable data. It is a place to hold them for use in the HPC. Anything important should be stored elsewhere.

other information to be filled later

Resources

GPU

To request GPU for your jobs, you need to include the line:

#SBATCH -P ampere24

in your bash script for submitting the jobs, to specify the a100 nodes. This is in addition to the gpu resource request.

Note that it is not possible to use the GPU on the head node (i.e. the node you logged in on), for testing code that requires the GPU, you'll need to request a temporary session with sinteractive, see relevant information below under SLURM.

Scratch disk

There is a small SSD local disk attached to the HPC: /bigdata-local, the disk is only 1TB in size thus will be cleaned regularly without prior warning. If your code is running too slow when reading data from your home directory, then putting your data on the local drive will greatly help. However, make sure that you clean up any of your data that is generated after your job has finished to allow other users to use the drive.

Note that if your job was run on a100-000 node, then the scratch will be on the /bigdata-local directory on that node (ditto for the other nodes). To clean up manually, you'll need to ssh into the node. Or you can just add a rm -rf line in your SLURM scrip to clean it up.

Software

To check available modules on the HPC, use the command module avail, to load a module use module load Python/Python3.7.0 for example.

Python related

To use PyTorch, you'll need the following combinations:

• Python 3.7 + Torch 1.9 Cuda 11
• Python 3.9 + Torch 1.9 + cuda 11
• Python 3.10 + Torch 12.1 Cuda 11.3

To view the available modules, use module avail. To load a module, use module load modulename. E.g. to load python3.10 with pytorch, use module load PyTorch/Python3.10.

Conda environment

If you have a specific set of packages you want to use for your project, then it's best to set up a conda environment for it (or use a singularity, see below). To do that, you need to first load the anaconda module, then create a new environment:

conda create -p ~/.conda/envs/<env_name> python==<python version>

note: as of 3 Oct 2024, the -p flag actually breaks conda, and conda create is creating the env dir in users' home dir.

Note that this is not how we normally create conda environment. The way conda is set up on wolfe means it somehow wants to create the environment in the base directory, where users don't have write permission. So, you create the environment under you own directory. You can give it any directory name you want and it doesn't have to be .conda/envs under your home directory.

Once that's set up, you first need to initilise the shell before you can activate your environment:

conda init <shell name>

The shell name can be bash, zsh, etc. I would go with bash if you don't know what any of the shell names mean. You'll then need to log off and log on wolfe to make sure the shell now works.

You can now activate your environment, the set up should tell you how to activate, but in general, it should be:

conda activate /rusers/<your group>/<your home dir>/<env path>/<env_name>

You can then install the various packages for your project.

Note for Pytorch: The PyTorch set up is pretty specific, so far only Python 3.9 is tested, and you need to install pytorch with the following command:

pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

This is because you'll need cuda 11+ for the A100, the previous versions doesn't have the chip in it's build.

To submit a job: instead of conda activate, you'll need source activate in your job script (see SLURM section below), no need to load anaconda. Everything else is the same.

Singularity/Container

apptainer is installed, but not as a module. Also according to David Minored "However, it has an issue with the current Kernel. Moving the OS up the the next version fixes that. However, it breaks at least one thing I have found. Once I have a fix for that, I'll do the upgrade.".

7-zip

To unzip files using 7-zip use the command 7za instead of 7z

Using SLURM

The HPC uses SLURM to for its job scheduler. Commonly used commands to control SLURM are:

• squeue shows the currently queued and running jobs.
• sinfo provides information about the nodes.
• sbatch is used in conjunction with a SLURM script to submit jobs to the queue
• scancel to stop a currently queued job.

SLURM Script

Submitting jobs to SLURM requires a job script. Below is a sample script to get started with.

#! /usr/bin/env bash
#
#SBATCH --job-name=simple
#SBATCH --output=simple.txt
#
#SBATCH --ntasks=1
#SBATCH --time=05:00 # this sets the maximum time the job is allowed before killed

#SBATCH --partition=a100
##SBATCH --partition=cpu # the double hash means that SLURM won't read this line.

# load the python module
module load Python/Python3.10 # make sure to load the modules needed

python3.10 simple.py # the program that is run

David said not to set mem-per-cpu, cputime or related items at the moment, there's some error with the system. It's on the to-do list.

Note: to request the 80GB node for your job, include in your script the line #SBATCH --nodelist=a100-100

Submitting a Job

Once the SLURM script is ready, the job can be submitted using sbatch script.sh, where script.sh is the name of the SLURM script. The progress of the job can be viewed using squeue or by examining the job output file (set to simple.txt in the above sample SLURM script).

Interactive Jobs

If you need to test code, it is highly recommended that you request time on the nodes via sinteractive:

sinteractive -p ampere24 --gres:gpu:a30:1

Will open a shell on an ampere24 node - if available, for 10 minutes. You can override the default 10 minute time by adding eg: --time 0:15:0 which is 15 minutes.

Note the --gres:gpu:a30:1 flag. This should be applied on ALL GPU jobs, because this tells slurm to not share a GPU node.

Note2: for the ampere80 queue it is (--gres:gpu:a100:1) and when the ampere40 queue comes back online also use --gres:gpu:a100:1.