HPC documentation

General information

To access the HPC, start a terminal session and ssh into:

• cody.scem.westernsydney.edu.au (for CPU only)
• wolfe.scem.westernsydney.edu.au (for GPU)

The current set up has the following partitions:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST 
k6000        up   infinite      4   idle bd-client-[01-04]
cpu*         up   infinite      8   idle compute-[000-007]
a100-dev     up 7-00:00:00      1   idle a100-dev
a100         up 7-00:00:00      6   idle a100-[000-005]

Each k6000 node has 8 CPUs, cpu node has 16 CPUs, a100-dev (i.e. Wolfe) has 8 CPUS, and the a100 nodes have 32 CPU each.

Both k6000 and a100 have GPUs attached. The a100 nodes are the most recent addition, and each has the full GPU capability of a A100 chip, that is 9612 cuda cores and 40GB memory. The k6000 nodes have the GTX6000 chips, each with 4600 cuda cores and 24GB memory.

So for the A100 nodes:

• a100-dev: 8 CPUs, 32 GB RAM,10GB GPU RAM 
• a100-00[0,1] nodes:32 CPUs, 128GB RAM,40GB of GPU RAM.

Note: we also have the node a100-100, which has 80GB of GPU RAM. Request this node if you need a lot of memory for your job.

Copying files to and from the HPC

Use scp to copy files to and from the HPC.

To copy a local file to the cluster (do this from your local machine):

scp ./<filename> <user>@wolfe.scem.westernsydney.edu.au:<path>

If you don't use the path, then it should appear under your home directory.

To copy a file from the cluster:

scp wolfe.scem.westernsydney.edu.au:<path>/file .

This will copy the file under path to your current directory.

Note about the HDD

There are no backups. If a file or files go missing, that is the end of them. The HDD for the HPC is not meant to store valuable data. It is a place to hold them for use in the HPC. Anything important should be stored elsewhere.

other information to be filled later

Resources

GPU

To request GPU for your jobs, you need to include the line:

#SBATCH -P a100

in your bash script for submitting the jobs, to specify the a100 nodes (or the k6000 nodes). This is in addition to the gpu resource request.

Note that it is not possible to use the GPU on the head node (i.e. the node you logged in on), for testing code that requires the GPU, you'll need to request a temporary session with sinteractive, see relevant information below under SLURM.

Scratch disk

There is a small SSD local disk attached to the HPC: /bigdata-local, the disk is only 1TB in size thus will be cleaned regularly without prior warning. If your code is running too slow when reading data from your home directory, then putting your data on the local drive will greatly help. However, make sure that you clean up any of your data that is generated after your job has finished to allow other users to use the drive.

Note that if your job was run on a100-000 node, then the scratch will be on the /bigdata-local directory on that node (ditto for the other nodes). To clean up manually, you'll need to ssh into the node. Or you can just add a rm -rf line in your SLURM scrip to clean it up.

Software

To check available modules on the HPC, use the command module avail, to load a module use module load Python/Python3.7.0 for example.

Python related

To use PyTorch, you'll need the following combinations:

• Python 3.7 + Torch 1.9 Cuda 11
• Python 3.9 + Torch 1.9 + cuda 11
• Python 3.10 + Torch 12.1 Cuda 11.3

To view the available modules, use module avail. To load a module, use module load modulename. E.g. to load python3.10 with pytorch, use module load PyTorch/Python3.10.

Conda environment

If you have a specific set of packages you want to use for your project, then it's best to set up a conda environment for it (or use a singularity, see below). To do that, you need to first load the anaconda module, then create a new environment:

conda create -p ~/.conda/envs/<env_name> python==<python version>

note: as of 3 Oct 2024, the -p flag actually breaks conda, and conda create is creating the env dir in users' home dir.

Note that this is not how we normally create conda environment. The way conda is set up on wolfe means it somehow wants to create the environment in the base directory, where users don't have write permission. So, you create the environment under you own directory. You can give it any directory name you want and it doesn't have to be .conda/envs under your home directory.

Once that's set up, you first need to initilise the shell before you can activate your environment:

conda init <shell name>

The shell name can be bash, zsh, etc. I would go with bash if you don't know what any of the shell names mean. You'll then need to log off and log on wolfe to make sure the shell now works.

You can now activate your environment, the set up should tell you how to activate, but in general, it should be:

conda activate /rusers/<your group>/<your home dir>/<env path>/<env_name>

You can then install the various packages for your project.

Note for Pytorch: The PyTorch set up is pretty specific, so far only Python 3.9 is tested, and you need to install pytorch with the following command:

pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

This is because you'll need cuda 11+ for the A100, the previous versions doesn't have the chip in it's build.

To submit a job: instead of conda activate, you'll need source activate in your job script (see SLURM section below), no need to load anaconda. Everything else is the same.

Singularity/Container

apptainer is installed, but not as a module. Also according to David Minored "However, it has an issue with the current Kernel. Moving the OS up the the next version fixes that. However, it breaks at least one thing I have found. Once I have a fix for that, I'll do the upgrade.".

7-zip

To unzip files using 7-zip use the command 7za instead of 7z

Using SLURM

The HPC uses SLURM to for its job scheduler. Commonly used commands to control SLURM are:

• squeue shows the currently queued and running jobs.
• sinfo provides information about the nodes.
• sbatch is used in conjunction with a SLURM script to submit jobs to the queue
• scancel to stop a currently queued job.

SLURM Script

Submitting jobs to SLURM requires a job script. Below is a sample script to get started with.

#! /usr/bin/env bash
#
#SBATCH --job-name=simple
#SBATCH --output=simple.txt
#
#SBATCH --ntasks=1
#SBATCH --time=05:00 # this sets the maximum time the job is allowed before killed

#SBATCH --partition=a100
##SBATCH --partition=cpu # the double hash means that SLURM won't read this line.

# load the python module
module load Python/Python3.10 # make sure to load the modules needed

python3.10 simple.py # the program that is run

David said not to set mem-per-cpu, cputime or related items at the moment, there's some error with the system. It's on the to-do list.

Note: to request the 80GB node for your job, include in your script the line #SBATCH --nodelist=a100-100

Submitting a Job

Once the SLURM script is ready, the job can be submitted using sbatch script.sh, where script.sh is the name of the SLURM script. The progress of the job can be viewed using squeue or by examining the job output file (set to simple.txt in the above sample SLURM script).

Interactive Jobs

If you need to test code, it is highly recommended that you request time on the nodes via sinteractive:

sinteractive -p ampere24 --gres:gpu:a30:1

Will open a shell on an ampere24 node - if available, for 10 minutes. You can override the default 10 minute time by adding eg: --time 0:15:0 which is 15 minutes.

Note the --gres:gpu:a30:1 flag. This should be applied on ALL GPU jobs, because this tells slurm to not share a GPU node.

Note2: for the ampere80 queue it is (--gres:gpu:a100:1) and when the ampere40 queue comes back online also use --gres:gpu:a100:1.