Setup and Installation

Setup and Installation#

Cluster Setup#

To set up on the cluster do the following

Get Hatch, then Python:

  1. Download Hatch:

wget https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz

  1. Make a ~/.local/bin folder incase you never set it up before, if its there nothing happens:

mkdir -p ~/.local/bin

  1. Unpack Hatch into your ~/.local/bin:

tar -xzf hatch-x86_64-unknown-linux-gnu.tar.gz -C ~/.local/bin

  1. Install Python using Hatch

HATCH_PYTHON_VARIANT_LINUX=v2 hatch python install 3.12

Warning

Stellar is a heterogeneous cluster. If you don’t specify HATCH_PYTHON_VARIANT_LINUX=v2, hatch will download the most optimized Python for the head node, which won’t work on the workers.

  1. Log out or re-source your config, however you want to get Python on the path.

CUDA Setup#

  1. Make sure CUDA is available:

ml cudatoolkit/12.4

That last line is optional, but keeps you from having to set it up later.

  1. If you haven’t already cloned this repo, please do so and cd into it as follows:

git clone https://github.com/robelgeda/peytonites2024.git

cd peytonites2024

  1. Now you can use the command below to start up a Python terminal with everything present, including cupy!

hatch run cuda12:python

You can run import cupy to check if everything worked:

Python 3.12.3 (main, Apr 15 2024, 17:48:30) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cupy

Build Docs Locally (optional):

If you want to build and serve the docs (locally, for example, not on the cluster), you can use hatch run docs:serve.

Jupyter Lab (optional):

For Jupyter lab (such as locally), use:

hatch run jupyter:lab

If you want to use Jupyter on the cluster, run this command:

hatch run cuda12:ipykernel

This will install a file in your user directory that allows Jupyter to find the kernel, which will be named “cupy12”. Same thing ccan be done for the “jax” environment, with a matching name.

Submitting jobs#

Tip

You should put all your code in the peytonites2024/hackathon/main.py file.

  1. The job submission material is in the peytonites2024/hackathon subfolder for the repository. Before you go to the next step, please ensure you are in that dir:

pwd

or

cd hackathon

  1. Make sure the environment is updated on the head node first:

hatch env create cuda12

(Or any hatch run cuda12:... command does this, like hatch run cuda12:echo Synced)

  1. To submit:

sbatch submit.sbatch

from the hackathon folder.

Interactive running#

You can start an interactive session:

salloc --nodes=1 --ntasks=1 --mem=32G --time=00:30:00 --gres=gpu:1 --reservation=openhack

You can use hatch run cuda12:python to drop into Python, or hatch shell cuda12 to “activate” the environment. (Note that hatch is just making a normal virtual environment, and you can manually activate it like any other venv if you want. hatch env find cuda12 will show the actual env location.)

gh Tool#

Optional: if you like the gh tool, you can install it like this:

wget https://github.com/cli/cli/releases/download/v2.50.0/gh_2.50.0_linux_amd64.tar.gz

mkdir ~/.local

tar -xzf gh_2.50.0_linux_amd64.tar.gz --strip-components=1 -C ~/.local

Conda#

If you want to use conda, you can:

module load anaconda3/2023.3 cudatoolkit/12.4
source /usr/licensed/anaconda3/2023.3/etc/profile.d/conda.sh
conda env create -n peytonites python==3.12 pip
conda activate peytonites
pip install -e .[cuda12,jax]

Use the following lines in your batch jobs instead:

module load anaconda3/2023.3 cudatoolkit/12.4
source /usr/licensed/anaconda3/2023.3/etc/profile.d/conda.sh
conda activate peytonites
Contents