STAT DSCP: HPC

UW-Statistics High Performance Computing (HPC) Cluster

We will use the HPC cluster to:

make long-running jobs possible by removing their reliance on a laptop's battery, network connection, sleep state, login state, etc.
learn the basics of the Slurm workload manager to run parallel jobs (Slurm is used on most supercomputers)
make long-running jobs run faster by breaking them into sub-jobs and running them in parallel

Here is a quick introduction to using the HPC cluster.

Login via ssh STATuser@slurm-submit-00.cs.wisc.edu (where STATuser is your statistics user ID). We can submit computing jobs from this machine. (Or from public0[234] just run ssh slurm-submit-00.cs.wisc.edu.)
Try cd /workspace/STATuser
and then
pwd
to see that you got to that directory. If it works, you are done.
If it fails, run
mkdir /workspace/STATuser
This makes a directory that is visible to all computing nodes on the HPC cluster.
Do all your HPC work there.
Revise your path. When you type a command, bash looks for it in a list of directories saved in the environment variable $PATH. (Run "echo $PATH" to see it.) Run this command to add a line to the bottom of your "~/.bashrc" initialization file:
echo "export PATH=$PATH:/workspace/software/bin" >> ~/.bashrc
It says, "Add /workspace/software/bin, the directory containing python software, and ., the current directory, to my path." Log out and log back in to make this change effective. (Or you can run "source ~/.bashrc".)
The HPC cluster is a large computer shared by many users. To avoid conflict, we should use these workflows.
- Use two separate command lines, as follows.
  - Use slurm-submit-00, a shared submit server, only to edit and run Slurm commands that manage jobs. These commands are essential:
    - sbatch --partition short <script.sh> starts a batch job for execution on a computing node. It is called from ./submit.sh in each example, below (except the 5mtcarsPractice example, an exercise).
      Here are some additional options:
      - To receive an email with memory usage and other things, change "-STATuser" to your Statistics ID in the following line:
        --mail-user=STATuser@wisc.edu
        and include
        --mail-type=END,FAIL
        too indicate you want to receive email as jobs END or FAIL.
      - To request more memory (default is 50M, maximum is 2600M), include e.g.
        --mem-per-cpu=500M
    - squeue -u <STATuser> lists my jobs.
    - scancel -u <STATuser> cancels all my jobs; scancel <jobid> cancels one job.
    - srun --partition short --pty /bin/bash runs an interactive job, giving you a command prompt on a computing node.
    - seff <jobid> shows a "Slurm job efficiency report" of memory usage, etc.
      On 3/17/25, the command was at /workspace/software/slurm/contribs/seff/seff. We can use that long path, or run echo "export PATH=$PATH:/workspace/software/slurm/contribs/seff" >> ~/.bashrc to add its directory to our PATH. (Then log out and log back in or run "source ~/.bashrc".)
  - Run an interactive job (via srun --partition short --pty /bin/bash from slurm-submit-00) to get a command line on a computing node in the cluster if necessary for running jobs manually while debugging.
    1. You can ignore the error, "bash: ... .bashrc: Permission denied".
      (It says bash cannot read the configuration file ~/.bashrc on starting, which is because the computing nodes cannot read home directories. We are working in /workspace/STAT_USER for the same reason.)
    2. Running our usual emacs -nw tries to load an emacs initialization file from our home directory, which is not readable on the HPC computing nodes; this takes time as emacs waits for the load to fail. Instead, run
      emacs -nw --no-init-file
      to get the usual quick start-up.
- Start small before going big, to ease debugging and prevent the launching of 100 jobs that fail (wasting your money in the real world).
  - While debugging, run only a few jobs on a tiny data set.
  - Check and adjust resource requirements (memory, disk, #CPUs): too small and your job will bog down, too big and shared resources are wasted.
  - When your code is working, scale up to more jobs and more data, still monitoring resource use.
  - Test ".sh" and ".R" scripts serially in an interactive job. Slurm sets the environment variable SLURM_ARRAY_TASK_ID to 1, 2, 3, and so on for each of several parallel jobs in a job array. We can simulate this serially by using, e.g.
    export SLURM_ARRAY_TASK_ID=1; jobArray.sh
    export SLURM_ARRAY_TASK_ID=2; jobArray.sh
    export SLURM_ARRAY_TASK_ID=3; jobArray.sh
    ("man export" says exporting a variable makes it available to a subsequently executed command.)
    There is an example of this in the comment at the bottom of examples/4jobArrayWithDependencies/submit.sh.
- Cancel your failed jobs, as otherwise they cause confusion on your next run and bog down the system.
- Here are several small examples illustrating use of the HPC cluster. You can download them via
  wget www.stat.wisc.edu/~jgillett/DSCP/HPC/examples.tar
  tar -xvf examples.tar
  We will discuss them in lecture.
- After listening to the lecture, work on the exercise in the 5mtcarsPractice directory. It will be part of a homework assignment soon.
- Here are links to more information:
  Statistics
  - HPC Overview and Users' Guide
  - Slurm Workload Manager
  - Options recommended by CHTC (coming soon) for transfering files.

Your Statisitcs accounts created for DSCP are temporary.
10 days after the last class day of the semester, your DSCP accounts will be removed, so you should copy any files you want to retain elsewhere. I suggest making a ".tar" file of your code and other human-written files (omitting most data files and most output files) and then copying that single file to your own computer.