UW-Statistics High Performance Computing (HPC) Cluster

We will use the HPC cluster to:

• make long-running jobs possible by removing their reliance on a laptop's battery, network connection, sleep state, login state, etc.
• learn the basics of the Slurm workload manager to run parallel jobs (Slurm is used on most supercomputers)
• make long-running jobs run faster by breaking them into sub-jobs and running them in parallel

Here is a quick introduction to using the HPC cluster.

1. Login via ssh CS_username@slurm-submit-00.cs.wisc.edu (where CS_username is your statistics user ID). We can submit computing jobs from this machine. (Or from public0[234] just run ssh slurm-submit-00.cs.wisc.edu.)
   Try cd /workspace/CS_username
   and then
   pwd
   to see that you got to that directory. If it works, you are done.
   If it fails, run
   mkdir /workspace/CS_username
   This makes a directory that is visible to all computing nodes on the HPC cluster.
   Do all your HPC work there.
2. Revise your path. When you type a command, bash looks for it in a list of directories saved in the environment variable $PATH. (Run "echo $PATH" to see it.) Run this command to add a line to the bottom of your "~/.bashrc" initialization file:
   echo "export PATH=$PATH:/workspace/software/bin" >> ~/.bashrc
   It says, "Add /workspace/software/bin, the directory containing python software, and ., the current directory, to my path." Restart your shell to make this change effective (or you can run "source ~/.bashrc").
3. Here are several small examples illustrating use of the HPC cluster. You can download them via
   wget pages.stat.wisc.edu/~jgillett/DSCP/HPC/examples.tar
tar -xvf examples.tar
   We will discuss them in lecture.
   
4. The HPC cluster is a large computer shared by many users. To avoid conflict, we should use these workflows.
   • Use two separate command lines, as follows.
     • Use slurm-submit-00, a shared submit server, only to edit and run Slurm commands that manage jobs. These commands are essential:
       • sbatch --partition short <script.sh> starts a batch job for execution on one or more computing nodes. It is called from ./submit.sh in each example, below (except the 5mtcarsPractice example, an exercise).
         Here are some additional options:
         • To receive an email with memory usage and other things, change "-CS_username" to your Statistics ID in the following line:
           --mail-user=NetID@wisc.edu
           and include
           --mail-type=END,FAIL
           too indicate you want to receive email as jobs END or FAIL.
         • To request more memory (default is 50M, maximum is 2600M), include e.g.
           --mem-per-cpu=500M
           
       • squeue -u <CS_username> lists my jobs.
       • scancel -u <CS_username> cancels all my jobs; scancel <jobid> cancels one job.
       • srun --partition short --pty /bin/bash runs an interactive job, giving you a command prompt on a computing node.
     • Run an interactive job (via srun --partition short --pty /bin/bash from slurm-submit-00) to get a command line on a computing node in the cluster if necessary for running jobs manually while debugging.
       
       1. You can ignore the error, "bash: ... .bashrc: Permission denied".
          (It says bash cannot read the configuration file ~/.bashrc on starting, which is because the computing nodes cannot read home directories. We are working in /workspace/CS_username for the same reason.)
       2. Running our usual emacs -nw tries to load an emacs initialization file from our home directory, which is not readable on the HPC computing nodes; this takes time as emacs waits for the load to fail. Instead, run
          emacs -nw --no-init-file
          to get the usual quick start-up.
          
   • Start small before going big, to ease debugging and prevent the launching of 100 jobs that fail (wasting your money in the real world).
     • While debugging, run only a few jobs on a tiny data set.
     Add print statements to track the values of variables and progress through scripts. Then check the log, stdout, and stderr files to investigate bugs. Here are examples:
     echo "file=$file, n=$n, sum=$sum"
     echo "step 3 starting ..."
echo "... step 3 done"
     Almost always when students ask for help with remote computing bugs, I spend 10 seconds looking and then add print statements to figure out where the program is failing and what the variable values are.
     • When your code is working, scale up to more jobs and more data, still monitoring resource use.
     • Test ".sh" and ".R" scripts serially in an interactive job. Slurm sets the environment variable SLURM_ARRAY_TASK_ID to 1, 2, 3, and so on for each of several parallel jobs in a job array. We can simulate this serially by using, e.g.
       export SLURM_ARRAY_TASK_ID=1; jobArray.sh
export SLURM_ARRAY_TASK_ID=2; jobArray.sh
export SLURM_ARRAY_TASK_ID=3; jobArray.sh
       ("man export" says exporting a variable makes it available to a subsequently executed command.)
       There is an example of this in the comment at the bottom of examples/4jobArrayWithDependencies/submit.sh.
       
   • Cancel your failed jobs, as otherwise they cause confusion on your next run and bog down the system.
   • After listening to the lecture, work on the exercise in the 5mtcarsPractice directory. It will be part of a homework assignment soon.
   • Here are links to more information:
     Statistics
     • HPC Users' Guide
       
     • Slurm Workload Manager
     • Options recommended by CHTC (coming soon) for transfering files.

---

Your Statisitcs accounts created for DSCP are temporary.
10 days after the last class day of the semester, your DSCP accounts will be removed, so you should copy any files you want to retain elsewhere. I suggest making a ".tar" file of your code and other human-written files (omitting most data files and most output files) and then copying that single file to your own computer.