Using Slurm on Hutch Systems

Edit this Page via GitHub       Comment by Filing an Issue      Have Questions? Ask them here.

Slurm

Slurm is the workload manager for Scicomp clusters. It manages both your jobs and the resources available in the cluster. There are two main clusters in use today- the on-campus Gizmo cluster and the cloud-based Beagle cluster. Commands work the same in either environment.

Examples of Use

A GitHub repository has been created that is an evolving resource for the community containing working examples of using Slurm at Fred Hutch. Please see the Slurm Examples repo for more specific guidance on using Slurm in variety of settings.

Basic Slurm Terminology

Cluster

A cluster is a collection of compute resources (nodes) under the control of the workload manager (Slurm in our case). At the Hutch we have two clusters, Beagle and Gizmo. From most hosts the default cluster will be gizmo- selection of the target cluster is done via an argument to Slurm commands (see Multi-Cluster Operation below)

Partition

A partition is a collection of resources (nodes) inside of a cluster. There are defaults, so specifying a partition name is not required. While the different clusters may have different partitions, there are two partitions- a default partition with smaller nodes named campus and a partition with more capable nodes (more memory and CPUs) named largenode.

Node

A node is the basic computing unit that shares processors, memory, and some (limited) local disk. As a rule, you don’t want to worry about choosing a node for your jobs.

Job

A job is a collection of tasks, typically implemented as a shell script. Jobs have an ID (just a number) and a name. The ID is automatically assigned, but you can assign a name to your job.

Account

When we refer to an “account” in the context of Slurm, we are referring to the PI account used to enforce limits and priority and not your HutchNet ID. Your HutchNet ID is associated with an account.

Commands for Managing Jobs

squeue

The squeue command allows you to see the jobs running and waiting in the job queue. squeue takes many options to help you select the jobs displayed by the command

option/argument function
-M cluster Only show jobs running on the indicated cluster
-u username Limit jobs displayed to those owned by a user
-A account Limit jobs displayed to those owned by an account
-p partition Only show jobs in the indicated partition
rhino2[~]: squeue -u edgar
      JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
   31071290    campus     wrap    edgar  R   19:03:13      1 gizmof404

The field NODELIST(REASON) will show either the name of the node(s) allocated for running jobs or the reason a job isn’t running.

There are many ways to alter which jobs are shown and how the output is formatted- refer to the squeue manpage for more details on using this command.

scancel

scancel allows you to signal jobs- most commonly this command is used to stop execution of a running job or remove a pending job from the job queue. A job IDis the common argument though scancel will take many other arguments that allow bulk management of jobs- it shares many of the same arguments as squeue. For example, the command:

rhino2[~]: scancel -u edgar

will cancel all jobs (pending or running) owned by the user edgar.

hitparade

The hitparade command will show a summary of all jobs running and queued on the cluster broken down by user and account. Note that this isn’t a Slurm command, rather something built in-house.

hitparade takes the -M argument to select a cluster:

rhino2[~]: hitparade -M beagle
loading Python/3.6.4-foss-2016b-fh2...

  === Queue: campus ======= (R / PD) ======
    poe_e (edgar) 300 / 72

  === Queue: largenode ======= (R / PD) ===
    schulz_cm (snoopy) 273 / 0

Commands for Submitting Jobs

sbatch and srun

sbatch is used to submit a job script to the cluster. These run jobs without your intervention or input (i.e. non-interactively). Common arguments are:

srun is used to run a task on the cluster. This is an interactive session, where you can directly view output as it’s produced or provide input (if needed by the task you are running).

These two take many of the same options:

  • -M select the cluster on which the job will run
  • -p change the partition
  • -t request a certain amount of time for the job.
  • -n request a number of tasks (default 1)
  • -c request a number of processors per task (default 1)
  • -j name a job

A useful option that is only applicable to sbatch is -o, which writes output to a different file. The default will write the file _slurm-.out_ in the directory in which you submitted the job.

sbatch/srun Examples:

Submit a batch job that will run in one day, six hours in the largenode partition in Beagle. This will run one instance of the job with one processor. Name the job “quoth-the-raven”.

sbatch -M beagle -p largenode -t 1-6 -j quoth-the-raven myscript.sh

Submit a job using 6 cores and redirect output to a file named “my-output”:

sbatch -c 6 -j my-output

MultiCluster Operation

Most Slurm commands can operate against remote clusters (i.e. beagle from gizmo). Typically the only change required is to add the argument -M <cluster name>.

sbatch -M beagle -c 6 -j my-output
scancel -M beagle 12345

hitparade also supports -M and can be used to show the queue on the different clusters. At this time, multi-cluster operations using the commands srun and salloc will not work. If use of those commands is necessary, please contact SciComp.

Updated:

Edit this Page via GitHub       Comment by Filing an Issue      Have Questions? Ask them here.