This document explains the architecture and core capabilities of the managed Slurm environment that Cluster Director uses to handle cluster management, job scheduling, and workload orchestration.
Slurm is an open-source, scalable, and fault-tolerant cluster manager and job scheduler. Cluster Director integrates and manages Slurm as the primary orchestrator for artificial intelligence (AI), machine learning (ML), and high performance computing (HPC) workloads. This managed environment lets you configure partitions, scale compute resources, and run jobs without needing to maintain the control plane.
Slurm architecture in Cluster Director
When you create and deploy a cluster, Cluster Director automatically sets up a pre-configured Slurm environment for you. This environment consists of the following nodes:
Controller node: the primary control plane component that monitors resources, manages system state, and processes all submitted jobs. When you create and deploy a cluster, Cluster Director automatically sets up the controller node in a fault-tolerant configuration to help ensure high availability and reliability.
Login nodes: the primary entry points for users to interact with the cluster. You connect to a login node by using SSH to submit jobs, review job statuses, and manage your workflows. The login nodes are pre-configured with the necessary Slurm commands and have access to the shared file systems.
Compute nodes: compute nodes run your jobs. Slurm manages these nodes in partitions, which are logical groupings of nodes with specific characteristics (such as machine type). You can configure compute nodes within each partition as follows:
Static nodes: you specify a minimum number of Compute Engine instances that must run in the cluster, which provides a baseline of available resources. For TPU nodesets, static nodes form persistent TPU slices with a fixed accelerator topology that remain active between jobs. If a static node encounters an error or powers down, then Slurm attempts to restart it.
Dynamic nodes: you specify a maximum number of additional compute instances that the cluster can create to handle increases in demand. When you submit jobs, Cluster Director creates these nodes on demand. If a dynamic GPU or CPU node remains in an
idlestate for five minutes, then Slurm powers it down. For dynamic TPU nodesets, Slurm provisions a TPU slice that matches the topology that your job requests. When the job completes, Slurm deletes the slice.
Slurm capabilities
Slurm provides several key capabilities that are essential for running large-scale AI, ML, or HPC workloads as follows:
Job scheduling and queuing: Slurm manages a queue of jobs that users submit. The scheduler allocates resources to these jobs based on priority, policies, and resource availability. This approach helps ensure that your cluster uses resources efficiently.
Resource management: Slurm tracks the cluster's resources, including vCPUs, memory, GPUs, Local SSD disks, and TPU chips. When you submit a job, you can request the specific resources that your job needs, and Slurm schedules the job on nodes that meet those requirements.
Topology-aware workload placement: Cluster Director uses Slurm's topology-aware scheduling capabilities to run your workloads. By using information about the physical network layout, Slurm colocates tasks of a job on compute instances that are close to each other in the network. For TPU workloads, Slurm provisions or selects interconnected 2D or 3D torus slices and coordinates single-slice jobs and multi-slice heterogeneous jobs. This placement minimizes latency for distributed training workloads.
Fault tolerance: if the compute nodes in your cluster have GPUs attached, then Slurm runs two automated scripts to verify the health of your GPUs:
Prolog health check: before a job starts, Slurm runs a prolog script named
990_chs_gpu_health_check_hcs.shto check for hardware issues. If a compute node fails this check, then Slurm drains the node and prevents jobs from running on it. This check helps ensure that your jobs only run on healthy hardware, preventing job failures or incorrect results.Background health check: every two minutes, Slurm runs a script named
health_check_program.shto detect errors. If the script finds an error, such as unresponsive GPUs or corrupted firmware, then the script logs the error to the compute node's system logs. You can then review the logs to troubleshoot the issue. This check helps you monitor your GPUs without draining your nodes.
Custom Slurm configuration: you can customize Slurm settings at the cluster, partition, and nodeset levels to adapt your cluster to the requirements of your workload. For a list of supported parameters, see Supported Slurm settings in Cluster Director.