Slurm TPU deployments overview

This document explains the architecture, slice topologies, OS image families, and capacity consumption options when you use Cluster Toolkit to deploy Slurm clusters with Cloud TPU compute nodes for artificial intelligence (AI) and machine learning (ML) workloads. For background information, see the Slurm overview and Introduction to Cloud TPU.

About Slurm in Cluster Toolkit

Slurm is an open-source cluster management and job scheduling system for high performance computing (HPC) and AI/ML workloads. Cluster Toolkit provides blueprints and modules that automate the deployment of Slurm clusters on Google Cloud, including the Slurm controller, login nodes, networking, shared storage, and compute partitions.

In a Slurm cluster that is deployed with Cluster Toolkit, you organize compute nodes into nodesets and partitions. Each nodeset can provision either always-on static nodes or on-demand dynamic nodes that autoscale. To learn how to choose between static and dynamic nodes, see About node types.

Cloud TPU Slurm deployments in Cluster Toolkit

Cloud TPU is Google's custom-developed application-specific integrated circuit (ASIC) used to accelerate machine learning workloads. On Compute Engine, you can provision and manage Cloud TPU resources by using standard Compute Engine instances and managed instance groups (MIGs). For more information, see TPU resources in Compute Engine.

In a Slurm cluster, the Slurm controller provisions these Compute Engine instances and MIGs as compute nodes in your nodesets and partitions. When you submit a job, Slurm schedules your workload across those nodes.

To deploy a Slurm cluster with Cloud TPU nodes, you work with the following hardware concepts:

  • TPU slices: a Cloud TPU slice is a collection of Cloud TPU chips located inside the same pod that are physically connected over a dedicated, high-speed inter-chip interconnect (ICI).
  • TPU topologies: a topology defines the number and physical arrangement of Cloud TPU chips within a slice. Each supported *-4t compute instance contains 4 Cloud TPU chips. Depending on the Cloud TPU generation, chips are arranged in either a 2D (AxB) or 3D (AxBxC) torus interconnect topology. For static Cloud TPU partitions in Slurm, the number of static nodes, as defined by the tpu_static_nodes field, must be a multiple of the number of compute instances per slice.

The following table lists the Cloud TPU generations and machine types supported in Slurm blueprints:

TPU generation Machine type (tpu_machine_type) Slurm --layout type Topology format TPU chips per compute instance
Ironwood (TPU7x) tpu7x-standard-4t tpu7x 3D torus (AxBxC) 4 (8 TPU cores)
TPU v6e ct6e-standard-4t v6e 2D torus (AxB) 4 (4 TPU cores)
TPU v5p ct5p-hightpu-4t v5p 3D torus (AxBxC) 4 (8 TPU cores)
TPU v5e ct5lp-hightpu-4t v5e 2D torus (AxB) 4 (4 TPU cores)

The following table lists common 2D and 3D slice topologies and their corresponding chip and compute instance counts per slice:

2D topology for TPU v6e and TPU v5e 3D topology for Ironwood (TPU7x) and TPU v5p TPU chips per slice Compute instances per slice
2x2 2x2x1 4 1
2x4 2x2x2 8 2
4x4 2x2x4 16 4
4x8 2x4x4 32 8
8x8 4x4x4 64 16

Slurm cluster architecture

Each blueprint organizes cluster resources into two deployment groups so that you can update or redeploy the Slurm cluster and keep your foundational network and storage:

  • cluster-env group: provisions a Virtual Private Cloud (VPC) network, configures private services access, and creates a Google Cloud Managed Lustre file system that is mounted at the /home directory across all of the nodes in the cluster for shared environments, datasets, and checkpoints.
  • cluster group: provisions a Slurm controller, a login node, and two Cloud TPU partitions:

    • Static partition (-p tpu): provisions always-on static Cloud TPU nodes that are grouped into slices with a fixed 2D or 3D topology.
    • Dynamic partition (-p tpu-dyn): provisions on-demand dynamic Cloud TPU nodes that create a slice to match your job's --layout flag and power down when idle.

Image families and boot disks

Compute nodes in the Cloud TPU partitions use prebuilt Advanced Compute Images that require 100 GB of disk space. You configure the type of boot disk, such as Google Cloud Hyperdisk or the pd-ssd disk type, in the blueprint rather than in the image.

The following table lists the prebuilt image family and default boot disk for each Cloud TPU generation:

TPU generation Image family Default boot disk
Ironwood (TPU7x) aci-tpu-tpu7x-u2404-slurm-2605-amd64 Hyperdisk Balanced
TPU v6e aci-tpu-v5e-v5p-v6e-u2204-slurm-2605-amd64 Hyperdisk Balanced
TPU v5p aci-tpu-v5e-v5p-v6e-u2204-slurm-2605-amd64 Balanced Persistent Disk (pd-balanced)
TPU v5e aci-tpu-v5e-v5p-v6e-u2204-slurm-2605-amd64 Balanced Persistent Disk (pd-balanced)

Consumption options

To balance capacity availability, workload duration, and cost, Cluster Toolkit supports on-demand capacity, reserved capacity, Dynamic Workload Scheduler with Flex-start mode, and Spot VMs for your Cloud TPU partitions:

  • On-demand: provisions standard on-demand instances without a reservation. This option is the default in the blueprint.
  • Reservation-bound: consumes capacity from a specific local or shared Compute Engine reservation. This option is the default in the example deployment files.
  • Dynamic Workload Scheduler with Flex-start mode: requests Cloud TPU capacity on demand for up to seven days by using Dynamic Workload Scheduler with Flex-start mode.
  • Spot VMs: provisions preemptible Spot VMs at a discounted price for fault-tolerant workloads.

What's next