This document explains the architecture, slice topologies, OS image families, and capacity consumption options when you use Cluster Toolkit to deploy Slurm clusters with Cloud TPU compute nodes for artificial intelligence (AI) and machine learning (ML) workloads. For background information, see the Slurm overview and Introduction to Cloud TPU.
About Slurm in Cluster Toolkit
Slurm is an open-source cluster management and job scheduling system for high performance computing (HPC) and AI/ML workloads. Cluster Toolkit provides blueprints and modules that automate the deployment of Slurm clusters on Google Cloud, including the Slurm controller, login nodes, networking, shared storage, and compute partitions.
In a Slurm cluster that is deployed with Cluster Toolkit, you organize compute nodes into nodesets and partitions. Each nodeset can provision either always-on static nodes or on-demand dynamic nodes that autoscale. To learn how to choose between static and dynamic nodes, see About node types.
Cloud TPU Slurm deployments in Cluster Toolkit
Cloud TPU is Google's custom-developed application-specific integrated circuit (ASIC) used to accelerate machine learning workloads. On Compute Engine, you can provision and manage Cloud TPU resources by using standard Compute Engine instances and managed instance groups (MIGs). For more information, see TPU resources in Compute Engine.
In a Slurm cluster, the Slurm controller provisions these Compute Engine instances and MIGs as compute nodes in your nodesets and partitions. When you submit a job, Slurm schedules your workload across those nodes.
To deploy a Slurm cluster with Cloud TPU nodes, you work with the following hardware concepts:
- TPU slices: a Cloud TPU slice is a collection of Cloud TPU chips located inside the same pod that are physically connected over a dedicated, high-speed inter-chip interconnect (ICI).
- TPU topologies: a topology defines the number and physical arrangement
of Cloud TPU chips within a slice. Each supported
*-4tcompute instance contains 4 Cloud TPU chips. Depending on the Cloud TPU generation, chips are arranged in either a 2D (AxB) or 3D (AxBxC) torus interconnect topology. For static Cloud TPU partitions in Slurm, the number of static nodes, as defined by thetpu_static_nodesfield, must be a multiple of the number of compute instances per slice.
The following table lists the Cloud TPU generations and machine types supported in Slurm blueprints:
| TPU generation | Machine type (tpu_machine_type) |
Slurm --layout type |
Topology format | TPU chips per compute instance |
|---|---|---|---|---|
| Ironwood (TPU7x) | tpu7x-standard-4t |
tpu7x |
3D torus (AxBxC) |
4 (8 TPU cores) |
| TPU v6e | ct6e-standard-4t |
v6e |
2D torus (AxB) |
4 (4 TPU cores) |
| TPU v5p | ct5p-hightpu-4t |
v5p |
3D torus (AxBxC) |
4 (8 TPU cores) |
| TPU v5e | ct5lp-hightpu-4t |
v5e |
2D torus (AxB) |
4 (4 TPU cores) |
The following table lists common 2D and 3D slice topologies and their corresponding chip and compute instance counts per slice:
| 2D topology for TPU v6e and TPU v5e | 3D topology for Ironwood (TPU7x) and TPU v5p | TPU chips per slice | Compute instances per slice |
|---|---|---|---|
2x2 |
2x2x1 |
4 | 1 |
2x4 |
2x2x2 |
8 | 2 |
4x4 |
2x2x4 |
16 | 4 |
4x8 |
2x4x4 |
32 | 8 |
8x8 |
4x4x4 |
64 | 16 |
Slurm cluster architecture
Each blueprint organizes cluster resources into two deployment groups so that you can update or redeploy the Slurm cluster and keep your foundational network and storage:
cluster-envgroup: provisions a Virtual Private Cloud (VPC) network, configures private services access, and creates a Google Cloud Managed Lustre file system that is mounted at the/homedirectory across all of the nodes in the cluster for shared environments, datasets, and checkpoints.clustergroup: provisions a Slurm controller, a login node, and two Cloud TPU partitions:- Static partition (
-p tpu): provisions always-on static Cloud TPU nodes that are grouped into slices with a fixed 2D or 3D topology. - Dynamic partition (
-p tpu-dyn): provisions on-demand dynamic Cloud TPU nodes that create a slice to match your job's--layoutflag and power down when idle.
- Static partition (
Image families and boot disks
Compute nodes in the Cloud TPU partitions use prebuilt
Advanced Compute Images that require 100 GB of
disk space. You configure the type of boot disk, such as Google Cloud Hyperdisk or
the pd-ssd disk type, in the blueprint rather than in the image.
The following table lists the prebuilt image family and default boot disk for each Cloud TPU generation:
| TPU generation | Image family | Default boot disk |
|---|---|---|
| Ironwood (TPU7x) | aci-tpu-tpu7x-u2404-slurm-2605-amd64 |
Hyperdisk Balanced |
| TPU v6e | aci-tpu-v5e-v5p-v6e-u2204-slurm-2605-amd64 |
Hyperdisk Balanced |
| TPU v5p | aci-tpu-v5e-v5p-v6e-u2204-slurm-2605-amd64 |
Balanced Persistent Disk (pd-balanced) |
| TPU v5e | aci-tpu-v5e-v5p-v6e-u2204-slurm-2605-amd64 |
Balanced Persistent Disk (pd-balanced) |
Consumption options
To balance capacity availability, workload duration, and cost, Cluster Toolkit supports on-demand capacity, reserved capacity, Dynamic Workload Scheduler with Flex-start mode, and Spot VMs for your Cloud TPU partitions:
- On-demand: provisions standard on-demand instances without a reservation. This option is the default in the blueprint.
- Reservation-bound: consumes capacity from a specific local or shared Compute Engine reservation. This option is the default in the example deployment files.
- Dynamic Workload Scheduler with Flex-start mode: requests Cloud TPU capacity on demand for up to seven days by using Dynamic Workload Scheduler with Flex-start mode.
- Spot VMs: provisions preemptible Spot VMs at a discounted price for fault-tolerant workloads.