This document describes how to deploy a Slurm cluster that uses TPU v6e machine types and Google Cloud Managed Lustre storage by using Cluster Toolkit. For information about the cluster architecture, partitions, and slice topologies, see Slurm Cloud TPU deployments overview.
Before you begin
Before you create a Slurm cluster, complete the following prerequisites:
- Choose a consumption option for how you obtain and use Cloud TPU resources.
- Verify that you have
ct6e-standard-4tcapacity in your target zone for your chosen consumption option. Verify that you have enough Managed Lustre quota in your target region. By default, the blueprint provisions 36 TiB of capacity.
Set up Cluster Toolkit to provision Slurm clusters.
Set up your command-line environment
Install the Google Cloud CLI, and then sign in to the gcloud CLI with your federated identity. After signing in, initialize the Google Cloud CLI by running the following command:
gcloud initRequired roles
To get the permissions that you need to deploy a Slurm cluster, ask your administrator to grant you the following IAM roles:
- Storage Admin (
roles/storage.admin) on your project - Compute Instance Admin (v1) (
roles/compute.instanceAdmin.v1) on your project - Service Account User (
roles/iam.serviceAccountUser) on the service account itself -
To configure private services access and Managed Lustre:
- Managed Lustre Admin (
roles/lustre.admin) on your project - Compute Network Admin (
roles/compute.networkAdmin) on your project
- Managed Lustre Admin (
-
To consume a single-project reservation:
Compute Instance Admin (v1) (
roles/compute.instanceAdmin.v1) on your project -
To consume a shared reservation:
Compute Instance Admin (v1) (
roles/compute.instanceAdmin.v1) on both the owner project and any consumer project where you want to consume the reservation
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Set up a storage bucket
Cluster blueprints use Terraform modules to provision Cloud infrastructure. A best practice when working with Terraform is to store the state remotely in a version enabled file. On Google Cloud, you can create a Cloud Storage bucket that has versioning enabled.
To create this bucket and enable versioning from the CLI, run the following commands:
gcloud storage buckets create gs://BUCKET_NAME \
--project=PROJECT_ID \
--default-storage-class=STANDARD --location=BUCKET_REGION \
--uniform-bucket-level-access
gcloud storage buckets update gs://BUCKET_NAME --versioning
Replace the following:
BUCKET_NAME: a name for your Cloud Storage bucket that meets the bucket naming requirements.PROJECT_ID: your project ID.BUCKET_REGION: any available location.
Configure the deployment file
Configure the deployment file that specifies your Cloud Storage bucket, project and location settings, TPU partition sizes, and consumption option.
To configure the
examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml
file,
select your consumption option and update the file with the following
configuration:
Reservation-bound
In this configuration, you provision TPU nodes by using capacity from a specific local or shared reservation:
---
terraform_backend_defaults:
type: gcs
configuration:
bucket: BUCKET_NAME
vars:
deployment_name: DEPLOYMENT_NAME
project_id: PROJECT_ID
region: REGION
zone: ZONE
tpu_machine_type: ct6e-standard-4t
tpu_static_nodes: TPU_STATIC_NODES
tpu_dynamic_max_nodes: TPU_DYNAMIC_MAX_NODES
tpu_accelerator_topology: TPU_ACCELERATOR_TOPOLOGY
tpu_reservation_name: RESERVATION_NAME
DWS Flex-start
In this configuration, you use Dynamic Workload Scheduler with Flex-start mode to request TPU capacity on demand for up to seven days. For more information about how to use Dynamic Workload Scheduler with Flex-start mode in Slurm, see Dynamic Workload Scheduler with Flex-start mode in Slurm on GitHub.
---
terraform_backend_defaults:
type: gcs
configuration:
bucket: BUCKET_NAME
vars:
deployment_name: DEPLOYMENT_NAME
project_id: PROJECT_ID
region: REGION
zone: ZONE
tpu_machine_type: ct6e-standard-4t
tpu_static_nodes: TPU_STATIC_NODES
tpu_dynamic_max_nodes: TPU_DYNAMIC_MAX_NODES
tpu_accelerator_topology: TPU_ACCELERATOR_TOPOLOGY
tpu_dws_flex_enabled: true
Spot VMs
In this configuration, you provision TPU nodes as preemptible Spot VMs at a discounted price for fault-tolerant workloads:
---
terraform_backend_defaults:
type: gcs
configuration:
bucket: BUCKET_NAME
vars:
deployment_name: DEPLOYMENT_NAME
project_id: PROJECT_ID
region: REGION
zone: ZONE
tpu_machine_type: ct6e-standard-4t
tpu_static_nodes: TPU_STATIC_NODES
tpu_dynamic_max_nodes: TPU_DYNAMIC_MAX_NODES
tpu_accelerator_topology: TPU_ACCELERATOR_TOPOLOGY
tpu_enable_spot_vm: true
Replace the following:
BUCKET_NAME: the name of the Cloud Storage bucket that you created to store Terraform state.DEPLOYMENT_NAME: a unique name for your deployment, such astpu-v6e-slurm.PROJECT_ID: the ID of your Google Cloud project.REGION: the region where your cluster is deployed, such asus-central1.ZONE: the zone where your cluster is deployed, such asus-central1-b.TPU_STATIC_NODES: the number of static TPU nodes in the defaulttpupartition, such as8. This value must be a multiple of the number of compute instances per slice for theTPU_ACCELERATOR_TOPOLOGYvalue. Because eachct6e-standard-4tinstance contains 4 TPU chips, you can calculate the number of instances per slice by using the following formula:Instances per slice = total chips in topology / 4For example, when you set the
tpu_static_nodesfield to8:- A
4x4topology (16 chips) provisions two 4-instance slices. - A
4x8topology (32 chips) provisions one 8-instance slice.
- A
TPU_DYNAMIC_MAX_NODES: the maximum number of on-demand dynamic TPU nodes in thetpu-dynpartition, such as4.TPU_ACCELERATOR_TOPOLOGY: the 2D topology per slice for the statictpupartition, such as the4x8or4x4topology. For supported topologies and their compute instance counts, see TPU topologies.RESERVATION_NAME: the name of a reservation in the same project, or the full path of a shared reservation:projects/HOST_PROJECT_ID/reservations/SHARED_RESERVATION_NAME.
By default, the blueprint provisions on-demand capacity, and it consumes a
reservation only when you specify a value for the tpu_reservation_name field
in the
examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml
file.
If you want to provision on-demand capacity without a reservation, then omit the
tpu_reservation_name field or set it to "". For more information about these
provisioning models, see Consumption
options.
Provision a Slurm cluster
To provision the cluster, run the ./gcluster deploy
command from the
cluster-toolkit directory. This step takes approximately 20 to 30 minutes:
./gcluster deploy \
-d examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml \
examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-blueprint.yaml
Connect to the Slurm cluster
To access your cluster, sign in to the login node for Slurm by using either the Google Cloud console or the Google Cloud CLI:
Console
In the Google Cloud console, go to the Compute Engine > VM instances page.
Locate the login node, which uses the
TRUNCATED_DEPLOYMENT_NAME-slurm-login-001naming pattern, such astpuv6eslur-slurm-login-001.In the Connect column for the login node, click SSH.
gcloud
Look up the name of the login node by using the
gcloud compute instances listcommand:gcloud compute instances list --zones=ZONE --filter="name ~ login" --format="value(name)"Replace
ZONEwith the zone where your cluster is deployed.Connect to the login node by using the
gcloud compute sshcommand:gcloud compute ssh LOGIN_NODE_NAME \ --project=PROJECT_ID \ --zone=ZONE \ --tunnel-through-iapReplace the following:
LOGIN_NODE_NAME: the name of the login node from the previous step, such astpuv6eslur-slurm-login-001.PROJECT_ID: the ID of your Google Cloud project.ZONE: the zone where your cluster is deployed.
Verify the cluster
After you connect to the login node, complete the following steps to verify your partitions and run a test job with JAX:
Verify that the
tpuandtpu-dynpartitions are available:sinfoCreate a shared virtual environment for Python in the
/homedirectory and install JAX with TPU support:sudo mkdir -p "$HOME" && sudo chown "$(id -u):$(id -g)" "$HOME" python3 -m venv ~/jax-env ~/jax-env/bin/pip install -U pip "jax[tpu]" \ -f https://storage.googleapis.com/jax-releases/libtpu_releases.htmlRun a verification job on either the
tpustatic partition or thetpu-dyndynamic partition by using the--layoutflag:Static partition: set the
--layoutflag to match thetpu_accelerator_topologyvariable in your deployment file, and set the-Nflag to the number of compute instances in that slice. For example, run the job on 8 nodes for a4x8topology:srun -N 8 -p tpu --layout=v6e=4x8 ~/jax-env/bin/python -c \ "import jax; jax.distributed.initialize(); print(f'Host {jax.process_index()}: total={jax.device_count()}, local={jax.local_device_count()}')"Dynamic partition: set the
--layoutflag to the slice topology that you want Slurm to provision on demand, and set the-Nflag to the number of compute instances in that slice. For example, run the job on 2 nodes for a2x4topology:srun -N 2 -p tpu-dyn --layout=v6e=2x4 ~/jax-env/bin/python -c \ "import jax; jax.distributed.initialize(); print(f'Host {jax.process_index()}: total={jax.device_count()}, local={jax.local_device_count()}')"
Redeploy the Slurm cluster
To update your Slurm cluster configuration and keep the existing
VPC network and Managed Lustre instance in the
cluster-env group, specify the --only cluster flag when you redeploy:
./gcluster deploy \
-d examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml \
examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-blueprint.yaml \
--only cluster -w
To change the tpu_accelerator_topology variable on a deployed static nodeset
and keep your cluster-env network and Managed Lustre instance,
destroy and redeploy the cluster group:
./gcluster destroy DEPLOYMENT_FOLDER --only cluster
./gcluster deploy \
-d examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml \
examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-blueprint.yaml \
--only cluster -w
Replace DEPLOYMENT_FOLDER with the name of your deployment
folder, which is typically the same as the DEPLOYMENT_NAME
value.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this document, destroy the Slurm cluster and delete the state bucket if you no longer need it.
Destroy the Slurm cluster
- If you are connected to the login node, then disconnect from it.
- Navigate to the root of the
cluster-toolkitdirectory. Destroy the cluster by using the
./gcluster destroycommand:./gcluster destroy DEPLOYMENT_FOLDERReplace
DEPLOYMENT_FOLDERwith the name of your deployment folder, which is typically the same as theDEPLOYMENT_NAMEvalue.
Optional: Delete the storage bucket
Because the ./gcluster destroy command preserves your Terraform state bucket,
delete the Cloud Storage bucket when
you no longer need the deployment state.