Create a custom AI-optimized GKE cluster which uses G2 or G4

General GPUs

This document describes how you can create custom Google Kubernetes Engine (GKE) clusters that use the G2 (NVIDIA L4) or G4 (NVIDIA RTX PRO 6000) accelerator-optimized machine series to support your artificial intelligence (AI) and machine learning (ML) workloads. G2 and G4 are General GPU machine series, which provides independent compute resources designed for mainstream AI tasks such as small-to-medium inference and graphics-intensive applications.

GKE provides a single platform surface to run a diverse set of workloads for your organization, reducing the operational burden of managing multiple platforms.

To create an AI-optimized GKE cluster with G2 or G4, you'll do the following:

1. Create the GKE environment
2. Connect to your cluster
3. Configure your Pod manifests

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
  Note: For existing gcloud CLI installations, make sure to set the compute/region property. If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Required roles

To get the permissions that you need to create and manage a GKE cluster, ask your administrator to grant you the following IAM roles on the project:

• Kubernetes Engine Admin (roles/container.admin)
• Compute Admin (roles/compute.admin)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Choose a consumption option and obtain capacity

1. Choose a consumption option. Make your choice based on how you want to get and use GPU resources. For more information, see Choose a consumption option.

   For GKE, consider the following additional information when you choose a consumption option:

   • For more information about flex-start (Preview) and GKE, see About GPU obtainability with flex-start.
   • Flex-start uses best-effort compact placement. To examine your topology, see View the physical topology of nodes in your GKE cluster.
   • You can only get topology information when using Spot VMs if you configure compact placement.

2. Obtain capacity. Learn how to obtain capacity for your consumption option.

Requirements

To create an AI-optimized GKE cluster which uses G2 or G4, ensure you meet the following requirements:

• GKE version: G4 (RTX PRO 6000) machines require GKE version 1.32.4-gke.1698000 or higher.
• GPU drivers: your GPU nodes must use NVIDIA driver version 535 or later.

Limitations

The following limitations apply to G2 and G4 as General GPU machine series:

• Local SSDs: G2 and G4 machine types don't include Local SSD disks by default. You must manually attach them if required.
• NCCL Fast Socket: you cannot use NCCL Fast Socket with G2 or G4 machines on GKE. While G2 and G4 machines support multi-node communication, they use standard VPC networking. For large-scale distributed training that requires maximum network throughput, we recommend using the Clustered GPU machine series, such as A3 High or A3 Mega (which use GPUDirect TCPX) or A3 Ultra and A4 (which use GPUDirect RDMA).

Create the GKE environment

You can create a cluster in Autopilot or Standard mode.

Autopilot

1. To create an Autopilot cluster, run the following command:

   gcloud container clusters create-auto CLUSTER_NAME \
       --region=REGION
   

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • REGION: the region for your cluster.

Standard

Create a Standard cluster and GPU node pool:

1. To create a Standard cluster, run the following command:

   gcloud container clusters create CLUSTER_NAME \
       --region=REGION \
       --enable-ip-alias
   

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • REGION: the region for your cluster.

2. Create the node pool. After creating your cluster, you can add a node pool with G2 or G4. For multi-node workloads using G2 (L4) or G4 (RTX PRO 6000), we recommend using a compact placement policy to minimize networking latency between nodes.

   To create a node pool, use the following command:

   Tip: To run graphics-intensive workloads on this VM, you must use the nvidia-l4-vws (for G2) or nvidia-rtx-pro-6000-vws (for G4) accelerator type because it supports NVIDIA RTX Virtual Workstation (vWS). When you create an instance that uses NVIDIA RTX Virtual Workstation, Compute Engine automatically adds a vWS license. For information about pricing for virtual workstations, see the GPU pricing page.

   G2 (NVIDIA L4)

   gcloud container node-pools create NODE_POOL_NAME \
       --cluster=CLUSTER_NAME \
       --region=REGION \
       --node-locations=ZONE \
       --machine-type=MACHINE_TYPE \
       --accelerator=type=nvidia-l4,count=AMOUNT,gpu-driver-version=LATEST \
       --placement-type=COMPACT \
       --scopes="https://www.googleapis.com/auth/cloud-platform" \
       --local-ssd-count=LOCAL_SSD_COUNT
   

   G4 (NVIDIA RTX PRO 6000)

   gcloud container node-pools create NODE_POOL_NAME \
       --cluster=CLUSTER_NAME \
       --region=REGION \
       --node-locations=ZONE \
       --machine-type=MACHINE_TYPE \
       --accelerator=type=nvidia-rtx-pro-6000,count=AMOUNT,gpu-driver-version=LATEST \
       --disk-type=hyperdisk-balanced \
       --placement-type=COMPACT \
       --scopes="https://www.googleapis.com/auth/cloud-platform" \
       --local-ssd-count=LOCAL_SSD_COUNT
   

   G4 Virtual Workstation (vWS)

   gcloud container node-pools create NODE_POOL_NAME \
       --cluster=CLUSTER_NAME \
       --region=REGION \
       --node-locations=ZONE \
       --machine-type=MACHINE_TYPE \
       --accelerator=type=nvidia-rtx-pro-6000-vws,count=AMOUNT,gpu-driver-version=LATEST \
       --disk-type=hyperdisk-balanced \
       --placement-type=COMPACT \
       --scopes="https://www.googleapis.com/auth/cloud-platform" \
       --local-ssd-count=LOCAL_SSD_COUNT
   

   Replace the following:

   • NODE_POOL_NAME: the name of your node pool.
   • CLUSTER_NAME: the name of your cluster.
   • REGION: the region for your cluster.
   • ZONE: one or more zones within your region that has the requested GPUs available, for example, us-central1-a. This is required when using compact placement.
   • MACHINE_TYPE: the machine type for your nodes, for example, g2-standard-4 for G2 machines and g4-standard-48 for G4 machines.
   • AMOUNT: the number of GPUs to attach to each node.
   • LOCAL_SSD_COUNT: the number of Local SSD volumes to provision on each node.

Connect to your cluster

Connect to your cluster so that you can run the kubectl commands in the next sections:

gcloud container clusters get-credentials CLUSTER_NAME \
    --region=REGION

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• REGION: the region for your cluster.

For more information, see Install kubectl and configure cluster access.

Configure your Pod manifests (Autopilot only)

If you created an Autopilot cluster, you must select the appropriate GPUs in your Pod manifests so that GKE provisions the hardware.

1. Specify the chosen GPU type and specific reservation by using node selectors:

   spec:
     nodeSelector:
       cloud.google.com/gke-accelerator: ACCELERATOR
       cloud.google.com/gke-gpu-driver-version: latest
       # Optional: Include if using reserved capacity
       cloud.google.com/reservation-name: RESERVATION_NAME
       cloud.google.com/reservation-affinity: "specific"
   

   Replace the following:

   • ACCELERATOR: the accelerator that you reserved. You must use nvidia-l4 (for G2), nvidia-rtx-pro-6000 (for G4), or nvidia-rtx-pro-6000-vws (for G4 with Virtual Workstation).
   • RESERVATION_NAME: the name of the Compute Engine capacity reservation. To consume shared reservations, or specific blocks and sub-blocks of reservations, see the respective sections in Consuming reserved zonal path resources.

2. Add the following resources to the container that requests GPUs:

   containers:
     - name: my-container
       resources:
         limits:
           nvidia.com/gpu: AMOUNT
   

   Replace AMOUNT with the number of GPUs to attach to each node.

What's next

• Schedule GKE workloads
• Enable node health prediction