Provision Managed Lustre on GKE using Cluster Toolkit

This document explains how to integrate Managed Lustre with GKE to create an optimized environment for demanding, data-intensive workloads like artificial intelligence (AI), machine learning (ML), and high performance computing (HPC) by using Cluster Toolkit.

In this document, you provision a GKE cluster and a Managed Lustre instance declaratively with Cluster Toolkit, automatically configure the Managed Lustre CSI driver and Kubernetes PersistentVolume resources, and run a sample workload on the cluster.

This document is intended for Machine learning (ML) engineers and Data and AI specialists who are interested in exploring Kubernetes container orchestration capabilities backed by Managed Lustre instances. To learn more about common roles and example tasks referenced in Google Cloud content, see Common GKE user roles and tasks.

Background

This section describes the key technologies used in this document.

Cluster Toolkit

Cluster Toolkit is an open-source tool that simplifies deploying and managing AI/ML and HPC environments on Google Cloud. Cluster Toolkit uses modular, declarative blueprints to automate the provisioning of GKE clusters, networks, compute accelerators, and high-performance storage.

For more information, see the Cluster Toolkit overview.

Managed Lustre

Managed Lustre is a high-performance, parallel file system service designed for demanding workloads. The Managed Lustre CSI driver lets you integrate Managed Lustre instances with GKE, by using standard Kubernetes PersistentVolumeClaims (PVCs) and PersistentVolumes (PVs). This driver is particularly beneficial for AI/ML and HPC workloads requiring persistent, scalable, and high-throughput storage.

For more information, see About the Managed Lustre CSI driver.

Accelerator consumption options

Flex-start and Spot VMs let you optimize compute provisioning by paying only for the resources you need. For more information, see About accelerator consumption options for AI/ML workloads in GKE.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Cloud Managed Lustre API and the Google Kubernetes Engine API.
    • Enable APIs
  • To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.

Prepare your environment

  1. Set up Cluster Toolkit.

  2. Set your default environment variables and get the IP address of your deployment machine:

    gcloud config set project PROJECT_ID
export PROJECT_ID=$(gcloud config get project)
export REGION=REGION
export ZONE=ZONE
export DEPLOYMENT_NAME=DEPLOYMENT_NAME
export AUTHORIZED_CIDR="$(curl -s ifconfig.me)/32"

    Replace the following values:

    • PROJECT_ID: your Google Cloud project ID.
    • REGION: the region for your deployment, for example us-central1.
    • ZONE: a zone supporting Managed Lustre, for example us-central1-a.
    • DEPLOYMENT_NAME: the name for your deployment.

Deploy the cluster and storage

  1. Deploy the infrastructure by using the examples/gke-managed-lustre.yaml blueprint from the Cluster Toolkit repository:

    gcluster deploy examples/gke-managed-lustre.yaml \
    --vars project_id=${PROJECT_ID},deployment_name=${DEPLOYMENT_NAME},region=${REGION},zone=${ZONE},authorized_cidr=${AUTHORIZED_CIDR}

    Cluster Toolkit provisions the VPC, creates the Managed Lustre instance, provisions the GKE cluster with the Managed Lustre CSI driver enabled, and creates the Kubernetes PersistentVolume and PersistentVolumeClaim.

  2. Configure kubectl to access your new cluster:

    gcloud container clusters get-credentials ${DEPLOYMENT_NAME} \
    --zone=${ZONE} \
    --project=${PROJECT_ID}

Run a workload with Managed Lustre

Submit a workload that mounts the Managed Lustre volume by using the gcluster job submit command. For more information about job submission options, see the Cluster Toolkit Job Submission Guide.

gcluster job submit \
    --name="test-lustre-$(date +%H%M)" \
    --cluster=${DEPLOYMENT_NAME} \
    --project=${PROJECT_ID} \
    --location=${ZONE} \
    --compute-type=n2-standard-16 \
    --image=ubuntu:latest \
    --command="df -h && echo 'Hello from Managed Lustre on GKE' > /lustre-data/hello.txt && cat /lustre-data/hello.txt" \
    --mount="gke-lustre-instance-pvc;/lustre-data;rw"

To inspect workload logs and execution status, use the gcluster job logs command:

gcluster job logs "test-lustre-$(date +%H%M)" \
    --cluster=${DEPLOYMENT_NAME} \
    --project=${PROJECT_ID} \
    --location=${ZONE}

The output is similar to the following:

Filesystem                Size  Used Avail Use% Mounted on
overlay                   193G   11G  183G   6% /
tmpfs                      64M     0   64M   0% /dev
10.8.68.5@tcp://lustrefs   35T   22M   35T   1% /lustre-data
shm                        64M     0   64M   0% /dev/shm
Hello from Managed Lustre on GKE

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this document, delete the deployment by using Cluster Toolkit:

gcluster destroy ${DEPLOYMENT_NAME}

What's next