Scale GKE workloads to and from zero using HPA

Autopilot Standard

This tutorial shows you how to optimize resource utilization in Google Kubernetes Engine (GKE) by configuring workloads to automatically scale to zero replicas when idle, and scale back up as demand increases. This approach integrates the horizontal Pod autoscaler (HPA) with GKE's managed autoscaling infrastructure to manage scaling based on external metrics.

Configure your deployment to scale to zero by setting the value of the minReplicas field to 0 and defining a metric with the External or Object type within your HPA manifest. GKE monitors these metrics through the AutoscalingMetric custom resource, which helps ensure efficient resource management for your applications.

With this configuration, you don't need to use third-party metrics adapters, such as KEDA, to scale GKE workloads. This solution manages metric ingestion and scaling recommendations directly in the GKE control plane, reducing cluster management overhead.

In this tutorial, you deploy an example asynchronous worker application that processes messages from a Pub/Sub queue. You configure a horizontal Pod autoscaler to monitor the queue depth (pubsub.googleapis.com:num_undelivered_messages) using an AutoscalingMetric custom resource:

When messages arrive in the subscription: GKE scales up worker Pods to process the queue.
When the queue is empty: GKE automatically scales the worker Deployment down to zero replicas.

This tutorial is for Application developers, Platform admins and operators, and DevOps who want to optimize resource usage in GKE by scaling workloads to zero when they are idle.

Considerations

Before configuring workloads to scale to zero, review the following considerations:

Scaling workloads to and from zero using HPA requires GKE cluster control plane and nodes to run version 1.37 or later in both new and upgraded existing clusters. If you use an existing cluster, verify its version or upgrade your cluster or its nodes to version 1.37 or later.
Your HPA manifest must use the apiVersion: autoscaling/v2 configuration to support the minReplicas: 0 setting and external metrics.
Before downgrading node pools to a version earlier than 1.37, update any HPA manifests configured to scale to and from zero by setting the minReplicas field to 1 or greater. Versions earlier than 1.37 don't support the minReplicas: 0 setting, which can cause workloads to remain stuck at zero replicas.
You must configure at least one External or Object metric (such as a queue depth) in your horizontal Pod autoscaler. GKE cannot collect CPU or memory (Resource) metrics when a workload has zero Pods, so resource metrics alone cannot trigger scaling up from zero.
The AutoscalingMetric, HorizontalPodAutoscaler, and target Deployment must reside in the same Kubernetes namespace.

Before you begin

Install the Google Cloud CLI.
Configure the gcloud CLI to use your federated identity.

For more information, see Sign in to the gcloud CLI with your federated identity.
To initialize the gcloud CLI, run the following command:
```
gcloud init
```
Create or select a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
- Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
- Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
  Replace PROJECT_ID with a name for the Google Cloud project you are creating.
- Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
  Replace PROJECT_ID with your Google Cloud project name.
Verify that billing is enabled for your Google Cloud project.
Enable the GKE and Pub/Sub APIs:
Roles required to enable APIs
To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.
```
gcloud services enable container.googleapis.com pubsub.googleapis.com
```

Required roles

To get the permissions that you need to complete this tutorial, ask your administrator to grant you the following IAM roles on your project:

GKE Cluster Admin (roles/container.clusterAdmin)
Pub/Sub Admin (roles/pubsub.admin)
Project IAM Admin (roles/resourcemanager.projectIamAdmin)
Service Account User (roles/iam.serviceAccountUser)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Set up your environment

For simplicity, the commands in this tutorial create all resources (the GKE cluster, and the Pub/Sub topic and subscription) within a single Google Cloud project (PROJECT_ID).

To set up your environment, follow these steps:

Set environment variables:
```
export PROJECT_ID=PROJECT_ID
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format 'get(projectNumber)')
export LOCATION=LOCATION
```
Replace the following:
- PROJECT_ID: your Google Cloud project ID.
- LOCATION: the region or zone where you want to create your GKE cluster, such as us-central1. For Autopilot clusters, specify a region.
Note: These environment variables are used throughout this tutorial. If you open a new shell session or disconnect, run these commands again to set the variables.
Create a GKE cluster running version 1.37 or later with Workload Identity Federation for GKE enabled. We recommend that you use an Autopilot cluster for a fully managed Kubernetes experience and to maximize cost savings when workloads scale to zero. To choose the mode of operation that's the best fit for your workloads, see Choose a GKE mode of operation:
Autopilot
Create an Autopilot cluster:
gcloud container clusters create-auto scale-to-zero \ --project=${PROJECT_ID} \ --location=${LOCATION}
Workload Identity Federation for GKE is enabled by default on Autopilot clusters.
Standard
Create a Standard cluster with Workload Identity Federation for GKE enabled:
gcloud container clusters create scale-to-zero \ --project=${PROJECT_ID} \ --location=${LOCATION} \ --workload-pool=${PROJECT_ID}.

Configure kubectl to communicate with your cluster:

gcloud container clusters get-credentials scale-to-zero \
    --project=${PROJECT_ID} \
    --location=${LOCATION}

Create Pub/Sub resources

This tutorial uses Pub/Sub queue depth as an example external metric source.

To create a Pub/Sub topic and subscription, follow these steps:

Create a Pub/Sub topic:

gcloud pubsub topics create my-worker-topic \
    --project=${PROJECT_ID}

Create a subscription attached to the topic:

gcloud pubsub subscriptions create my-worker-subscription \
    --topic=my-worker-topic \
    --project=${PROJECT_ID}

Set up Workload Identity Federation for GKE

Configure Workload Identity Federation for GKE to allow your worker application to authenticate with Google Cloud APIs and consume messages from Pub/Sub.

GKE automatically handles authentication with Cloud Monitoring for AutoscalingMetric resources in the same project. To learn more about defining metrics for autoscaling, see Fetch custom or external metrics from Cloud Monitoring.

To configure Workload Identity Federation for GKE for your worker workload, follow these steps:

Create a Kubernetes service account for your worker application in the default namespace:
```
kubectl create serviceaccount async-worker-sa \
    --namespace default
```

Grant the roles/pubsub.subscriber role to the Kubernetes service account so the application can receive messages from your Pub/Sub subscription:

gcloud projects add-iam-policy-binding projects/${PROJECT_ID} \
    --role=roles/pubsub.subscriber \
    --member=principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}./subject/ns/default/sa/async-worker-sa

For more information, see Configure applications to use Workload Identity Federation for GKE.

Create the example Deployment

Before you can create an HPA object, you must create the workload it will monitor.

To create the example Deployment, follow these steps:

Save the following manifest as async-worker.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: async-worker
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: async-worker
  template:
    metadata:
      labels:
        app: async-worker
    spec:
      containers:
      - name: async-worker
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 100Mi

Apply the async-worker.yaml Deployment:
```
kubectl apply -f async-worker.yaml
```

Configure a workload to scale to and from zero

In this section, you configure the async-worker Deployment to scale down to zero when the Pub/Sub queue is empty, and scale back up as new messages arrive.

Create the AutoscalingMetric resource

To define the external signal monitored by GKE, create the AutoscalingMetric custom resource. In the following example manifest, the metric queries Cloud Monitoring for the number of undelivered Pub/Sub messages on the my-worker-subscription subscription.

To create the AutoscalingMetric resource, follow these steps:

Save the following manifest as the pubsub-metric.yaml file:

apiVersion: autoscaling.gke.io/v1beta1
kind: AutoscalingMetric
metadata:
  name: pubsub-queue-depth
  namespace: default
spec:
  metrics:
  - promql:
      name: pubsub-undelivered
      query: >
          {
            "pubsub.googleapis.com/subscription/num_undelivered_messages",
            subscription_id="my-worker-subscription"
          }

Apply the pubsub-metric.yaml manifest:
```
kubectl apply -f pubsub-metric.yaml
```
Verify the metric status and retrieve the metric identifier:
```
kubectl describe autoscalingmetric pubsub-queue-depth
```
In the Status section of the output, verify that no errors are listed and note the Hpa Name value that's in the autoscaling.gke.io|CUSTOM_RESOURCE_NAME|METRIC_NAME format. You'll reference this external metric identifier when you create the HorizontalPodAutoscaler object in the next section. If the Status section reports configuration errors or metrics are not retrieved as expected, see Troubleshoot metrics that are fetched for autoscaling.

Configure the horizontal Pod autoscaler

To configure autoscaling behavior, create a HorizontalPodAutoscaler resource targeting the Deployment.

To configure the horizontal Pod autoscaler, follow these steps:

Save the following manifest as the worker-hpa.yaml file:
```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: async-worker-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: async-worker
  minReplicas: 0
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: autoscaling.gke.io|pubsub-queue-depth|pubsub-undelivered
      target:
        type: AverageValue
        averageValue: "10"
```
This manifest configures the following key fields:
- minReplicas: 0: enables scale-to-zero by allowing the controller to scale the Deployment down to 0 replicas when demand drops to zero.
- type: External: configures an external metric source so the HPA can trigger scale-up when the workload has zero Pods.
- name: autoscaling.gke.io|pubsub-queue-depth|pubsub-undelivered: maps the HPA directly to the AutoscalingMetric resource created in the previous step by using the autoscaling.gke.io|CUSTOM_RESOURCE_NAME|METRIC_NAME identifier format.
Apply the worker-hpa.yaml manifest:
```
kubectl apply -f worker-hpa.yaml
```

Verify zero-scale behavior and conditions

When all messages in the Pub/Sub subscription are processed, the horizontal Pod autoscaler evaluates the zero demand and scales the Deployment down to 0 replicas.

To verify that the horizontal Pod autoscaler actuated the zero state, inspect the status conditions of the async-worker-hpa resource by running the following command:

kubectl describe hpa async-worker-hpa

The output is similar to the following:

Name:             async-worker-hpa
Namespace:        default
Reference:        Deployment/async-worker
Metrics:          ( current / target )
  "autoscaling.gke.io|pubsub-queue-depth|pubsub-undelivered" (external metric):  0 / 10
Min replicas:     0
Max replicas:     20
Deployment pods:  0 current / 0 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from external metric
  ScaledToZero    True    ScaledToZero         the HPA has scaled the target resource to 0 replicas due to zero metric demand

Understand the `ScaledToZero` condition

The ScaledToZero condition indicates whether the horizontal Pod autoscaler has scaled the workload to zero replicas:

ScaledToZero: True (Reason: ScaledToZero): indicates that the HPA controller successfully scaled your workload to 0 replicas because external metric demand dropped to zero. The HPA remains active (ScalingActive: True) and continuously polls GKE to detect when workload demand increases.
ScaledToZero: False: indicates that the workload has scaled up to one or more replicas.

If you manually scale a Deployment to zero replicas, for example, with the kubectl scale --replicas=0 command, the HPA pauses autoscaling (ScalingActive: False) to prevent conflicting changes. To resume autoscaling, scale the deployment back to one or more replicas (kubectl scale deployment async-worker --replicas=1).

For troubleshooting scenarios where workloads fail to scale to zero or fail to scale up from zero, see Troubleshoot scaling GKE workloads to and from zero using HPA. If the horizontal Pod autoscaler reports missing or invalid external metrics, see Troubleshoot metrics that are fetched for autoscaling.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, follow these steps:

Delete the GKE cluster:

gcloud container clusters delete scale-to-zero \
    --project=${PROJECT_ID} \
    --location=${LOCATION}

Delete the Pub/Sub subscription and topic:

gcloud pubsub subscriptions delete my-worker-subscription \
    --project=${PROJECT_ID}
gcloud pubsub topics delete my-worker-topic \
    --project=${PROJECT_ID}

What's next

Learn how to mitigate cold-start latency by using GKE Capacity buffers.
Learn how to diagnose scaling issues when scaling to and from zero using HPA.