Troubleshoot issues with deployed workloads

Autopilot Standard

This page shows you how to resolve errors with your deployed workloads in Google Kubernetes Engine (GKE).

For more general advice about troubleshooting your applications, see Troubleshooting Applications in the Kubernetes documentation.

All errors: Check Pod status

If there are issues with a workload's Pods, Kubernetes updates the Pod status with an error message. View these errors by checking the status of a Pod using the Google Cloud console or the kubectl command-line tool.

Console

Perform the following steps:

1. In the Google Cloud console, go to the Workloads page.

   Go to Workloads

2. Select the workload that you want to investigate. The Overview tab displays the status of the workload.

3. From the Managed Pods section, click any error status message.

kubectl

To see all Pods running in your cluster, run the following command:

kubectl get pods

The output is similar to the following:

NAME       READY  STATUS             RESTARTS  AGE
POD_NAME   0/1    CrashLoopBackOff   23        8d

Potential errors are listed in the Status column.

To get more information about a specific Pod, run the following command:

kubectl describe pod POD_NAME

Replace POD_NAME with the name of the Pod that you want to investigate.

In the output, the Events field shows more information about errors.

If you'd like more information, view the container logs:

kubectl logs POD_NAME

These logs can help you identify if a command or code in the container caused the Pod to crash.

After you identify the error, use the following sections to try and resolve the issue.

Error: CrashLoopBackOff

A status of CrashLoopBackOff doesn't mean there's a specific error, instead it indicates that a container is repeatedly crashing after restarting.

For more information, see Troubleshoot CrashLoopBackOff events.

Errors: ImagePullBackOff and ErrImagePull

A status of ImagePullBackOff or ErrImagePull indicates that the image used by a container cannot be loaded from the image registry.

For guidance on troubleshooting these statuses, see Troubleshoot image pulls.

Error: OutOfPods

A status of OutOfPods indicates that a node cannot run a Pod because the node has reached its maximum Pod capacity.

Symptoms

You might see a message in the Pod's events similar to the following:

Node didn't have enough resource: pods, requested: 1, used: 32, capacity: 32

Cause

This error occurs when there is a request to schedule a Pod on a node that is already at capacity. This situation can commonly occur during node startup, for example when the kube-scheduler component assigns Pods to a new node before the kubelet agent has reported the presence of static Pods like the kube-proxy component, which require their own Pod capacity.

Resolution

To resolve this issue, try one of the following solutions:

• Increase the maximum number of Pods per node. If your nodes consistently reach their Pod limit, increase the --max-pods-per-node setting for your node pools. Increasing the number of Pods might require larger nodes to handle the increased resource demands.

• Enable cluster autoscaler and node auto-provisioning. If you frequently run out of Pod capacity, enabling the cluster autoscaler and node auto-provisioning can help ensure that your cluster has enough nodes to meet the demand of your workloads.

• Change the autoscaling profile. If you already use the cluster autoscaler, try changing the autoscaling profile to the balanced profile instead of the optimize-utilization profile. The optimize-utilization profile can increase the likelihood of OutOfPods errors because it attempts to place Pods on the most utilized nodes.

Error: Pod unschedulable

A status of PodUnschedulable indicates that your Pod cannot be scheduled because of insufficient resources or some configuration error.

If you have configured control plane metrics, you can find more information about these errors in scheduler metrics and API server metrics.

Use the unschedulable Pods interactive playbook

You can troubleshoot PodUnschedulable errors using the interactive playbook in the Google Cloud console:

1. Go to the unschedulable Pods interactive playbook:

   Go to Playbook

2. In the Cluster drop-down list, select the cluster that you want to troubleshoot. If you can't find your cluster, enter the name of the cluster in the Filter field.

3. In the Namespace drop-down list, select the namespace that you want to troubleshoot. If you can't find your namespace, enter the namespace in the Filter field.

4. To help you identify the cause, work through each of the sections in the playbook:

   1. Investigate CPU and Memory
   2. Investigate Max Pods per Node
   3. Investigate Autoscaler Behavior
   4. Investigate Other Failure Modes
   5. Correlate Change Events

5. Optional: To get notifications about future PodUnschedulable errors, in the Future Mitigation Tips section, select Create an Alert.

Error: Insufficient resources

A PodUnschedulable status can occur if there are insufficient CPU, memory, or other resources to meet the Pod's requests.

Symptoms

You might encounter an error indicating a lack of CPU, memory, or another resource. For example: No nodes are available that match all of the predicates: Insufficient cpu (2). This message indicates that, on two nodes, there isn't enough CPU available to fulfill a Pod's requests.

Cause

If your Pod resource requests exceed that of a single node from any eligible node pools, GKE doesn't schedule the Pod and also doesn't trigger scale-up to add a new node.

Your cluster runs system containers in the kube-system namespace. Those containers also use cluster resources.

Resolution

Try the following solutions:

• Adjust the Pod's resource request by specifying a lower value in the spec: containers: resources: requests field. The default CPU request is 100m or 10% of a CPU (or one core).

• Create a new node pool with nodes that have sufficient resources to meet the Pod's requests.

• Enable node auto-provisioning so that GKE can automatically create node pools with nodes where the unscheduled Pods can run.

Error: MatchNodeSelector

A MatchNodeSelector error indicates that there are no nodes that match the Pod's label selector.

Symptoms

The Pod status or events show a MatchNodeSelector error.

Cause

The labels specified in the Pod manifest's nodeSelector field don't exist on any nodes in the cluster.

Resolution

To resolve this error, ensure that the labels specified in the Pod's nodeSelector field match the labels on at least one node in your cluster:

1. Identify the label requirements that the Pod is looking for by checking its spec: nodeSelector field.

2. To see if any labels match the Pod's requirements, view the actual labels assigned to the nodes in your cluster:

   kubectl get nodes --show-labels
   

3. If a node is intended to run this Pod, attach the necessary label:

   kubectl label nodes NODE_NAME LABEL_KEY=LABEL_VALUE
   

   Replace the following:

   • NODE_NAME: the node that you want to add a label to.
   • LABEL_KEY: the label's key.
   • LABEL_VALUE: the label's value.

For more information, see Assigning Pods to Nodes in the Kubernetes documentation.

Error: PodToleratesNodeTaints

A PodToleratesNodeTaints error indicates that the Pod can't be scheduled to any node because the Pod doesn't have tolerations that correspond to existing node taints.

Symptoms

The Pod status or events show a PodToleratesNodeTaints error.

Cause

The Pod can't be scheduled to any node because the Pod doesn't have tolerations that correspond to existing node taints.

Resolution

1. Check the taints on the node:

   kubectl describe nodes NODE_NAME
   

   In the output, check the Taints field, which lists key-value pairs and scheduling effects. If the effect listed is NoSchedule, then no Pod can be scheduled on that node unless it has a matching toleration.

2. Remove the taint from the node. For example, to remove a NoSchedule taint, run the following command:

   kubectl taint nodes NODE_NAME key:NoSchedule-
   

Error: PodFitsHostPorts

The PodFitsHostPorts error means that a node is trying to use a port that's already occupied.

Symptoms

The Pod status shows a PodFitsHostPorts error.

Cause

A Pod is requesting a host port that's already in use by another Pod or process on the target node.

Resolution

To resolve the issue, consider following Kubernetes best practices and use a NodePort Service instead of the hostPort setting.

If you must use a host port, check the manifests of the Pods and make sure that all Pods on the same node have unique values defined for the hostPort setting.

Error: Does not have minimum availability

This error can occur if a node has adequate resources but isn't available for scheduling.

Symptoms

• You see the Does not have minimum availability error.

• The node's status shows a SchedulingDisabled status or a Cordoned status.

Cause

The node's cordoned status prevents new Pods from being scheduled on it.

Resolution

To make the node available for scheduling Pods again, uncordon it:

Console

Perform the following steps:

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Select the cluster that you want to investigate. The Nodes tab displays the Nodes and their status.

To enable scheduling on the node, perform the following steps:

1. From the list, click the node that you want to investigate.

2. From the Node Details section, click Uncordon.

kubectl

To get statuses of your nodes, run the following command:

kubectl get nodes

To enable scheduling on the node, run:

kubectl uncordon NODE_NAME

Error: Maximum Pods per node limit reached

A Too many pods error indicates that a Pod cannot schedule because the target node has reached its maximum configured Pod capacity.

Symptoms

• The Pods are stuck in an Unschedulable state.
• You see a message including the phrase Too many pods.

Cause

The Maximum Pods per node limit is reached by all nodes in the cluster.

Resolution

To resolve this error, complete the following steps:

1. Check the Maximum pods per node configuration from the Nodes tab in GKE cluster details in the Google Cloud console.

2. Get a list of nodes:

   kubectl get nodes
   

3. For each node, verify the number of Pods running on the node:

   kubectl get pods -o wide | grep NODE_NAME | wc -l
   

4. If the limit is reached, add a new node pool or add additional nodes to the existing node pool.

Issue: Maximum node pool size reached with cluster autoscaler enabled

This issue occurs when a node pool has reached its maximum configured size under the cluster autoscaler.

Symptoms

GKE doesn't trigger scale up for a Pod that would otherwise be scheduled with this node pool. Instead, the Pod remains in a Pending state.

Cause

The node pool has reached its maximum size according to its cluster autoscaler configuration.

Resolution

Increase the maximum size of the node pool by changing the cluster autoscaler configuration.

Issue: Maximum node pool size reached with cluster autoscaler disabled

This issue occurs when a node pool has reached its maximum size and the cluster autoscaler is disabled.

Symptoms

GKE cannot schedule the Pod with the node pool.

Cause

The node pool has reached its maximum number of nodes, and the cluster autoscaler is disabled.

Resolution

To resolve this issue, try one of the following solutions:

• Increase the size of your node pool.
• Enable cluster autoscaler to resize your cluster automatically.

Error: Unbound PersistentVolumeClaims

An Unbound PersistentVolumeClaims error indicates that the Pod references a PersistentVolumeClaim that isn't bound.

Symptoms

The Pod status or events show an Unbound PersistentVolumeClaims error.

Cause

This error can occur because of one of the following reasons:

• Your PersistentVolume failed to provision.
• There was a configuration error during manual pre-provisioning of a PersistentVolume and its binding to a PersistentVolumeClaim.

Resolution

1. Verify if provisioning failed by getting the events for your PersistentVolumeClaim:

   kubectl describe pvc STATEFULSET_NAME-PVC_NAME-0
   

   Replace the following:

   • STATEFULSET_NAME: the name of the StatefulSet object.
   • PVC_NAME: the name of the PersistentVolumeClaim object.

2. Try to pre-provision the volume again.

Error: Insufficient quota

If GKE attempts to scale up your cluster to schedule a Pod but encounters quota constraints, the scale-up fails.

Symptoms

You receive the scale.up.error.quota.exceeded error message in your Pod's events.

Cause

Scaling up the cluster would exceed your project's available quota.

Resolution

Verify that your project has sufficient Compute Engine quota for GKE to scale up your cluster. For more information, see ScaleUp errors.

Issue: Deprecated APIs

Using APIs that are no longer supported in your manifests can prevent workload deployment.

Symptoms

Workloads fail to deploy or run because of deprecated API usage.

Cause

Your manifests use deprecated APIs that are removed in your cluster's minor version.

Resolution

Ensure that you are not using deprecated APIs. Update your manifests to use supported APIs. For more information, see Feature and API deprecations.

Error: Didn't have free ports for the requested Pod ports

Binding a Pod to a host port limits where GKE can schedule the Pod because each hostIP address, hostPort setting, and protocol value combination must be unique.

Symptoms

You see an error similar to the following:

0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Cause

Multiple Pods on the same node specify the same value defined in the hostPort field.

Resolution

To resolve this issue, try one of the following solutions:

• Follow Kubernetes best practices and use a NodePort Service instead of a host port.
• If you must use a host port, check the manifests of the Pods and make sure that all Pods on the same node have unique values defined for the hostPort field.

Issue: Application and probe failures in Pods

This issue occurs when you run applications that use HTTPS to communicate with a server.

Symptoms

Failures in these applications are similar to the following:

• Pods don't start, and containers crash with exit code 137.

• Liveness or readiness probes fail with an error message similar to the following:

  probeResult="failure" output="Get "https://example.com/healthy": EOF"
  

• Pods run as expected, but application logs show connection failures.

Cause

Kubernetes versions 1.30 and later use Golang versions that disable the following TLS cipher suites:

• TLS_RSA_WITH_AES_128_GCM_SHA256
• TLS_RSA_WITH_AES_256_GCM_SHA384
• TLS_RSA_WITH_AES_128_CBC_SHA
• TLS_RSA_WITH_AES_256_CBC_SHA
• TLS_RSA_WITH_3DES_EDE_CBC_SHA

Resolution

Use supported cipher suites from TLS 1.2 and later.

What's next

• If you can't find a solution to your problem in the documentation, see Get support for further help, including advice on the following topics:

  • Opening a support case by contacting Cloud Customer Care.
  • Getting support from the community by asking questions on StackOverflow and using the google-kubernetes-engine tag to search for similar issues. You can also join the #kubernetes-engine Slack channel for more community support.
  • Opening issues or feature requests by using the public issue tracker.