Troubleshoot issues with external Application Load Balancers

This guide describes how to troubleshoot configuration issues for external Application Load Balancers. Before investigating issues, familiarize yourself with the following pages:

Backends have incompatible balancing modes

When creating a load balancer, you might see the error:

Validation failed for instance group INSTANCE_GROUP:

backend services 1 and 2 point to the same instance group
but the backends have incompatible balancing_mode. Values should be the same.

This happens when you try to use the same backend in two different load balancers, and the backends don't have compatible balancing modes.

For more information, see the following:

Troubleshoot general connectivity issues

Unexplained '5XX' errors

An `HTTP 5XX' error can be returned by a first-layer GFE, a second-layer GFE, or a backend, depending on where the error condition occurs.

This section describes how to troubleshoot 5XX errors that can occur at different stages of the request distribution process for GFE-based external Application Load Balancers.

Identify source of '5XX' errors using Cloud Logging

For error conditions caused by a communications issue between the load balancer proxy and its backends, the load balancer generates an HTTP error response code (5XX) and returns that error response code to the client. Not all HTTP 5XX errors are generated by the load balancer—for example, if a backend sends an HTTP 5XX response to the load balancer, the load balancer relays that response to its client.

To determine if an HTTP 5XX response was relayed from a backend or if it was generated by the load balancer proxy, check the statusDetails field in Cloud Logging.

If statusDetails is response_sent_by_backend, the load balancer relayed the 5XX response from the backend. Troubleshoot the issue on your backends.
If statusDetails is any other failure message, the 5XX response is generated by the load balancer.

Configuration changes to the global external Application Load Balancer, such as addition or removal of a backend service, can result in a brief period of time where you see HTTP 502 responses with statusDetails as failed_to_pick_backend. This is normal during propagation of configuration changes to GFEs globally.

Before you begin troubleshooting, verify backend health

Before troubleshooting 5XX errors, verify that your backends are healthy and that health checks are passing. If backends are unhealthy, the second-layer GFE cannot forward requests to them, which can cause 5XX errors even if everything else is configured correctly.

Verify that there is a firewall rule configured to allow health checks. In the absence of one, health checks fail, and load balancer logs might show a statusDetails of failed_to_pick_backend.
Verify that health check traffic reaches your backend VMs. To do this, enable health check logging and search for successful log entries. For new load balancers, you might not see successful health check log entries immediately. This might be because the backend's initial health state has not yet changed from UNHEALTHY to a different state. You see successful health check log entries only after the health check prober receives an HTTP 200 OK response from the backend.

If health checks are failing, troubleshoot your backend application and firewall rules. If health checks are passing but you are still seeing 5XX errors, check the statusDetails field in Cloud Logging to identify the source of error, as described in the following section.

Troubleshoot errors based on `statusDetails`

If HTTP 5XX errors persist, use the statusDetails field in Cloud Logging to identify the cause and troubleshoot accordingly.

The global external Application Load Balancer and the regional external Application Load Balancer generate meaningful HTTP status codes, such as HTTP 503 Service Unavailable and HTTP 504 Gateway Timeout.
The classic Application Load Balancer always uses the HTTP 502 Bad Gateway status code for all errors generated by the load balancer.

statusDetails	Potential cause and solution
`failed_to_pick_backend` `failed_to_pick_backend_by_hash`	Cause: The second-layer GFE was unable to select a healthy backend to route the request to. This error can occur for one of the following reasons: All regions are at or above capacity. The load balancer cannot find any available backend based on the computed hash. The load balancer cannot find any backends matching the service configuration. Solution: Verify that your backends are healthy by following the steps outlined in Verify backend health. Ensure that health checks are configured correctly with the right port, path, and protocol. Ensure that backend firewall rules allow health check traffic from the health check probers `35.191.0.0/16`. If you have some unhealthy backend instance groups, note that failover or overflow is restricted to a limited number of alternate instance groups. If you observe `failed_to_pick_backend` or `failed_to_pick_backend_by_hash` despite having healthy instance groups, drain the unhealthy instance groups by setting their capacity to zero. This forces the load balancer to forward traffic to the healthy instance groups.
`failed_to_connect_to_backend`	Cause: The second-layer GFE failed to establish a connection with a backend instance (could not get a SYN-ACK). This error can also be caused by an internal GFE error that prevents it from connecting to the backend, or from a regional outage or network disruption (such as a fiber cut) that prevents first-layer GFEs from communicating with second-layer GFEs. Solution: Verify that firewall rules on your backend VMs or in your VPC network allow traffic to your backends from second-layer GFE ranges `35.191.0.0/16`. Verify that the application on your backend instances is running and listening on the port configured in the backend service. Check backend instance metrics (CPU, memory, connections) for signs of resource exhaustion.
`backend_connection_closed_before_data_sent_to_client`	Cause: The connection between second-layer GFE and the backend was closed unexpectedly before the response could be sent to the client. This issue can be caused by the backend web server or an intermediate device. It can also occur when you use GKE if the pods are scaling down or terminating, and the load balancer backends are of type NEGs. Solution: Set the HTTP keepalive timeout on the backend web server (like Apache or Nginx) to be greater than the load balancer's 600-second timeout. The recommended value is 620 seconds. If you use GKE, this occurs when either the pods are scaling down or terminating, and the load balancer backends are NEGs. To resolve this, consider increasing `terminationGracePeriodSeconds` in your pod or deployment configuration to allow existing connections to close gracefully. The ideal value is: `T_grace > T_preStop + T_drain + T_buffer` Where: `T_preStop` is the container's `preStop` hook sleep duration, which is typically 120 seconds to allow the endpoint to be removed from the NEG. `T_drain` is the backend service connection draining timeout, which is typically 60 seconds. `T_buffer` is an additional 30–45 second buffer to allow the pod to safely shut down. For standard configurations, the recommended value is at least 210 seconds (3.5 minutes). Check backend application logs for errors that might cause the connection to be closed. Check backend instance metrics (CPU, memory, connections) for signs of resource exhaustion.
`backend_timeout`	Cause: second-layer GFE established a connection with the backend, but the backend did not send a response within the configured backend service timeout. Solution: If your application needs more time to process requests, increase the backend service timeout. Check for performance issues on your backend instances (CPU, I/O, application delays) that might prevent a timely response. If backends are overloaded, consider scaling backend resources or optimizing the application.
`retriable_error`	Cause: This 503 error can occur if infrastructure-as-code tools (like Terraform) update load balancer rules in a way that temporarily removes URL map rules before re-adding them, or due to transient internal Google network or configuration issues during rollouts or configuration pushes. Solution: If using automation, check that it performs in-place updates rather than delete-then-create updates for URL map rules. If you see this error during or after a configuration change, it might be transient; wait a few minutes for the configuration to propagate globally.

Resolving HTTP `408` errors

With HTTP traffic, the maximum amount of time for the client to complete sending its request is equal to the backend service timeout. If you see HTTP 408 responses with the jsonPayload.statusDetail client_timed_out, this means that there was insufficient progress while the request from the client was proxied or the response from the backend was proxied. If the problem is because of clients that are experiencing performance issues, you can resolve this issue by increasing the backend service timeout.

Load balanced traffic does not have the source address of the original client

The source IP address for packets, as seen by the backends, is not the external IP address of the load balancer. Proxy-based load balancers such as the external Application Load Balancers use two TCP connections to transmit traffic from the client to the backends:

Connection 1, from original client to the load balancer (GFE or proxy-only subnet)
Connection 2, from the load balancer (GFE or proxy-only subnet) to the backend VM or endpoint

The source and destination IP addresses for each connection differ based on the type of external Application Load Balancer you're using. For details, see Source IP addresses for client packets .

Getting a permission error when trying to view an object in my Cloud Storage bucket

In order to serve objects through load balancing, the Cloud Storage objects must be publicly accessible. Make sure to update the permissions of the objects being served so they are publicly readable.

URL doesn't serve expected Cloud Storage object

The Cloud Storage object to serve is determined based on your URL map and the URL that you request. If the request path maps to a backend bucket in your URL map, the Cloud Storage object is determined by appending the full request path onto the Cloud Storage bucket that the URL map specifies.

For example, if you map /static/* to gs://[EXAMPLE_BUCKET], the request to https://<GCLB IP or Host>/static/path/to/content.jpg will try to serve gs://[EXAMPLE_BUCKET]/static/path/to/content.jpg. If that object doesn't exist, you will get the following error message instead of the object:


NoSuchKey
The specified key does not exist.

Compression isn't working

An external Application Load Balancer does not compress or decompress responses itself, but it can serve responses generated by your backend service that are compressed by using tools such as gzip or DEFLATE.

If responses served by the load balancer are not compressed but should be, check to be sure that the web server software running on your instances is configured to compress responses. By default, some web server software automatically disables compression for requests that include a Via header, which indicates that the request was forwarded by a proxy. Because it is a proxy, the external Application Load Balancer adds a Via header to each request as required by the HTTP specification. To enable compression, you may have to override your web server's default configuration to tell it to compress responses even if the request had a Via header.

To configure nginx backends to serve compressed responses proxied through an external Application Load Balancer:

Set the gzip_proxied directive appropriately (for example, to any), and
Set the gzip_vary directive to on.

To configure Apache backends to serve compressed responses proxied through an external Application Load Balancer:

Use the DEFLATE filter, and
Add Vary Accept-Encoding to the response header using the mod_headers module.

Troubleshoot unhealthy backends

Troubleshoot issues with HTTP/2 to the backends

Make sure that your backend instance is healthy and supports HTTP/2 protocol. You can verify this by testing connectivity to the backend instance using HTTP/2. Ensure that the VM uses HTTP/2 spec-compliant cipher suites. For example, certain TLS 1.2 cipher suites are disallowed by HTTP/2. Refer to the TLS 1.2 Cipher Suite Black List.

After you verify that the VM uses the HTTP/2 protocol, make sure your firewall setup allows the health checker and load balancer to pass through.

If there are no problems with the firewall setup, ensure that the load balancer is configured to talk to the correct port on the VM.

Troubleshoot external backend and internet NEG issues

Before investigating issues, familiarize yourself with the following pages:

Traffic does not reach the endpoints

After you configure a service, the new endpoint becomes reachable through the external Application Load Balancer when:

The endpoint is attached to the internet NEG.
The associated FQDN can be DNS resolved successfully (if you are using FQDN endpoint type).
The endpoint is accessible over the internet.

If traffic cannot reach the endpoint, which results in a 502 error code, query the _cloud-eoips.googleusercontent.com DNS TXT record using a tool like dig or nslookup. Note the CIDRs (following ip4:) and ensure these ranges are allowed by your firewall or cloud access control list (ACL).

After configuring an external backend, requests to external backend failed with a 5xx error

Check Logging.
Verify that the network endpoint group is configured with the correct IP:Port or FQDN:Port for your external backend.
If you are using FQDN, make sure that it is resolvable through Google Public DNS. You can verify that the FQDN is resolvable through Google Public DNS using these steps or the web interface directly.
If you are accessing the load balancer on its external IP only, and your origin web-server is expecting a hostname, ensure that you are sending a valid HTTP Host header to your backend by configuring a custom request header.
If you're communicating with a backend over HTTPS or HTTP2 (as set in the protocol field of the backend service) configured as an INTERNET_FQDN_PORT external backend endpoint, ensure that your origin is presenting a valid TLS (SSL) certificate and the configured FQDN matches a SAN (Subject Alternative Name) in the certificates' list of SANs. A valid certificate is defined as one signed by a public Certificate Authority and that has not expired.
When using INTERNET_FQDN_PORT external backend endpoints, self-signed certificates are not accepted by the load balancer, and are rejected.
When using HTTPS or HTTP/2 with INTERNET_IP_PORT type endpoints, no SSL certificate validation/SAN check is performed. This means one can use self-signed certificates. When using SSL, our recommendation is to use INTERNET_FQDN_PORT endpoints to make sure server certificates and SANs can be validated.

Troubleshoot Google tag gateway injection issues

Google tag gateway is not injecting tags correctly or is causing errors.

To resolve this issue, use the Google Cloud console Logs Explorer to analyze the logs generated by your load balancer. Google tag gateway issues don't affect the overall HTTP response code or the statusDetails of the webpage request. The HTTP response will be sent without modification even if Google tag injection fails. To identify gateway errors, check the grpcStatus of the plugin execution in your load balancer's JSON payload.

Ensure that Cloud Logging is enabled on the backend services that serve the website content for the domains where Google tag gateway is active. Configure this in the Google Cloud console by navigating to Load balancing > Edit > Backend configuration. For more information, see Enabling logging on an existing backend service. A logging sample rate greater than 0.0 must be set for that backend service.

In the Google Cloud console, go to the Logs Explorer page and select the correct project.

Go to Logs Explorer
Adjust the time range to cover the period when you suspect issues occurred. For more information, see Use the time-range selector.

To see logs for requests handled by the Google tag gateway injection plugin, use this query:

resource.type="http_load_balancer" logName="projects/PROJECT_ID/logs/requests"  \
jsonPayload.serviceExtensionInfo.extension="0-google-tag-gateway"

Replace PROJECT_ID with your project ID.

Identify potential errors by examining the jsonPayload.serviceExtensionInfo.perProcessingRequestInfo.grpcStatus field. This shows the status of the Google tag gateway extension itself.
- OK: The extension completed its processing without a gRPC error.
- Any value other than OK (for example, INTERNAL, UNAVAILABLE, DEADLINE_EXCEEDED): This indicates an error with execution of the Google tag gateway extension.

To find logs where the Google tag gateway extension reported an error, use the following query:

resource.type="http_load_balancer" logName="projects/PROJECT_IDlogs/requests" \
jsonPayload.serviceExtensionInfo.extension="0-google-tag-gateway" NOT jsonPayload.serviceExtensionInfo.perProcessingRequestInfo.grpcStatus="OK"

When the preceding query returns results, examine the entire log entry to correlate extension status with request outcome:
- Google tag gateway extension failure: a grpcStatus other than OK (especially on the RESPONSE_BODY event) means the tag injection process encountered an error. The specific gRPC code provides clues (for example, DEADLINE_EXCEEDED indicates a timeout).
- Impact on user request: if the grpcStatus for the Google tag gateway extension is not OK, check the httpRequest.status and jsonPayload.statusDetails in the same log entry. For example, a non-OK grpcStatus combined with httpRequest.status: 500 and jsonPayload.statusDetails: service_extensions_error suggests that the Google tag gateway extension failure caused the user to receive a server error.
- Issue frequency: analyzing the timestamps and frequency of these error logs helps determine if Google tag gateway injection issues are ongoing, intermittent, or tied to specific events.

If you encounter issues during setup that the documentation can't resolve, contact Google Ads Support.

Troubleshoot issues with external Application Load Balancers Stay organized with collections Save and categorize content based on your preferences.