Troubleshooting overview

This page provides general troubleshooting information for API Gateway.

Cannot run "gcloud api-gateway" commands

To run the gcloud api-gateway ... commands, you must have updated the Google Cloud CLI and enabled the necessary Google services. See Configuring your development environment for more.

Command "gcloud api-gateway api-configs create" says service account does not exist

If you run the gcloud api-gateway api-configs create ... command and receive an error in the form:

ERROR: (gcloud.api-gateway.api-configs.create) FAILED_PRECONDITION:
Service Account "projects/-/serviceAccounts/service_account_email" does not exist

Rerun the command but this time include the --backend-auth-service-account option to explicitly specify the email address of the service account to use:

gcloud api-gateway api-configs create CONFIG_ID \
  --api=API_ID --openapi-spec=API_DEFINITION \
  --backend-auth-service-account=SERVICE_ACCOUNT_EMAIL

Ensure that you have already assigned the necessary permissions to the service account as described in Configuring your development environment.

Determining the source of API error responses

If requests to your deployed API result in an error (HTTP status codes 400 through 599), it may not be clear from the response itself if the error originates from the Gateway or from your backend. To determine this:

  1. Go to the Logs Explorer page and select your project.

    Go to Logs Explorer

  2. Filter to the relevant gateway resource using the following Log query:

    resource.type="apigateway.googleapis.com/Gateway"
resource.labels.gateway_id="GATEWAY_ID"
resource.labels.location="GCP_REGION"

    Where:

    • GATEWAY_ID specifies the name of the gateway.
    • GCP_REGION is the Google Cloud region for the deployed gateway.

  3. Find the log entry matching the HTTP error response you want to investigate. For example, filter by httpRequest.status.

  4. Inspect the contents of the field jsonPayload.responseDetails.

If the value of the jsonPayload.responseDetails field is "via_upstream", then the error response originates from your backend and you will need to troubleshoot your backend directly. If it is any other value, then the error response originates from the Gateway; see the following sections of this document for further troubleshooting tips.

API request returns an HTTP 403 error

If a request to a deployed API returns an HTTP 403 error to the API client, it means the requested URL is valid but access is forbidden for some reason.

A deployed API has the permissions associated with roles granted to the service account that you used when you created the API config. Typically, the reason for the HTTP 403 error is that the service account does not have the necessary permissions to access the backend service.

If you defined the API and the backend service in the same Google Cloud Project, ensure that the service account has the Editor role assigned to it, or the role necessary to access the backend service. For example, if the backend service is implemented using Cloud Run functions, ensure that the service account has the Cloud Function Invoker role assigned to it.

API request returns an HTTP 401 or 500 error

If a request to a deployed API returns an HTTP 401 or 500 error to the API client, there may be a problem using the service account used when you created the API config to call your backend service.

A deployed API has the permissions associated with roles granted to the service account that you used when you created the API config. The service account is checked to make sure that it both exists, and can be used by the API gateway when the API is deployed.

If the service account is deleted or disabled after the gateway is deployed, the following sequence of events may occur:

  1. Immediately after the service account is deleted or disabled, you may see 401 HTTP responses in your gateway logs. If the jsonPayload.responseDetails field is set to "via_upstream" in the log entry's jsonPayload, this indicates that deleting or disabling the service account is the cause of the error.

  2. You may also see an HTTP 500 error without any corresponding log entry in the API Gateway's logs. If there are no requests to your gateway immediately after the service account is deleted or disabled, you may not see the HTTP 401 responses, but the HTTP 500 errors without corresponding API gateway logs are an indication that the gateway's service account may no longer be active.

If the backend for the failing request is another Google Cloud API (such as bigquery.googleapis.com), you will see 401 HTTP responses in your gateway logs with the jsonPayload.responseDetails field set to "via_upstream". This is because API Gateway authenticates to backends with an ID Token while other Google Cloud APIs require an Access Token.

API request returns an HTTP 500 error for a quota-enforced method

If you receive the following error, the gateway couldn't allocate quota for your request:

HTTP/2 500
{"code":500,"message":"Failed to call Service Control Quota."}

This error commonly occurs when you call a method that has a quota configured, but the quota metrics no longer exist for the API. On a gRPC gateway, the same failure is returned as the gRPC status code Internal.

Confirm the cause in your gateway logs

  1. Go to the Logs Explorer page and select your project.

    Go to Logs Explorer

  2. Run the following log query:

    resource.type="apigateway.googleapis.com/Gateway"
resource.labels.gateway_id="GATEWAY_ID"
jsonPayload.responseDetails="service_control_quota_error"
httpRequest.status=500

    Where GATEWAY_ID specifies the name of the gateway.

    The query filters on the status code as well as on jsonPayload.responseDetails because API Gateway uses the same responseDetails value for every quota rejection. A request that legitimately exceeded its quota produces the same value with an httpRequest.status of 429.

  3. Inspect the jsonPayload.apiConfig and jsonPayload.apiMethod fields of any matching entry. They identify the API config and the method whose quota configuration is invalid.

Why an API config can have an invalid quota configuration

You define quota metrics and limits in an API config, but API Gateway applies them to the entire API. Each time you create an API config, the metrics and limits it declares replace the ones declared by the API's previous API configs. Only the values from the most recently created API config are enforced.

In contrast, the metrics consumed by each method are defined in the API config that the gateway serves. If a gateway runs an older API config, it asks Service Control to allocate quota against a metric that exists in its own config, but might not exist in the API. If the metric doesn't exist, the allocation call fails, and the gateway rejects the request.

For example, the following sequence leaves the first gateway broken:

  1. You create API config config-v1, which declares the metric quota-metric-v1, and deploy it to gateway-1.
  2. You create API config config-v2 for the same API, which declares the metric quota-metric-v2, and deploy it to gateway-2.

gateway-2 works, but requests to the quota-enforced methods of gateway-1 start failing, because quota-metric-v1 is no longer defined for the API.

The following changes can cause errors for any gateway that's still deployed with an earlier API config:

  • Renaming or removing a metric.
  • Changing which metric a quota limit applies to.
  • Changing the metric named in the per-method quota costs (x-google-quota for OpenAPI documents, or quota.metric_rules for gRPC service configs).

Changing only the value of a limit doesn't cause errors. However, because limits are also applied at the API level, the new value is enforced on every gateway of that API, including gateways deployed with an earlier API config.

Compare the deployed quota configurations

  1. List your gateways and the API config that each one serves:

    gcloud api-gateway gateways list \
 --format="table(name.basename(),apiConfig)"

  2. List the API configs of the affected API, most recently created first:

    gcloud api-gateway api-configs list --api=API_ID \
 --format="table(name.basename(),createTime:sort=1:reverse)"

    The first entry is the API config whose quota metrics and limits are enforced for the whole API. Sort with the --format flag as shown: this command doesn't support the --sort-by flag, and it doesn't return API configs in a predictable order.

  3. Display the API definition that an API config was created from:

    gcloud api-gateway api-configs describe CONFIG_ID --api=API_ID \
 --view=FULL --format="value(openapiDocuments[0].document.contents)" \
 | tr '_-' '/+' | base64 --decode

    The tr command is required because the contents field is base64url encoded, which base64 --decode can't read directly.

    For a gRPC API, the quota configuration is in the service config rather than in an OpenAPI document, so replace openapiDocuments[0].document.contents with managedServiceConfigs[0].contents.

  4. Run the command in step 3 for the API config at the top of the list from step 2, and then for each of the other API configs that step 1 shows as still deployed to a gateway.

  5. Compare the results. Every metric that an older API config charges its methods against must also be defined in the most recently created API config. If a metric is missing from that config, the gateways that serve the older API config fail.

Restore a valid quota configuration

Audit your quota metrics and limits to ensure that they are consistent across all active configurations. To do this, take one of the following actions:

  • Update every gateway of the API to use the most recently created API config, as described in Update a gateway.
  • Create a new API config that declares every metric used by the API configs that are still deployed, and keep the existing gateways on their current API configs.

To avoid allocation errors, keep metric names consistent across the API configs of an API. When you change a quota, change the value of the limit rather than the name of the metric.

High latency API requests

Like Cloud Run and Cloud Run functions, API Gateway is subject to "cold start" latency. If your gateway has not received traffic for 15 to 20 minutes, requests made to your gateway within the first 10 to 15 seconds of the cold start will experience 3 to 5 seconds of latency.

If the issue persists after the initial "warm up" period, check the request logs of the backend service(s) you configured in your API Config. For example, if the backend service is implemented using Cloud Run functions, check the Cloud Logging entries of the associated Cloud Function request log.

Cannot view log information

If your API is responding correctly, but the logs contain no data, it typically means that you have not enabled all of the Google services required by API Gateway.

API Gateway requires that you enable the following Google Cloud services:

Name Service name
API Gateway API apigateway.googleapis.com
Service Management API servicemanagement.googleapis.com
Service Control API servicecontrol.googleapis.com

To enable required services:

Google Cloud console

  1. In the Google Cloud console, go to the APIs & Services > API Library page.

    Go to API Library

  2. On the API Library page, enter the required API name into the search bar.
  3. In the search results, select the API page.
  4. On the API page, click Enable.
  5. Repeat these steps for each of the services listed in the preceding table.

Google Cloud CLI

Use the following commands to enable the services:

gcloud services enable apigateway.googleapis.com
gcloud services enable servicemanagement.googleapis.com
gcloud services enable servicecontrol.googleapis.com

For more information about the gcloud services, see gcloud services.