Troubleshoot Agent Gateway

Gated Preview for Agent Gateway on Cloud Run

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This feature is in a gated Preview and is available only to projects on an allowlist. To request early access, sign up here for the preview.

When an outbound request from your Cloud Run agent fails or bypasses Agent Gateway, trace the request path using the following checks or see Common issues and resolutions. For setup instructions, see Govern agents with Agent Gateway.

Before traffic reaches Agent Gateway, the Cloud Run Envoy sidecar fetches routing and certificate configuration from the Traffic Director API. The sidecar sends outbound traffic through an mTLS CONNECT tunnel to Secure Web Proxy.

Verify outbound routing in the Envoy sidecar access logs

When the Cloud Run Envoy sidecar intercepts an outbound TCP connection, it writes a JSON access log entry to run.googleapis.com%2Fvar%2Flog%2Fenvoy_sidecar_log with the jsonPayload.upstream_host field.

Trigger an outbound request from your Cloud Run service and query the sidecar access logs:

gcloud logging read \
    'resource.type="cloud_run_revision" AND resource.labels.service_name="SERVICE_NAME" AND logName=~"envoy_sidecar_log$" AND jsonPayload.upstream_host:*' \
    --project=PROJECT_ID \
    --freshness=7d \
    --limit=20 \
    --format="table(timestamp, jsonPayload.sni, jsonPayload.upstream_host, jsonPayload.response_flags, jsonPayload.bytes_sent, jsonPayload.bytes_received, jsonPayload.duration)"

Replace the following:

• SERVICE_NAME: the name of your Cloud Run agent service.
• PROJECT_ID: your Google Cloud project ID.

A healthy connection through the gateway returns output similar to the following:

TIMESTAMP                     SNI          UPSTREAM_HOST                      RESPONSE_FLAGS  BYTES_SENT  BYTES_RECEIVED  DURATION
2026-10-09T05:07:48.340693Z   example.com  envoy://tunnel_internal_listener/  -               4101        2899            489

Check the UPSTREAM_HOST and RESPONSE_FLAGS values:

• If UPSTREAM_HOST is envoy://tunnel_internal_listener/ and RESPONSE_FLAGS is -, the connection is healthy. The Envoy sidecar matched your EgressRouting policy and forwarded the connection to Agent Gateway.
• If UPSTREAM_HOST is a destination IP address and port such as 93.184.216.34:443, the Traffic Director API couldn't match your EgressRouting policy or resolve the target gateway. Envoy used ORIGINAL_DST passthrough and bypassed Agent Gateway. See Audit EgressRouting and AgentGateway resource URIs.
• If UPSTREAM_HOST is envoy://tunnel_internal_listener/ and RESPONSE_FLAGS is UF or UH, or BYTES_RECEIVED is 0, Envoy couldn't open the CONNECT tunnel to Secure Web Proxy. Check your Virtual Private Cloud (VPC) subnet connectivity or see Updating the Secure Web Proxy gateway breaks Envoy tunneling.
• If no rows appear, check run.googleapis.com%2Frequests to confirm that the request reached your Cloud Run container. Verify that your active revision enables --ambient-networking. If rows still don't appear, see Check Envoy sidecar xDS and connection logs.

Check Envoy sidecar xDS and connection logs

If the Envoy sidecar access logs return no entries or show a connection failure, inspect the sidecar xDS and upstream connection logs:

1. Check for Traffic Director API xDS stream errors or startup status:

   gcloud logging read \
      'resource.type="cloud_run_revision" AND resource.labels.service_name="SERVICE_NAME" AND logName=~"envoy_sidecar_log$" AND (severity>=INFO OR textPayload:"DeltaAggregatedResources")' \
      --project=PROJECT_ID \
      --freshness=7d \
      --limit=30 \
      --format="table(timestamp, severity, textPayload, jsonPayload.message)"

2. Check upstream tunnel connection attempts to ambient-swp-connect-cluster:

   gcloud logging read \
      'resource.type="cloud_run_revision" AND resource.labels.service_name="SERVICE_NAME" AND logName=~"envoy_sidecar_log$" AND (textPayload:"ambient-swp-connect-cluster" OR jsonPayload.message:"ambient-swp-connect-cluster" OR jsonPayload.message:"tunnel_internal_listener")' \
      --project=PROJECT_ID \
      --freshness=7d \
      --limit=30 \
      --format="table(timestamp, severity, textPayload, jsonPayload.message)"

   Replace the following:

   • SERVICE_NAME: the name of your Cloud Run agent service.
   • PROJECT_ID: your Google Cloud project ID.

Check the log messages:

• If the logs contain DeltaAggregatedResources gRPC config stream and The caller does not have permission, the service identity is missing roles/trafficdirector.client. See Envoy sidecar can't receive xDS configuration.
• If the logs contain Creating connection to cluster ambient-swp-connect-cluster, Envoy received the EgressRouting configuration and tried to connect to Secure Web Proxy.

Audit EgressRouting and AgentGateway resource URIs

When you run gcloud alpha network-services egress-routings create or gcloud beta network-services agent-gateways import, the API validates URI syntax without checking whether the target resources exist. A typo in a resource name or region causes the Traffic Director API to skip EgressRouting translation, and Envoy uses passthrough routing.

Inspect the stored URIs and compare them against your deployed resources:

1. Inspect the EgressRouting resource. The output normalizes project IDs to project numbers:

   gcloud alpha network-services egress-routings describe EGRESS_ROUTING_NAME \
      --location=REGION \
      --project=PROJECT_ID

2. List all EgressRouting resources in the region to confirm that no duplicate policy targets the same service:

   gcloud alpha network-services egress-routings list \
      --location=REGION \
      --project=PROJECT_ID

3. Verify that the target Secure Web Proxy gateway matches the URI in EgressRouting.gateway and AgentGateway.selfManaged.resourceUris:

   gcloud network-services gateways describe SWP_NAME \
      --location=REGION \
      --project=PROJECT_ID

   Replace the following:

   • EGRESS_ROUTING_NAME: the name of your EgressRouting resource.
   • REGION: the region of your deployment.
   • PROJECT_ID: your Google Cloud project ID.
   • SWP_NAME: the name of your Secure Web Proxy gateway.

Verify that all three URIs match your project ID or number, region, and resource name:

• EgressRouting.sources[].workloadUri:
  • Service-scoped: //run.googleapis.com/projects/PROJECT_ID/locations/REGION/services/SERVICE_NAME
  • Project-scoped: //run.googleapis.com/projects/PROJECT_ID or //run.googleapis.com/projects/PROJECT_NUMBER
• EgressRouting.gateway: //networkservices.googleapis.com/projects/PROJECT_ID/locations/REGION/gateways/SWP_NAME
• AgentGateway.selfManaged.resourceUris[]: projects/PROJECT_ID/locations/REGION/gateways/SWP_NAME

Check Secure Web Proxy transaction logs

If the Envoy sidecar forwards traffic to Secure Web Proxy but requests fail, inspect your proxy logs as described in View proxy transaction logs:

• If no traffic appears in the proxy logs, requests aren't reaching Secure Web Proxy. Check your EgressRouting configuration in Audit EgressRouting and AgentGateway resource URIs and Common issues and resolutions.
• If you see unexpected DENY or ALLOW decisions, check jsonPayload.authzPolicyInfo and jsonPayload.agentGatewayInfo in the log entry to inspect the matched AuthzPolicy and destination metadata, and verify any ExtensionBinding rules attached to your Agent Gateway resource. After you create or update an ExtensionBinding resource, configuration changes can take up to five minutes to propagate to the gateway.

Common issues and resolutions

The following sections describe common issues and how to resolve them.

Outbound traffic bypasses Agent Gateway due to a mismatched resource URI

Outbound traffic from Cloud Run bypasses Agent Gateway or times out after egress-routings create succeeds. In envoy_sidecar_log, jsonPayload.upstream_host shows the destination IP address and port instead of envoy://tunnel_internal_listener/.

One of the EgressRouting or AgentGateway resource URIs references a nonexistent resource name or mismatched region. When the Traffic Director API can't resolve the chain from the Cloud Run service to Secure Web Proxy, it skips tunnel translation and serves ORIGINAL_DST.

Run the commands in Audit EgressRouting and AgentGateway resource URIs to find the mismatched URI. Update the configuration with the matching resource name and region, and recreate the AgentGateway or EgressRouting resource.

Envoy sidecar can't receive xDS configuration

Outbound requests from the Cloud Run container time out or fail, and envoy_sidecar_log contains the following error:

DeltaAggregatedResources gRPC config stream to trafficdirector.REGION.rep.googleapis.com:443 closed: 7, The caller does not have permission

The Cloud Run runtime service identity doesn't have roles/trafficdirector.client. By default, deploying a service with --functional-type=agent and --identity-type=agent-identity grants roles/run.agentDefaultAccess, which includes roles/trafficdirector.client and roles/cloudtrace.agent. If your service identity is missing these permissions, grant the roles:

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="principal://agents.global.org-ORGANIZATION_ID.system.id.goog/resources/run/projects/PROJECT_NUMBER/locations/REGION/services/SERVICE_NAME" \
    --role="roles/trafficdirector.client"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="principal://agents.global.org-ORGANIZATION_ID.system.id.goog/resources/run/projects/PROJECT_NUMBER/locations/REGION/services/SERVICE_NAME" \
    --role="roles/cloudtrace.agent"

Replace the following:

• PROJECT_ID: your Google Cloud project ID.
• ORGANIZATION_ID: your Google Cloud organization ID.
• PROJECT_NUMBER: your Google Cloud project number.
• REGION: the region of your service.
• SERVICE_NAME: the name of your Cloud Run agent service. If your service uses a service account instead of Agent Identity, set --member to serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID..

Confirm that the Cloud Run service agent can access your VPC network. By default, roles/run.serviceAgent includes these permissions. If you modified the service agent permissions, grant roles/compute.networkUser:

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:service-PROJECT_NUMBER@serverless-robot-prod." \
    --role="roles/compute.networkUser"

Updating the Secure Web Proxy gateway breaks Envoy tunneling

After you update an existing Secure Web Proxy gateway's ServerTlsPolicy, certificate, IP address, or port, outbound requests fail with 502, 503, or 504 errors, or envoy_sidecar_log reports RESPONSE_FLAGS: UF.

You can't change addresses or ports after you create a Secure Web Proxy gateway. In addition, updates to serverTlsPolicy or certificateUrls on an existing Secure Web Proxy gateway don't propagate to EgressRouting endpoints in the Traffic Director API.

To change these Secure Web Proxy fields without workload downtime, complete the following steps:

1. Create a replacement Secure Web Proxy gateway with a different resource name and the updated configuration.
2. Update selfManaged.resourceUris in the AgentGateway resource to reference the replacement Secure Web Proxy gateway URI.
3. Recreate your EgressRouting resource to point to the replacement Secure Web Proxy gateway.
4. After the replacement gateway routes traffic, remove the old Secure Web Proxy gateway from the AgentGateway resource and delete the old Secure Web Proxy gateway.

Outbound requests fail with 403 errors

Outbound requests from your Cloud Run agent, including calls to model endpoints such as Vertex AI at REGION-aiplatform.googleapis.com or aiplatform.googleapis.com, fail with 403 Forbidden errors after you configure policies on Agent Gateway.

Because --vpc-egress=all routes all outbound container traffic through Secure Web Proxy, requests fail for one of the following reasons:

• Your Identity-Aware Proxy (IAP) ExtensionBinding omits matchConditions (or matches hosts outside Model Context Protocol (MCP) servers) without an allow rule for the destination in your Identity and Access Management (IAM) Unified Access Policy.
• An ALLOW AuthzPolicy is attached to your Agent Gateway resource (or to the underlying Secure Web Proxy Gateway), and the outbound requests don't match its ALLOW rules.

To resolve this issue:

• Scope matchConditions in your IAP ExtensionBinding to prefix: "urn:mcp" for MCP servers (and optionally prefix: "urn:agent" for agent-to-agent traffic) so model calls and other non-MCP destinations bypass IAP evaluation, or add a destination.unregistered.host allow rule for your destination (such as REGION-aiplatform.googleapis.com or aiplatform.googleapis.com) in your IAM Unified Access Policy. For instructions, see Configure IAP egress policies.
• If you attached an ALLOW AuthzPolicy, add ALLOW rules for your required destinations, or remove the AuthzPolicy if you govern traffic only with ExtensionBinding resources.

Secure Web Proxy ignores GatewaySecurityPolicyRules

You created GatewaySecurityPolicyRule resources on the Secure Web Proxy Gateway, but they have no effect on outbound agent traffic.

When Secure Web Proxy is associated with an AgentGateway resource, Secure Web Proxy ignores all GatewaySecurityPolicyRule resources on the underlying Gateway (its TlsInspectionPolicy remains active).

Attach ExtensionBinding resources to your Agent Gateway resource instead. If you use AuthzPolicy resources, attach them only to your Agent Gateway resource; attaching AuthzPolicy resources to both the AgentGateway and the underlying Secure Web Proxy Gateway causes a configuration conflict. For instructions, see Configure policies on Agent Gateway.

Outbound requests fail TLS inspection

Outbound requests from your Cloud Run agent fail with TLS errors on every connection.

The Secure Web Proxy service account is missing roles/privateca.certificateManager on your Certificate Authority Service pool.

Grant roles/privateca.certificateManager on the CA pool to your Secure Web Proxy service account. For instructions, see Create a service account.

Outbound TLS-inspected connections fail with SSLV3_ALERT_HANDSHAKE_FAILURE or RemoteDisconnected

Outbound requests from your agent fail on the first connection to a destination host with SSLV3_ALERT_HANDSHAKE_FAILURE, or fail with RemoteDisconnected.

On the first TLS-inspected connection to a destination host, Secure Web Proxy mints and caches a leaf certificate from CA Service, which can cause a transient handshake error. HTTP clients that send Connection: close by default, such as urllib.request.urlopen, can fail with RemoteDisconnected during TLS interception.

To prevent these errors in your agent code:

• Use HTTP clients that reuse persistent connections, such as httpx.AsyncClient or requests.Session().
• Configure connection and handshake retries on your HTTP client, such as httpx.AsyncHTTPTransport(retries=2) or urllib3.util.Retry.

What's next

• Govern agents with Agent Gateway
• Configure ambient networking for Cloud Run
• Troubleshoot Cloud Run issues