Run colocated Python with Pathways

To avoid routing host-bound work such as data loading and checkpointing through a single Pathways controller on Google Kubernetes Engine (GKE), run those functions on your Cloud TPU worker VMs by using the JAX colocated Python API.

This document, which is intended for machine learning engineers, describes how colocated Python works with Pathways. You learn how to build, deploy, and verify a sidecar container with a JobSet manifest or Cluster Toolkit.

Before you begin

Complete the following prerequisites:

Required roles

Grant the following Identity and Access Management (IAM) roles:

  • Your user account:
    • Artifact Registry Writer (roles/artifactregistry.writer) on your Google Cloud project to push custom sidecar images to Artifact Registry.
    • Kubernetes Engine Developer (roles/container.developer) on your Google Cloud project to deploy JobSet resources, forward ports, and read Pod logs on GKE.
  • GKE node pool service account: Pathways Pods use this service account when your node pools use the --workload-metadata=GCE_METADATA flag, as configured in Create a GKE cluster with Pathways:
    • Artifact Registry Reader (roles/artifactregistry.reader) on the Google Cloud project that hosts your private repository to pull custom sidecar images.
    • Storage Object Admin (roles/storage.objectAdmin) on your Cloud Storage scratch bucket for the compilation cache and other temporary files.

Optional: To let the Pathways resource manager (pathways-rm) container add a bucket lifecycle rule that expires old compilation-cache entries, also grant the Storage Admin role (roles/storage.admin) on the scratch bucket. Without this role, the pathways-rm container logs a warning and continues using the cache. To skip the lifecycle update, pass the --enable_gcs_olm=false flag to the pathways-rm container.

If your node pools use Workload Identity Federation for GKE instead, grant the scratch bucket role to the Kubernetes service account that the Pathways Pods use, or to its bound IAM service account. By default, Pathways Pods use the default service account unless your JobSet manifest sets the serviceAccountName field. Keep the Artifact Registry Reader role on the node pool service account.

How colocated Python works with Pathways

In Pathways, your JAX controller script runs on a single CPU host while Cloud TPU worker Pods run the C++ pathways-worker container. To run @colocated_python functions and @colocated_python_class methods on worker hosts, colocated Python uses the following components:

  • Your JAX controller: Uses the jax.experimental.colocated_python module to pair each Cloud TPU chip with a host CPU device and serialize your Python function and type metadata with the cloudpickle library.
  • The pathways-proxy container (in the pathways-head Pod): Receives the serialized function over gRPC and routes it to the pathways-worker container on the worker VMs that own the call's input CPU devices. Passing the --sidecar_name=external flag configures the pathways-proxy container to target the colocated-python-sidecar container.
  • The pathways-worker container (on each worker Pod): Loads the serialized function into the local sidecar container, triggers execution over local gRPC, and exchanges array buffers through a shared in-memory emptyDir volume (medium: Memory).
  • The colocated-python-sidecar container (on each worker Pod): Unpacks and runs the function in an isolated Python 3.12 runtime once per VM across that VM's local shards, writing output arrays directly into shared host memory.

To learn more about the Pathways architecture, see Pathways components.

Execution flow

When you invoke a decorated function, Pathways runs it across your cluster in two stages:

  1. Function routing: The jax.experimental.colocated_python module packs the function into an Interim Framework Runtime (IFRT) program and sends it to the pathways-proxy container. The proxy forwards the program only to worker VMs holding the input shards, running once per VM across all local shards.
  2. Shared-memory execution: The pathways-worker container writes input shards to the in-memory shared volume (/tmp/sidecar or /tmp/shared-memory) and invokes the colocated-python-sidecar container over loopback TCP (localhost:50051) or a Unix domain socket (/sock/sidecar.sock). Output arrays return through shared memory without gRPC copies and move directly into Cloud TPU high-bandwidth memory (HBM) when placed on Cloud TPU devices.

For JAX API usage such as discovering colocated CPU devices and declaring output specifications for asynchronous dispatch, see the JAX colocated Python guide.

Design guidelines

When you write colocated Python functions, follow these guidelines:

  • Minimize captured variables: Pass array data as sharded jax.Array arguments rather than capturing large arrays or non-serializable objects (such as open files or network clients) in closures. Open files or clients inside a @colocated_python_class instance on the worker.
  • Recreate @colocated_python_class state after restarts: Stateful sidecar objects persist across steps in memory, but aren't saved automatically if a worker Pod restarts or slice counts change during elastic training. Recreate stateful objects from a checkpoint after rescaling or restarts.

Prebuilt images and tags

Google Cloud publishes prebuilt Pathways images in a public Artifact Registry repository:

  • Colocated Python sidecar: us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways/colocated_python_sidecar:TAG
  • Pathways resource manager and worker: us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways/server:TAG
  • IFRT proxy server: us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways/proxy_server:TAG

Use matching tags across the server, proxy_server, and colocated_python_sidecar images.

Prebuilt sidecar images support JAX versions 0.10.1, 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0, and 0.12.0.1. Replace TAG with one of the following tag formats:

  • jax-JAX_VERSION (recommended): a rolling tag for a specific JAX release, such as jax-0.12.0.1.
  • YYYYMMDD-jax_JAX_VERSION: a date-stamped build tag, such as 20261110-jax_0.12.0.1.

If you need an exact image version that never changes, pin the image by its SHA-256 digest. To look up an image digest, use the gcloud artifacts docker images describe command:

IMAGE_REPO="us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways"
gcloud artifacts docker images describe \
    "${IMAGE_REPO}/colocated_python_sidecar:jax-JAX_VERSION" \
    --format="value(image_summary.digest)"

Replace JAX_VERSION with your JAX version, such as 0.12.0.1.

Build a custom sidecar image

The prebuilt sidecar image includes the jax, jaxlib, numpy, scipy, ml-dtypes, opt-einsum, cloudpickle, and absl-py packages. If your colocated Python functions use only these packages, skip to Deploy a Pathways JobSet with the sidecar or Deploy with Cluster Toolkit.

To add custom Python dependencies, layer them onto the prebuilt image matching your JAX version:

  1. In an empty build directory, create a requirements.txt file for your extra packages. Omit the jax and jaxlib packages, which are already installed and pinned in the /opt/venv/server_constraints.txt file:

    grain
    
  2. Create a Dockerfile that installs the packages in the requirements.txt file, constrained by the /opt/venv/server_constraints.txt file:

    ARG JAX_VERSION=0.12.0.1
    ARG IMAGE_REPO=us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways
    FROM ${IMAGE_REPO}/colocated_python_sidecar:jax-${JAX_VERSION}
    
    WORKDIR /app
    COPY requirements.txt .
    
    RUN uv pip install -r requirements.txt \
            -c /opt/venv/server_constraints.txt && \
        uv cache clean
    

    Keep the base image's ENTRYPOINT and CMD instructions unchanged. If the requirements.txt file installs packages from Git URLs, add the following RUN instruction before the uv pip install RUN instruction, because the slim base image omits the git tool:

    RUN apt-get update && \
        apt-get install -y --no-install-recommends git && \
        rm -rf /var/lib/apt/lists/*
    
  3. Authenticate Docker to your Artifact Registry region by using the gcloud auth configure-docker command:

    gcloud auth configure-docker REGISTRY_LOCATION-docker.pkg.dev
    

    Replace REGISTRY_LOCATION with your Artifact Registry location, such as us or us-central1.

  4. Build and push your custom sidecar image:

    export JAX_VERSION=JAX_VERSION
    IMAGE_PREFIX=REGISTRY_LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY
    export SIDECAR_IMAGE="${IMAGE_PREFIX}/colocated-sidecar:jax-${JAX_VERSION}"
    
    docker build --build-arg JAX_VERSION="${JAX_VERSION}" \
        --tag "${SIDECAR_IMAGE}" .
    docker push "${SIDECAR_IMAGE}"
    

    Replace the following:

    • JAX_VERSION: your JAX version, such as 0.12.0.1.
    • REGISTRY_LOCATION: your Artifact Registry location, such as us or us-central1.
    • PROJECT_ID: your Google Cloud project ID.
    • REPOSITORY: your Artifact Registry Docker repository name.

Deploy a Pathways JobSet with the sidecar

You can deploy Pathways with the sidecar either by applying a JobSet manifest with the kubectl command-line tool in this section, or by submitting the job directly with the gcluster CLI in Deploy with Cluster Toolkit.

To generate and apply a JobSet manifest, complete the following steps:

  1. Generate a JobSet manifest (pathways-colocated-jobset.yaml) with either of the following tools:

    • The pathways-utils Python builder: Configure a JobSet with the PathwaysJobSet builder by calling the .add_colocated_python("SIDECAR_IMAGE") and .export_yaml("pathways-colocated-jobset.yaml") methods. If you use the pathwaysutils package (version 0.1.14 or earlier from PyPI), also pass the proxy_container_kwargs={"args": ["--sidecar_name=external"]} argument to the PathwaysJobSet constructor, or install the pathways-utils package from the main branch.
    • Cluster Toolkit (gcluster): Run the gcluster job submit command with the --pathways-colocated-python-sidecar-image flag and append the --dry-run-out=pathways-colocated-jobset.yaml flag to render the manifest locally without submitting the job.

    Replace SIDECAR_IMAGE with your custom sidecar image URI or the prebuilt colocated_python_sidecar image URI.

    Both generators configure the manifest to do the following:

    • Run the colocated-python-sidecar container under the initContainers field with restartPolicy: Always in each worker Pod. The gcluster CLI names the worker replicated job worker, whereas the PathwaysJobSet builder names it pathways-worker.
    • Pass the --sidecar_name=external flag to the pathways-proxy container, and connect the pathways-worker container to the sidecar over loopback TCP on localhost:50051. When the pathways-proxy container runs under the initContainers field, it also uses restartPolicy: Always.
    • Mount an in-memory emptyDir volume with medium: Memory in both worker Pod containers: at /tmp/sidecar in gcluster manifests, or at /tmp/shared-memory in PathwaysJobSet manifests. Both generators also set the CLOUD_PATHWAYS_SIDECAR_SHM_DIRECTORY environment variable on the sidecar and the --cloud_pathways_sidecar_shm_directory flag on the pathways-worker container.
  2. Apply the JobSet manifest and wait for the Pods to reach the Running status:

    kubectl apply -f pathways-colocated-jobset.yaml
    kubectl get pods -l jobset.sigs.k8s.io/jobset-name=JOBSET_NAME
    

    Replace JOBSET_NAME with the metadata.name value of your JobSet.

Run one JAX client session per JobSet deployment. After a client disconnects, a restarted pathways-worker container can't reuse a running sidecar. Delete and recreate the JobSet before starting another session.

Files in an in-memory emptyDir volume count toward Pod memory usage. Set the emptyDir.sizeLimit field (such as 175Gi) on the shared-memory volume to cap usage per node. By default, both containers retain freed shared memory for 30 seconds of idle time for reuse. To change that window, set both the --pathways_sidecar_shared_memory_idle_release_secs=SECONDS flag in the args field of the colocated-python-sidecar container (alongside the --port=50051 or --unix_domain_socket_path flag) and the --cloud_pathways_shared_memory_idle_release_secs=SECONDS flag in the args field of the pathways-worker container. Replace SECONDS with the idle timeout in seconds. Setting both flags to 0 releases freed pages within about 1 second, which reduces memory usage at the cost of extra allocation overhead.

Optional: Use a Unix domain socket

To use a Unix domain socket (UDS) at /sock/sidecar.sock instead of opening a TCP port on worker Pods that set hostNetwork: true, apply the following overrides to your JobSet manifest before running the kubectl apply command:

  • Mount an emptyDir: {} volume named sidecar-socket at /sock in both the pathways-worker and colocated-python-sidecar containers.
  • Set args: ["--unix_domain_socket_path=/sock/sidecar.sock"] and an exec startupProbe (test -S /sock/sidecar.sock) on the colocated-python-sidecar container.
  • Append the --cloud_pathways_external_sidecar_unix_domain_socket_path=/sock/sidecar.sock flag to the args field of the pathways-proxy container.

Deploy with Cluster Toolkit

In Cluster Toolkit (gcluster version 1.108.0 or later), passing the --pathways-colocated-python-sidecar-image flag to the gcluster job submit command configures the colocated-python-sidecar container over loopback TCP (localhost:50051), mounts the in-memory volume at /tmp/sidecar, and sets the --sidecar_name=external flag. Pass the --pathways-server-image and --pathways-proxy-server-image flags explicitly so all three images use matching cloud-pathways tags.

Submit a headless Pathways deployment with your sidecar image so you can run interactive client scripts against it:

IMAGE_REPO="us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways"

gcluster job submit \
    --pathways \
    --pathways-headless \
    --pathways-gcs-location="gs://BUCKET_NAME/pathways-artifacts" \
    --pathways-server-image="${IMAGE_REPO}/server:jax-JAX_VERSION" \
    --pathways-proxy-server-image="${IMAGE_REPO}/proxy_server:jax-JAX_VERSION" \
    --pathways-colocated-python-sidecar-image="SIDECAR_IMAGE" \
    --name=WORKLOAD \
    --cluster=CLUSTER \
    --project=PROJECT_ID \
    --location=LOCATION \
    --num-slices=WORKLOAD_NODEPOOL_COUNT \
    --compute-type=COMPUTE_TYPE \
    --topology=TOPOLOGY

Replace the following:

  • BUCKET_NAME: your Cloud Storage scratch bucket.
  • JAX_VERSION: your JAX version, such as 0.12.0.1.
  • SIDECAR_IMAGE: your custom or prebuilt sidecar image URI.
  • WORKLOAD: a unique JobSet name (22 characters or fewer).
  • CLUSTER: your GKE cluster name.
  • PROJECT_ID: your Google Cloud project ID.
  • LOCATION: your GKE cluster region or zone.
  • WORKLOAD_NODEPOOL_COUNT: the number of Cloud TPU slices, such as 1.
  • COMPUTE_TYPE: the Cloud TPU machine type, such as ct5lp-hightpu-4t or ct6e-standard-4t.
  • TOPOLOGY: the Cloud TPU slice topology, such as 4x8 or 2x4.

You can also customize how the gcluster job submit command runs or exports the job:

  • Batch workloads: To run a containerized batch job inside the pathways-head Pod instead of a headless deployment, replace the --pathways-headless flag with the --image="CLIENT_IMAGE" and --command="COMMAND" flags. Use a Python 3.12 JAX client image built with server_constraints.txt pins and pass your entry command (such as python colocated_example.py).
  • Local manifest rendering: Append the --dry-run-out=pathways-colocated-jobset.yaml flag to render the JobSet manifest locally without submitting it.
  • Subsequent headless sessions: Run the gcluster job cancel command and resubmit the job before starting another client session.

Run a colocated Python workload

Run your JAX client script with the JAX_PLATFORMS=proxy environment variable against the pathways-proxy container, either in the pathways-head workload container or over port forwarding:

  1. To align serialization dependencies with your sidecar image, extract the /opt/venv/server_constraints.txt file and install matching packages in a Python 3.12 virtual environment:

    IMAGE_REPO="us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways"
    docker run --rm --entrypoint cat \
        "${IMAGE_REPO}/colocated_python_sidecar:jax-JAX_VERSION" \
        /opt/venv/server_constraints.txt > constraints.txt
    python3.12 -m venv .venv
    source .venv/bin/activate
    python -m pip install -c constraints.txt \
        jax jaxlib cloudpickle pathwaysutils
    

    Replace JAX_VERSION with your JAX version, such as 0.12.0.1.

  2. If you run the client outside the pathways-head Pod, forward local port 29000 to the pathways-proxy container in a separate terminal:

    JOBSET_LABEL="jobset.sigs.k8s.io/jobset-name=JOBSET_NAME"
    ROLE_LABEL="jobset.sigs.k8s.io/replicatedjob-name=pathways-head"
    HEAD_POD=$(kubectl get pods -l "${JOBSET_LABEL},${ROLE_LABEL}" \
        -o jsonpath='{.items[0].metadata.name}')
    kubectl port-forward "${HEAD_POD}" 29000:29000
    

    Replace JOBSET_NAME with the metadata.name value of your JobSet (or the WORKLOAD name that you passed to the gcluster job submit command).

  3. Save the following script as colocated_example.py. This script allocates a 4 MiB NumPy array in shared host memory on each Cloud TPU worker's colocated CPU device and transfers it to Cloud TPU HBM:

    import jax
    from jax.experimental import colocated_python
    import numpy as np
    import pathwaysutils
    
    pathwaysutils.initialize()
    
    @colocated_python.colocated_python
    def make_host_array(seed_arr: jax.Array) -> jax.Array:
      host_array = np.full(
          (1024, 1024), fill_value=float(seed_arr), dtype=np.float32
      )
      return jax.device_put(host_array, seed_arr.device)
    
    tpu_devices = jax.devices()
    cpu_devices = colocated_python.colocated_cpu_devices(tpu_devices)
    print(
        f"TPU devices: {len(tpu_devices)}, "
        f"colocated CPU devices: {len(cpu_devices)}"
    )
    
    host_arrays = [
        make_host_array(jax.device_put(np.int32(i + 1), cpu_dev))
        for i, cpu_dev in enumerate(cpu_devices)
    ]
    tpu_arrays = [
        jax.device_put(host_arr, tpu_dev)
        for host_arr, tpu_dev in zip(host_arrays, tpu_devices)
    ]
    for arr in tpu_arrays:
      arr.block_until_ready()
    
    print(
        f"Transferred {len(tpu_arrays)} arrays of shape {tpu_arrays[0].shape} "
        f"to TPU HBM; sample sum={float(tpu_arrays[0].sum()):.1f}"
    )
    
  4. Run the script against the IFRT proxy server:

    JAX_PLATFORMS=proxy JAX_BACKEND_TARGET=grpc://127.0.0.1:29000 \
        python colocated_example.py
    

    The output is similar to the following:

    TPU devices: 8, colocated CPU devices: 8
    Transferred 8 arrays of shape (1024, 1024) to TPU HBM; sample sum=1048576.0
    

    In multislice production workloads, pass a sharded jax.Array argument placed on the devices returned by the colocated_python.colocated_cpu_devices(mesh) call and set the out_specs_fn parameter with the .specialize(out_specs_fn=...) method so Pathways runs one parallel call across all worker sidecars without waiting.

    Don't run client scripts inside a worker Pod with the JAX_PLATFORMS=cpu environment variable, because that setting runs JAX locally on one CPU and bypasses Pathways. You can ignore any IFRT proxy server disconnected: CANCELLED message printed on exit.

For additional usage patterns, see the following resources:

Verify the sidecar deployment

Run the kubectl logs command on a headless JobSet before canceling or deleting it, or while a batch workload is still running. In batch JobSet deployments, the JobSet controller deletes worker Pods as soon as the pathways-head Job completes. View completed batch logs in Cloud Logging instead, as described in Troubleshoot Pathways on Cloud.

To verify a running deployment with the kubectl command-line tool, complete the following steps:

  1. Inspect the colocated-python-sidecar container logs:

    JOBSET_LABEL="jobset.sigs.k8s.io/jobset-name=JOBSET_NAME"
    ROLE_LABEL="jobset.sigs.k8s.io/replicatedjob-name=worker"
    WORKER_POD=$(kubectl get pods -l "${JOBSET_LABEL},${ROLE_LABEL}" \
        -o jsonpath='{.items[0].metadata.name}')
    kubectl logs "${WORKER_POD}" -c colocated-python-sidecar
    

    Replace JOBSET_NAME with the metadata.name value of your JobSet (or the WORKLOAD name that you passed to the gcluster job submit command). If you generated your manifest with the PathwaysJobSet builder, set replicatedjob-name=pathways-worker in the ROLE_LABEL variable.

    The output is similar to the following:

    Started SidecarGrpcServer on localhost:50051
    ...
    SidecarServer agreed on protocol version: 6
    Installed shared memory allocator for colocated Python using file /tmp/sidecar/pathways_shm_sidecar
    

    Confirm that the output shows the gRPC startup message and, after the first @colocated_python call, the shared memory allocator message. Manifests generated with the PathwaysJobSet builder use the /tmp/shared-memory directory, and UDS deployments log unix:/sock/sidecar.sock instead of localhost:50051.

  2. Inspect the pathways-worker container logs on the same Pod:

    kubectl logs "${WORKER_POD}" -c pathways-worker \
        | grep -iE "shared memory|external sidecar"
    

    The output is similar to the following:

    Enabled shared memory for external sidecar in /tmp/sidecar
    External sidecar connected to backing GRPC sidecar at localhost:50051
    

    Confirm that the pathways-worker container enabled shared memory and connected to the sidecar.

When you run the kubectl exec command to inspect the shared-memory directory, target the colocated-python-sidecar container with the -c colocated-python-sidecar flag, because the pathways-worker container doesn't include a shell.

Troubleshoot colocated Python

The following sections describe how to resolve common colocated Python issues.

Unable to unpickle error

If your workload fails with the following error, Python or package versions differ between your JAX client and the sidecar container:

jax.errors.JaxRuntimeError: UNKNOWN: Unable to unpickle the callable and specialization

Check that your client runs Python 3.12 (python --version) and install client packages with the -c constraints.txt flag extracted from the sidecar image. For more information, see Run a colocated Python workload.

Dependency conflict during a docker build run

If the uv pip install command fails with the No solution found when resolving dependencies error, a pin in your requirements.txt file conflicts with the /opt/venv/server_constraints.txt file. Remove the jax and jaxlib packages from the requirements.txt file and relax version pins on packages such as numpy, ml-dtypes, or scipy.

Sidecar not enabled error

If the pathways-proxy container exits with the Sidecar create args were set, but sidecar was not enabled error, add the --sidecar_name=external flag to the args field of the pathways-proxy container.

Sidecar already initialized error

If your workload fails with the FAILED_PRECONDITION: Sidecar already initialized error, an earlier client disconnect restarted the pathways-worker container while the colocated-python-sidecar container stayed initialized. Delete and recreate the JobSet before starting another client session.

Shared memory or sidecar connection errors

Use the following checks to resolve shared-memory mount, gRPC connection, or host memory errors:

  • Failed to stat sidecar_shm_directory: Mount the shared-memory emptyDir volume at the matching path (/tmp/sidecar or /tmp/shared-memory) in both the pathways-worker and colocated-python-sidecar containers.
  • Timed out connecting to sidecar gRPC service or Failed to connect to external sidecar: Confirm that the colocated-python-sidecar container runs under the initContainers field with restartPolicy: Always and that all three cloud-pathways images use matching tags. Check that your TCP (--port=50051) or UDS (/sock volume and --unix_domain_socket_path=/sock/sidecar.sock) settings match between the pathways-proxy and colocated-python-sidecar containers.
  • Worker host out-of-memory after large array bursts: Set the --pathways_sidecar_shared_memory_idle_release_secs=0 flag on the colocated-python-sidecar container (alongside the --port=50051 or --unix_domain_socket_path flag) and the --cloud_pathways_shared_memory_idle_release_secs=0 flag on the pathways-worker container. Setting both flags to 0 frees unused shared memory within about 1 second. Alternatively, reduce the peak amount of host array data that your workload keeps in memory at once.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this document, delete your deployment and temporary artifacts when you finish:

  • JobSet manifest: Delete the JobSet resource:

    kubectl delete jobset JOBSET_NAME
    

    Replace JOBSET_NAME with the metadata.name value of your JobSet.

  • Cluster Toolkit: Cancel the deployed job:

    gcluster job cancel WORKLOAD \
        --cluster=CLUSTER \
        --project=PROJECT_ID \
        --location=LOCATION
    

    Replace the following:

    • WORKLOAD: the name of your Pathways job.
    • CLUSTER: your GKE cluster name.
    • PROJECT_ID: your Google Cloud project ID.
    • LOCATION: your GKE cluster region or zone.

Optional: Delete any custom sidecar images in Artifact Registry and remove temporary objects in your Cloud Storage scratch bucket. When you no longer need your GKE cluster, delete it by following the Clean up steps in Create a GKE cluster with Pathways.

What's next