To avoid routing host-bound work such as data loading and checkpointing through a single Pathways controller on Google Kubernetes Engine (GKE), run those functions on your Cloud TPU worker VMs by using the JAX colocated Python API.
This document, which is intended for machine learning engineers, describes how
colocated Python works with Pathways. You learn how to build, deploy, and verify
a sidecar container with a JobSet manifest or Cluster Toolkit.
Before you begin
Complete the following prerequisites:
- Create a GKE cluster with Pathways.
- Install the
kubectlcommand-line tool and thegke-gcloud-auth-pluginplugin. - Install the Google Cloud CLI.
- Install Docker.
- Enable the Kubernetes Engine API and Artifact Registry API.
- Create a Cloud Storage scratch bucket for Pathways temporary files.
- If you plan to build a custom sidecar image, create an Artifact Registry Docker repository.
- Install Python 3.12 in the client environment that runs your JAX workload.
- Optional: To deploy with
Cluster Toolkit,
install version 1.108.0 or later of the
gclusterCLI.
Required roles
Grant the following Identity and Access Management (IAM) roles:
- Your user account:
- Artifact Registry
Writer
(
roles/artifactregistry.writer) on your Google Cloud project to push custom sidecar images to Artifact Registry. - Kubernetes Engine
Developer
(
roles/container.developer) on your Google Cloud project to deployJobSetresources, forward ports, and read Pod logs on GKE.
- Artifact Registry
Writer
(
- GKE node pool service account: Pathways Pods use this
service account when your node pools use the
--workload-metadata=GCE_METADATAflag, as configured in Create a GKE cluster with Pathways:- Artifact Registry
Reader
(
roles/artifactregistry.reader) on the Google Cloud project that hosts your private repository to pull custom sidecar images. - Storage Object
Admin
(
roles/storage.objectAdmin) on your Cloud Storage scratch bucket for the compilation cache and other temporary files.
- Artifact Registry
Reader
(
Optional: To let the Pathways resource manager (pathways-rm) container add a
bucket lifecycle rule that expires old compilation-cache entries, also grant the
Storage Admin role
(roles/storage.admin) on the scratch bucket. Without this role, the
pathways-rm container logs a warning and continues using the cache. To skip
the lifecycle update, pass the --enable_gcs_olm=false flag to the
pathways-rm container.
If your node pools use Workload Identity Federation for GKE instead, grant the
scratch bucket role to the Kubernetes service account that the Pathways Pods
use, or to its bound IAM service account. By default, Pathways Pods use the
default service account unless your JobSet manifest sets the
serviceAccountName field. Keep the Artifact Registry Reader role on the node
pool service account.
How colocated Python works with Pathways
In Pathways, your JAX controller script runs on a single CPU host while
Cloud TPU worker Pods run the C++ pathways-worker container. To run
@colocated_python functions and @colocated_python_class methods on worker
hosts, colocated Python uses the following components:
- Your JAX controller: Uses the
jax.experimental.colocated_pythonmodule to pair each Cloud TPU chip with a host CPU device and serialize your Python function and type metadata with thecloudpicklelibrary. - The
pathways-proxycontainer (in thepathways-headPod): Receives the serialized function over gRPC and routes it to thepathways-workercontainer on the worker VMs that own the call's input CPU devices. Passing the--sidecar_name=externalflag configures thepathways-proxycontainer to target thecolocated-python-sidecarcontainer. - The
pathways-workercontainer (on each worker Pod): Loads the serialized function into the local sidecar container, triggers execution over local gRPC, and exchanges array buffers through a shared in-memoryemptyDirvolume (medium: Memory). - The
colocated-python-sidecarcontainer (on each worker Pod): Unpacks and runs the function in an isolated Python 3.12 runtime once per VM across that VM's local shards, writing output arrays directly into shared host memory.
To learn more about the Pathways architecture, see Pathways components.
Execution flow
When you invoke a decorated function, Pathways runs it across your cluster in two stages:
- Function routing: The
jax.experimental.colocated_pythonmodule packs the function into an Interim Framework Runtime (IFRT) program and sends it to thepathways-proxycontainer. The proxy forwards the program only to worker VMs holding the input shards, running once per VM across all local shards. - Shared-memory execution: The
pathways-workercontainer writes input shards to the in-memory shared volume (/tmp/sidecaror/tmp/shared-memory) and invokes thecolocated-python-sidecarcontainer over loopback TCP (localhost:50051) or a Unix domain socket (/sock/sidecar.sock). Output arrays return through shared memory without gRPC copies and move directly into Cloud TPU high-bandwidth memory (HBM) when placed on Cloud TPU devices.
For JAX API usage such as discovering colocated CPU devices and declaring output specifications for asynchronous dispatch, see the JAX colocated Python guide.
Design guidelines
When you write colocated Python functions, follow these guidelines:
- Minimize captured variables: Pass array data as sharded
jax.Arrayarguments rather than capturing large arrays or non-serializable objects (such as open files or network clients) in closures. Open files or clients inside a@colocated_python_classinstance on the worker. - Recreate
@colocated_python_classstate after restarts: Stateful sidecar objects persist across steps in memory, but aren't saved automatically if a worker Pod restarts or slice counts change during elastic training. Recreate stateful objects from a checkpoint after rescaling or restarts.
Prebuilt images and tags
Google Cloud publishes prebuilt Pathways images in a public Artifact Registry repository:
- Colocated Python sidecar:
us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways/colocated_python_sidecar:TAG - Pathways resource manager and worker:
us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways/server:TAG - IFRT proxy server:
us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways/proxy_server:TAG
Use matching tags across the server, proxy_server, and
colocated_python_sidecar images.
Prebuilt sidecar images support JAX versions 0.10.1, 0.10.2, 0.11.0,
0.11.1, 0.11.2, 0.12.0, and 0.12.0.1. Replace
TAG with one of the following
tag formats:
jax-JAX_VERSION(recommended): a rolling tag for a specific JAX release, such asjax-0.12.0.1.YYYYMMDD-jax_JAX_VERSION: a date-stamped build tag, such as20261110-jax_0.12.0.1.
If you need an exact image version that never changes, pin the image by its
SHA-256 digest. To look up an image digest, use the
gcloud artifacts docker images describe
command:
IMAGE_REPO="us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways"
gcloud artifacts docker images describe \
"${IMAGE_REPO}/colocated_python_sidecar:jax-JAX_VERSION" \
--format="value(image_summary.digest)"
Replace JAX_VERSION with your
JAX version, such as 0.12.0.1.
Build a custom sidecar image
The prebuilt sidecar image includes the jax, jaxlib, numpy, scipy,
ml-dtypes, opt-einsum, cloudpickle, and absl-py packages. If your
colocated Python functions use only these packages, skip to
Deploy a Pathways JobSet with the sidecar or
Deploy with Cluster Toolkit.
To add custom Python dependencies, layer them onto the prebuilt image matching your JAX version:
In an empty build directory, create a
requirements.txtfile for your extra packages. Omit thejaxandjaxlibpackages, which are already installed and pinned in the/opt/venv/server_constraints.txtfile:grainCreate a
Dockerfilethat installs the packages in therequirements.txtfile, constrained by the/opt/venv/server_constraints.txtfile:ARG JAX_VERSION=0.12.0.1 ARG IMAGE_REPO=us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways FROM ${IMAGE_REPO}/colocated_python_sidecar:jax-${JAX_VERSION} WORKDIR /app COPY requirements.txt . RUN uv pip install -r requirements.txt \ -c /opt/venv/server_constraints.txt && \ uv cache cleanKeep the base image's
ENTRYPOINTandCMDinstructions unchanged. If therequirements.txtfile installs packages from Git URLs, add the followingRUNinstruction before theuv pip installRUNinstruction, because the slim base image omits thegittool:RUN apt-get update && \ apt-get install -y --no-install-recommends git && \ rm -rf /var/lib/apt/lists/*Authenticate Docker to your Artifact Registry region by using the
gcloud auth configure-dockercommand:gcloud auth configure-docker REGISTRY_LOCATION-docker.pkg.devReplace
REGISTRY_LOCATIONwith your Artifact Registry location, such asusorus-central1.Build and push your custom sidecar image:
export JAX_VERSION=JAX_VERSION IMAGE_PREFIX=REGISTRY_LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY export SIDECAR_IMAGE="${IMAGE_PREFIX}/colocated-sidecar:jax-${JAX_VERSION}" docker build --build-arg JAX_VERSION="${JAX_VERSION}" \ --tag "${SIDECAR_IMAGE}" . docker push "${SIDECAR_IMAGE}"Replace the following:
JAX_VERSION: your JAX version, such as0.12.0.1.REGISTRY_LOCATION: your Artifact Registry location, such asusorus-central1.PROJECT_ID: your Google Cloud project ID.REPOSITORY: your Artifact Registry Docker repository name.
Deploy a Pathways JobSet with the sidecar
You can deploy Pathways with the sidecar either by applying a JobSet manifest
with the kubectl command-line tool in this section, or by submitting the job
directly with the gcluster CLI in
Deploy with Cluster Toolkit.
To generate and apply a JobSet manifest, complete the following steps:
Generate a
JobSetmanifest (pathways-colocated-jobset.yaml) with either of the following tools:- The
pathways-utilsPython builder: Configure aJobSetwith thePathwaysJobSetbuilder by calling the.add_colocated_python("SIDECAR_IMAGE")and.export_yaml("pathways-colocated-jobset.yaml")methods. If you use thepathwaysutilspackage (version0.1.14or earlier from PyPI), also pass theproxy_container_kwargs={"args": ["--sidecar_name=external"]}argument to thePathwaysJobSetconstructor, or install thepathways-utilspackage from themainbranch. - Cluster Toolkit (
gcluster): Run thegcluster job submitcommand with the--pathways-colocated-python-sidecar-imageflag and append the--dry-run-out=pathways-colocated-jobset.yamlflag to render the manifest locally without submitting the job.
Replace
SIDECAR_IMAGEwith your custom sidecar image URI or the prebuiltcolocated_python_sidecarimage URI.Both generators configure the manifest to do the following:
- Run the
colocated-python-sidecarcontainer under theinitContainersfield withrestartPolicy: Alwaysin each worker Pod. ThegclusterCLI names the worker replicated jobworker, whereas thePathwaysJobSetbuilder names itpathways-worker. - Pass the
--sidecar_name=externalflag to thepathways-proxycontainer, and connect thepathways-workercontainer to the sidecar over loopback TCP onlocalhost:50051. When thepathways-proxycontainer runs under theinitContainersfield, it also usesrestartPolicy: Always. - Mount an in-memory
emptyDirvolume withmedium: Memoryin both worker Pod containers: at/tmp/sidecaringclustermanifests, or at/tmp/shared-memoryinPathwaysJobSetmanifests. Both generators also set theCLOUD_PATHWAYS_SIDECAR_SHM_DIRECTORYenvironment variable on the sidecar and the--cloud_pathways_sidecar_shm_directoryflag on thepathways-workercontainer.
- The
Apply the
JobSetmanifest and wait for the Pods to reach theRunningstatus:kubectl apply -f pathways-colocated-jobset.yaml kubectl get pods -l jobset.sigs.k8s.io/jobset-name=JOBSET_NAMEReplace
JOBSET_NAMEwith themetadata.namevalue of yourJobSet.
Run one JAX client session per JobSet deployment. After a client disconnects,
a restarted pathways-worker container can't reuse a running sidecar. Delete
and recreate the JobSet before starting another session.
Files in an in-memory emptyDir volume count toward Pod memory usage. Set the
emptyDir.sizeLimit field (such as 175Gi) on the shared-memory volume to cap
usage per node. By default, both containers retain freed shared memory for 30
seconds of idle time for reuse. To change that window, set both the
--pathways_sidecar_shared_memory_idle_release_secs=SECONDS
flag in the args field of the colocated-python-sidecar container (alongside
the --port=50051 or --unix_domain_socket_path flag) and the
--cloud_pathways_shared_memory_idle_release_secs=SECONDS
flag in the args field of the pathways-worker container. Replace
SECONDS with the idle timeout in
seconds. Setting both flags to 0 releases freed pages within about 1 second,
which reduces memory usage at the cost of extra allocation overhead.
Optional: Use a Unix domain socket
To use a Unix domain socket (UDS) at /sock/sidecar.sock instead of opening a
TCP port on worker Pods that set hostNetwork: true, apply the following
overrides to your JobSet manifest before running the kubectl apply command:
- Mount an
emptyDir: {}volume namedsidecar-socketat/sockin both thepathways-workerandcolocated-python-sidecarcontainers. - Set
args: ["--unix_domain_socket_path=/sock/sidecar.sock"]and anexecstartupProbe(test -S /sock/sidecar.sock) on thecolocated-python-sidecarcontainer. - Append the
--cloud_pathways_external_sidecar_unix_domain_socket_path=/sock/sidecar.sockflag to theargsfield of thepathways-proxycontainer.
Deploy with Cluster Toolkit
In Cluster Toolkit (gcluster version 1.108.0 or later), passing
the --pathways-colocated-python-sidecar-image flag to the
gcluster job submit
command configures the
colocated-python-sidecar container over loopback TCP (localhost:50051),
mounts the in-memory volume at /tmp/sidecar, and sets the
--sidecar_name=external flag. Pass the --pathways-server-image and
--pathways-proxy-server-image flags explicitly so all three images use
matching cloud-pathways tags.
Submit a headless Pathways deployment with your sidecar image so you can run interactive client scripts against it:
IMAGE_REPO="us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways"
gcluster job submit \
--pathways \
--pathways-headless \
--pathways-gcs-location="gs://BUCKET_NAME/pathways-artifacts" \
--pathways-server-image="${IMAGE_REPO}/server:jax-JAX_VERSION" \
--pathways-proxy-server-image="${IMAGE_REPO}/proxy_server:jax-JAX_VERSION" \
--pathways-colocated-python-sidecar-image="SIDECAR_IMAGE" \
--name=WORKLOAD \
--cluster=CLUSTER \
--project=PROJECT_ID \
--location=LOCATION \
--num-slices=WORKLOAD_NODEPOOL_COUNT \
--compute-type=COMPUTE_TYPE \
--topology=TOPOLOGY
Replace the following:
BUCKET_NAME: your Cloud Storage scratch bucket.JAX_VERSION: your JAX version, such as0.12.0.1.SIDECAR_IMAGE: your custom or prebuilt sidecar image URI.WORKLOAD: a uniqueJobSetname (22 characters or fewer).CLUSTER: your GKE cluster name.PROJECT_ID: your Google Cloud project ID.LOCATION: your GKE cluster region or zone.WORKLOAD_NODEPOOL_COUNT: the number of Cloud TPU slices, such as1.COMPUTE_TYPE: the Cloud TPU machine type, such asct5lp-hightpu-4torct6e-standard-4t.TOPOLOGY: the Cloud TPU slice topology, such as4x8or2x4.
You can also customize how the gcluster job submit command runs or exports the
job:
- Batch workloads: To run a containerized batch job inside the
pathways-headPod instead of a headless deployment, replace the--pathways-headlessflag with the--image="CLIENT_IMAGE"and--command="COMMAND"flags. Use a Python 3.12 JAX client image built withserver_constraints.txtpins and pass your entry command (such aspython colocated_example.py). - Local manifest rendering: Append the
--dry-run-out=pathways-colocated-jobset.yamlflag to render theJobSetmanifest locally without submitting it. - Subsequent headless sessions: Run the
gcluster job cancelcommand and resubmit the job before starting another client session.
Run a colocated Python workload
Run your JAX client script with the JAX_PLATFORMS=proxy environment variable
against the pathways-proxy container, either in the pathways-head workload
container or over port forwarding:
To align serialization dependencies with your sidecar image, extract the
/opt/venv/server_constraints.txtfile and install matching packages in a Python 3.12 virtual environment:IMAGE_REPO="us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways" docker run --rm --entrypoint cat \ "${IMAGE_REPO}/colocated_python_sidecar:jax-JAX_VERSION" \ /opt/venv/server_constraints.txt > constraints.txt python3.12 -m venv .venv source .venv/bin/activate python -m pip install -c constraints.txt \ jax jaxlib cloudpickle pathwaysutilsReplace
JAX_VERSIONwith your JAX version, such as0.12.0.1.If you run the client outside the
pathways-headPod, forward local port29000to thepathways-proxycontainer in a separate terminal:JOBSET_LABEL="jobset.sigs.k8s.io/jobset-name=JOBSET_NAME" ROLE_LABEL="jobset.sigs.k8s.io/replicatedjob-name=pathways-head" HEAD_POD=$(kubectl get pods -l "${JOBSET_LABEL},${ROLE_LABEL}" \ -o jsonpath='{.items[0].metadata.name}') kubectl port-forward "${HEAD_POD}" 29000:29000Replace
JOBSET_NAMEwith themetadata.namevalue of yourJobSet(or theWORKLOADname that you passed to thegcluster job submitcommand).Save the following script as
colocated_example.py. This script allocates a 4 MiB NumPy array in shared host memory on each Cloud TPU worker's colocated CPU device and transfers it to Cloud TPU HBM:import jax from jax.experimental import colocated_python import numpy as np import pathwaysutils pathwaysutils.initialize() @colocated_python.colocated_python def make_host_array(seed_arr: jax.Array) -> jax.Array: host_array = np.full( (1024, 1024), fill_value=float(seed_arr), dtype=np.float32 ) return jax.device_put(host_array, seed_arr.device) tpu_devices = jax.devices() cpu_devices = colocated_python.colocated_cpu_devices(tpu_devices) print( f"TPU devices: {len(tpu_devices)}, " f"colocated CPU devices: {len(cpu_devices)}" ) host_arrays = [ make_host_array(jax.device_put(np.int32(i + 1), cpu_dev)) for i, cpu_dev in enumerate(cpu_devices) ] tpu_arrays = [ jax.device_put(host_arr, tpu_dev) for host_arr, tpu_dev in zip(host_arrays, tpu_devices) ] for arr in tpu_arrays: arr.block_until_ready() print( f"Transferred {len(tpu_arrays)} arrays of shape {tpu_arrays[0].shape} " f"to TPU HBM; sample sum={float(tpu_arrays[0].sum()):.1f}" )Run the script against the IFRT proxy server:
JAX_PLATFORMS=proxy JAX_BACKEND_TARGET=grpc://127.0.0.1:29000 \ python colocated_example.pyThe output is similar to the following:
TPU devices: 8, colocated CPU devices: 8 Transferred 8 arrays of shape (1024, 1024) to TPU HBM; sample sum=1048576.0In multislice production workloads, pass a sharded
jax.Arrayargument placed on the devices returned by thecolocated_python.colocated_cpu_devices(mesh)call and set theout_specs_fnparameter with the.specialize(out_specs_fn=...)method so Pathways runs one parallel call across all worker sidecars without waiting.Don't run client scripts inside a
workerPod with theJAX_PLATFORMS=cpuenvironment variable, because that setting runs JAX locally on one CPU and bypasses Pathways. You can ignore anyIFRT proxy server disconnected: CANCELLEDmessage printed on exit.
For additional usage patterns, see the following resources:
- JAX colocated Python
notebook:
function specialization and stateful
@colocated_python_classobjects - MaxText
multihost_dataloading.pymodule:RemoteIteratormultihost input pipeline that uses colocated Python - Orbax Pathways
checkpointing
and Multi-Tier Checkpointing with
Pathways:
distributed checkpointing on worker sidecars with the
orbax.checkpoint.pathways.register_type_handlers(checkpointing_impl=CheckpointingImpl.COLOCATED_PYTHON)call (or thecolocated_python_checkpointing=Truesetting in MaxText) and theuse_colocated_python=Trueoption for multi-tier checkpoints
Verify the sidecar deployment
Run the kubectl logs command on a headless JobSet before canceling or
deleting it, or while a batch workload is still running. In batch JobSet
deployments, the JobSet controller deletes worker Pods as soon as the
pathways-head Job completes. View completed batch logs in Cloud Logging
instead, as described in
Troubleshoot Pathways on
Cloud.
To verify a running deployment with the kubectl command-line tool, complete
the following steps:
Inspect the
colocated-python-sidecarcontainer logs:JOBSET_LABEL="jobset.sigs.k8s.io/jobset-name=JOBSET_NAME" ROLE_LABEL="jobset.sigs.k8s.io/replicatedjob-name=worker" WORKER_POD=$(kubectl get pods -l "${JOBSET_LABEL},${ROLE_LABEL}" \ -o jsonpath='{.items[0].metadata.name}') kubectl logs "${WORKER_POD}" -c colocated-python-sidecarReplace
JOBSET_NAMEwith themetadata.namevalue of yourJobSet(or theWORKLOADname that you passed to thegcluster job submitcommand). If you generated your manifest with thePathwaysJobSetbuilder, setreplicatedjob-name=pathways-workerin theROLE_LABELvariable.The output is similar to the following:
Started SidecarGrpcServer on localhost:50051 ... SidecarServer agreed on protocol version: 6 Installed shared memory allocator for colocated Python using file /tmp/sidecar/pathways_shm_sidecarConfirm that the output shows the gRPC startup message and, after the first
@colocated_pythoncall, the shared memory allocator message. Manifests generated with thePathwaysJobSetbuilder use the/tmp/shared-memorydirectory, and UDS deployments logunix:/sock/sidecar.sockinstead oflocalhost:50051.Inspect the
pathways-workercontainer logs on the same Pod:kubectl logs "${WORKER_POD}" -c pathways-worker \ | grep -iE "shared memory|external sidecar"The output is similar to the following:
Enabled shared memory for external sidecar in /tmp/sidecar External sidecar connected to backing GRPC sidecar at localhost:50051Confirm that the
pathways-workercontainer enabled shared memory and connected to the sidecar.
When you run the kubectl exec command to inspect the shared-memory directory,
target the colocated-python-sidecar container with the
-c colocated-python-sidecar flag, because the pathways-worker container
doesn't include a shell.
Troubleshoot colocated Python
The following sections describe how to resolve common colocated Python issues.
Unable to unpickle error
If your workload fails with the following error, Python or package versions differ between your JAX client and the sidecar container:
jax.errors.JaxRuntimeError: UNKNOWN: Unable to unpickle the callable and specialization
Check that your client runs Python 3.12 (python --version) and install client
packages with the -c constraints.txt flag extracted from the sidecar image.
For more information, see Run a colocated Python workload.
Dependency conflict during a docker build run
If the uv pip install command fails with the
No solution found when resolving dependencies error, a pin in your
requirements.txt file conflicts with the /opt/venv/server_constraints.txt
file. Remove the jax and jaxlib packages from the requirements.txt file
and relax version pins on packages such as numpy, ml-dtypes, or scipy.
Sidecar not enabled error
If the pathways-proxy container exits with the
Sidecar create args were set, but sidecar was not enabled error, add the
--sidecar_name=external flag to the args field of the pathways-proxy
container.
Sidecar already initialized error
If your workload fails with the
FAILED_PRECONDITION: Sidecar already initialized error, an earlier client
disconnect restarted the pathways-worker container while the
colocated-python-sidecar container stayed initialized. Delete and recreate the
JobSet before starting another client session.
Shared memory or sidecar connection errors
Use the following checks to resolve shared-memory mount, gRPC connection, or host memory errors:
Failed to stat sidecar_shm_directory: Mount the shared-memoryemptyDirvolume at the matching path (/tmp/sidecaror/tmp/shared-memory) in both thepathways-workerandcolocated-python-sidecarcontainers.Timed out connecting to sidecar gRPC serviceorFailed to connect to external sidecar: Confirm that thecolocated-python-sidecarcontainer runs under theinitContainersfield withrestartPolicy: Alwaysand that all threecloud-pathwaysimages use matching tags. Check that your TCP (--port=50051) or UDS (/sockvolume and--unix_domain_socket_path=/sock/sidecar.sock) settings match between thepathways-proxyandcolocated-python-sidecarcontainers.- Worker host out-of-memory after large array bursts: Set the
--pathways_sidecar_shared_memory_idle_release_secs=0flag on thecolocated-python-sidecarcontainer (alongside the--port=50051or--unix_domain_socket_pathflag) and the--cloud_pathways_shared_memory_idle_release_secs=0flag on thepathways-workercontainer. Setting both flags to0frees unused shared memory within about 1 second. Alternatively, reduce the peak amount of host array data that your workload keeps in memory at once.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this document, delete your deployment and temporary artifacts when you finish:
JobSetmanifest: Delete theJobSetresource:kubectl delete jobset JOBSET_NAMEReplace
JOBSET_NAMEwith themetadata.namevalue of yourJobSet.Cluster Toolkit: Cancel the deployed job:
gcluster job cancel WORKLOAD \ --cluster=CLUSTER \ --project=PROJECT_ID \ --location=LOCATIONReplace the following:
WORKLOAD: the name of your Pathways job.CLUSTER: your GKE cluster name.PROJECT_ID: your Google Cloud project ID.LOCATION: your GKE cluster region or zone.
Optional: Delete any custom sidecar images in Artifact Registry and remove temporary objects in your Cloud Storage scratch bucket. When you no longer need your GKE cluster, delete it by following the Clean up steps in Create a GKE cluster with Pathways.