This document describes common issues with the cluster health scanner and how to resolve them.
Scan skips nodes or stops because active workloads were detected
Issue: The scan skips some GPU nodes, or stops during the pre-flight checks for a TPU node pool, and reports that active workloads were detected.
Cause: One or more target nodes are running workloads.
Resolution: Drain or stop the workloads on the target nodes, and then rerun the scan. For more information, see Verify that the target nodes are idle.
Scan stops because of mixed machine types
Issue: The scan stops during the pre-flight checks and reports that the target nodes use different machine types.
Cause: The node pools that you specified in the --nodepools flag or the
partitions that you specified in the --partitions flag don't all use the same
machine type. For example, a single scan can't mix GPU and TPU node pools, or A3
Mega and A4 nodes.
Resolution: Run a separate scan for each machine type.
Scan stops because of a storage permission error
Issue: The scan stops during the pre-flight checks and reports a Cloud Storage error.
Cause: The bucket doesn't exist, or the principal that writes the report doesn't have permission to write to the bucket.
Resolution: Verify that the bucket exists and that the principal has the Storage Object Admin role on the bucket. For more information, see Set up a Cloud Storage bucket for reports.
Command-line tool can't connect to the cluster
Issue: The chs command-line tool returns an authentication or connection
error.
Cause: The kubeconfig entry for the cluster is missing, or your
credentials aren't available inside the container.
Resolution: Run the gcloud container clusters get-credentials command, as
described in Connect to your cluster or
instance. Then
verify that your kubeconfig file and Google Cloud credentials are mounted in
the container, as described in Install the cluster health
scanner.
Caller doesn't have permission to access the project
Issue: A chs command returns the following error:
ERROR: The caller doesn't have permission to access the project PROJECT_ID
Cause: Your credentials are missing or expired, or the active identity doesn't have the required Identity and Access Management (IAM) permissions on the project.
Resolution: Set up Application Default Credentials with your user account:
gcloud auth application-default login
If you also need to authenticate the Google Cloud CLI with your user account rather than a service account, run the following command:
gcloud auth login --no-browser
GPU tests report driver or hardware errors
Issue: The health report shows a DEGRADED or INCOMPLETE status for some
nodes after the nvidia-smi or dcgm test runs.
Cause: A GPU on the node isn't detected, the GPU driver isn't working, or NVIDIA Data Center GPU Manager (DCGM) detected a hardware fault.
Resolution: Review the diagnostic logs in the workspace directory of your local output directory or Cloud Storage bucket. Then follow the steps for your environment in Remediate unhealthy nodes.
Network tests report low bandwidth or high latency
Issue: The health report shows low collective communication bandwidth or high latency for some nodes.
Cause: On GPUs, this is caused by a faulty GPU, NVLink connection, or network interface on one or more nodes. On TPUs, this is caused by a faulty ICI link or TPU chip on one or more nodes.
Resolution: For GPUs, use the nccl-pairwise test results to identify which
nodes are involved in the slow node pairs. For TPUs, review the
tpu-collectives and tpu-network-stress results in the health report. Then
follow the steps for your environment in Remediate unhealthy
nodes.
Resources remain on the cluster after a scan is stopped
Issue: Diagnostic jobs or resources remain on your
Google Kubernetes Engine (GKE) cluster after the chs process was terminated.
Cause: On GKE, the chs process must remain running for the
duration of a scan to clean up resources when the scan finishes.
Resolution: Cancel the stopped run or all runs on the cluster to remove leftover jobs and release the nodes:
chs cancel RUN_ID
To cancel all runs on the cluster, run chs cancel --all.
Scan fails on a standalone TPU instance when TPUs are busy
Issue: When you run a scan on a standalone TPU instance, the chs
command-line tool returns a TPUs are busy error.
Cause: Another process on the TPU instance is using the TPU devices.
Resolution: Wait for the other process to finish or stop the process, and then rerun the scan.
Scan fails with a memlock limit error on a standalone TPU instance
Issue: When you run a scan on a standalone TPU instance, the chs
command-line tool reports that the memlock limit is too low or that TPU tests
require root or an unlimited memlock limit.
Cause: The current user session has a restricted locked-in-memory address
space (memlock) limit.
Resolution: Run the cluster health scanner as the root user, or raise the
memlock limit before you run the scan:
ulimit -l unlimited
Container fails on a standalone TPU instance without access to the host
Issue: When you run a scan on a standalone TPU instance by using the
container image, the chs command-line tool fails with the following error:
CHS runs in a container without access to the host
Cause: In standalone mode on a TPU instance, the container must share the
host PID namespace and mount the host file system to inspect and use the TPU
devices under /dev/vfio.
Resolution: Start the container with the --privileged, --pid=host, and
-v /:/host flags. For the full docker run command, see Container
image.
Package installation fails with a Python version error
Issue: Running pip install fails with the error: Package 'chs' requires a
different Python.
Cause: The python3 version on your machine is earlier than 3.12, which is
common on Slurm login nodes and TPU instances.
Resolution: Use uv to create a Python 3.12 virtual environment and install
the cluster health scanner without root privileges:
python3 -m venv uv-venv
uv-venv/bin/pip install uv
uv-venv/bin/uv venv --python 3.12 chs-venv
source chs-venv/bin/activate
uv-venv/bin/uv pip install git+https://github.com/GoogleCloudPlatform/cluster-health-scanner.git
Scan skips a standalone TPU instance because TPU test dependencies aren't installed
Issue: When you run a scan on a standalone TPU instance, chs run skips the
instance and returns the following warning:
The TPU test dependencies are not installed. CHS installs them only on Linux x86_64 with Python 3.12.
Cause: The cluster health scanner Python package was installed with a Python
version other than 3.12 (for example, Python 3.13). On standalone TPU instances,
the TPU benchmark dependencies are installed only on Linux x86_64 with Python
3.12.
Resolution: Reinstall the cluster health scanner in a Python 3.12 virtual
environment by using uv (see Package installation fails with a Python version
error), or use the cluster health scanner container
image.