Troubleshoot the cluster health scanner

This document describes common issues with the cluster health scanner and how to resolve them.

Scan skips nodes or stops because active workloads were detected

Issue: The scan skips some GPU nodes, or stops during the pre-flight checks for a TPU node pool, and reports that active workloads were detected.

Cause: One or more target nodes are running workloads.

Resolution: Drain or stop the workloads on the target nodes, and then rerun the scan. For more information, see Verify that the target nodes are idle.

Scan stops because of mixed machine types

Issue: The scan stops during the pre-flight checks and reports that the target nodes use different machine types.

Cause: The node pools that you specified in the --nodepools flag or the partitions that you specified in the --partitions flag don't all use the same machine type. For example, a single scan can't mix GPU and TPU node pools, or A3 Mega and A4 nodes.

Resolution: Run a separate scan for each machine type.

Scan stops because of a storage permission error

Issue: The scan stops during the pre-flight checks and reports a Cloud Storage error.

Cause: The bucket doesn't exist, or the principal that writes the report doesn't have permission to write to the bucket.

Resolution: Verify that the bucket exists and that the principal has the Storage Object Admin role on the bucket. For more information, see Set up a Cloud Storage bucket for reports.

Command-line tool can't connect to the cluster

Issue: The chs command-line tool returns an authentication or connection error.

Cause: The kubeconfig entry for the cluster is missing, or your credentials aren't available inside the container.

Resolution: Run the gcloud container clusters get-credentials command, as described in Connect to your cluster or instance. Then verify that your kubeconfig file and Google Cloud credentials are mounted in the container, as described in Install the cluster health scanner.

Caller doesn't have permission to access the project

Issue: A chs command returns the following error:

ERROR: The caller doesn't have permission to access the project PROJECT_ID

Cause: Your credentials are missing or expired, or the active identity doesn't have the required Identity and Access Management (IAM) permissions on the project.

Resolution: Set up Application Default Credentials with your user account:

gcloud auth application-default login

If you also need to authenticate the Google Cloud CLI with your user account rather than a service account, run the following command:

gcloud auth login --no-browser

GPU tests report driver or hardware errors

Issue: The health report shows a DEGRADED or INCOMPLETE status for some nodes after the nvidia-smi or dcgm test runs.

Cause: A GPU on the node isn't detected, the GPU driver isn't working, or NVIDIA Data Center GPU Manager (DCGM) detected a hardware fault.

Resolution: Review the diagnostic logs in the workspace directory of your local output directory or Cloud Storage bucket. Then follow the steps for your environment in Remediate unhealthy nodes.

Network tests report low bandwidth or high latency

Issue: The health report shows low collective communication bandwidth or high latency for some nodes.

Cause: On GPUs, this is caused by a faulty GPU, NVLink connection, or network interface on one or more nodes. On TPUs, this is caused by a faulty ICI link or TPU chip on one or more nodes.

Resolution: For GPUs, use the nccl-pairwise test results to identify which nodes are involved in the slow node pairs. For TPUs, review the tpu-collectives and tpu-network-stress results in the health report. Then follow the steps for your environment in Remediate unhealthy nodes.

Resources remain on the cluster after a scan is stopped

Issue: Diagnostic jobs or resources remain on your Google Kubernetes Engine (GKE) cluster after the chs process was terminated.

Cause: On GKE, the chs process must remain running for the duration of a scan to clean up resources when the scan finishes.

Resolution: Cancel the stopped run or all runs on the cluster to remove leftover jobs and release the nodes:

chs cancel RUN_ID

To cancel all runs on the cluster, run chs cancel --all.

Scan fails on a standalone TPU instance when TPUs are busy

Issue: When you run a scan on a standalone TPU instance, the chs command-line tool returns a TPUs are busy error.

Cause: Another process on the TPU instance is using the TPU devices.

Resolution: Wait for the other process to finish or stop the process, and then rerun the scan.

Scan fails with a memlock limit error on a standalone TPU instance

Issue: When you run a scan on a standalone TPU instance, the chs command-line tool reports that the memlock limit is too low or that TPU tests require root or an unlimited memlock limit.

Cause: The current user session has a restricted locked-in-memory address space (memlock) limit.

Resolution: Run the cluster health scanner as the root user, or raise the memlock limit before you run the scan:

ulimit -l unlimited

Container fails on a standalone TPU instance without access to the host

Issue: When you run a scan on a standalone TPU instance by using the container image, the chs command-line tool fails with the following error:

CHS runs in a container without access to the host

Cause: In standalone mode on a TPU instance, the container must share the host PID namespace and mount the host file system to inspect and use the TPU devices under /dev/vfio.

Resolution: Start the container with the --privileged, --pid=host, and -v /:/host flags. For the full docker run command, see Container image.

Package installation fails with a Python version error

Issue: Running pip install fails with the error: Package 'chs' requires a different Python.

Cause: The python3 version on your machine is earlier than 3.12, which is common on Slurm login nodes and TPU instances.

Resolution: Use uv to create a Python 3.12 virtual environment and install the cluster health scanner without root privileges:

python3 -m venv uv-venv
uv-venv/bin/pip install uv
uv-venv/bin/uv venv --python 3.12 chs-venv
source chs-venv/bin/activate
uv-venv/bin/uv pip install git+https://github.com/GoogleCloudPlatform/cluster-health-scanner.git

Scan skips a standalone TPU instance because TPU test dependencies aren't installed

Issue: When you run a scan on a standalone TPU instance, chs run skips the instance and returns the following warning:

The TPU test dependencies are not installed. CHS installs them only on Linux x86_64 with Python 3.12.

Cause: The cluster health scanner Python package was installed with a Python version other than 3.12 (for example, Python 3.13). On standalone TPU instances, the TPU benchmark dependencies are installed only on Linux x86_64 with Python 3.12.

Resolution: Reinstall the cluster health scanner in a Python 3.12 virtual environment by using uv (see Package installation fails with a Python version error), or use the cluster health scanner container image.