Monitor workload framework metrics
This document explains how to collect and view Megascale XLA (MXLA), Orbax, and PyGrain (Grain) metrics for Cloud TPU workloads running on Google Kubernetes Engine (GKE), and how to view analyzers built on top of these metrics in the ML Diagnostics platform. By monitoring these metrics, you can identify and troubleshoot performance bottlenecks across multislice communication, checkpointing, and data loading.
Each set of metrics targets a distinct component of your workload to help you diagnose issues at different stages of your pipeline:
- Megascale XLA (MXLA) metrics: Measure cross-slice Data Center Network (DCN) transfer latencies, device-to-host (D2H) and host-to-device (H2D) transfer times, compute latency, and end-to-end collective operations in multislice TPU workloads. Use these metrics to identify stragglers and hardware bottlenecks.
- Orbax metrics: Track asynchronous checkpoint saves and restore durations, data volume, and write throughput from the Orbax checkpointing library. Use these metrics to detect checkpointing delays, stalls, or excessive training-thread blocking time.
- PyGrain (Grain) metrics: Measure dataset iteration latency, data source read times, total bytes read, and prefetch buffer readiness from the PyGrain data-loading library. Use these metrics to determine whether your workload is input-bound due to I/O or data-loading bottlenecks.
You can access these metrics through a Prometheus port
that you read directly, the TPU Monitoring
Library, the
tpu-info CLI, or Cloud Monitoring.
The ML Diagnostics platform also includes workload monitoring analyzers that automatically analyzes these metrics when a workload experiences performance degradation. These analyzers determine whether a drop in TPU duty cycle is caused by MXLA network transfer latency, Orbax asynchronous checkpointing, or PyGrain data loading.
Megascale metrics
Megascale XLA (MXLA) is a communication runtime and compiler extension that coordinates collective operations across multiple TPU slices over the Data Center Network (DCN). While Inter-Chip Interconnect (ICI) links handle high-speed communication between TPU chips within a single slice, MXLA manages data transfers between slices, host-side reductions, and data movement between host memory and TPU high bandwidth memory (HBM).
When you run multislice workloads on TPUs, you can collect MXLA metrics to troubleshoot cross-slice network and host-processing bottlenecks. GKE collects these metrics by default as GKE system metrics, and you can view them in Cloud Monitoring. The metrics are also automatically available in the ML Diagnostics system metrics tab, where they are aggregated automatically for the ML run workload context.
The following table lists the available Megascale metrics:
| Metric name | GKE metric name | Description |
|---|---|---|
| MXLA DCN transfer latency | container/multislice/network/dcn_transfer_latencies |
Measures round-trip Data Center Network (DCN) transfer latency for each chunk of data. Outliers can correlate with faulty network hardware. The metric has the following properties:
|
| MXLA inbound transfer latency | container/multislice/network/dcn_inbound_transfer_latencies |
Measures one-way inbound DCN transfer latency for each chunk of data. Outliers can correlate with faulty network hardware. The metric has the following properties:
|
| MXLA compute latency | container/multislice/accelerator/compute_latencies |
Measures the host compute latency required to complete a reduction operation. Outliers can indicate CPU or memory issues, such as dual inline memory module (DIMM) faults. The metric has the following properties:
|
| MXLA e2e collective latency | container/multislice/network/collective_end_to_end_latencies |
Measures end-to-end collective latency over the DCN, from when the host initiates the operation to when all peers receive the output. Includes host-side data reduction and sending output to the TPU. Results are strings that detail the buffer size, operation type, and mean, P50, P90, P95, and P99.9 latencies. The metric has the following properties:
|
| MXLA d2h latency | container/multislice/accelerator/device_to_host_transfer_latencies |
Measures one-way device-to-host (D2H) transfer latency for each chunk of data. Outliers can correlate with faulty network hardware. The metric has the following properties:
|
| MXLA h2d latency | container/multislice/accelerator/host_to_device_transfer_latencies |
Measures one-way host-to-device (H2D) transfer latency for each chunk of data. Outliers can correlate with faulty network hardware. The metric has the following properties:
|
Orbax metrics
Orbax is a distributed checkpointing library for JAX workloads that saves and restores model weights, optimizer states, and dataset iterators across single-host and multi-host environments. To minimize training downtime, Orbax supports asynchronous checkpointing, which copies states from TPU memory to host memory during the synchronous phase and then writes the checkpoint to storage in a background thread while training continues.
If your workload uses Orbax for checkpointing, you can collect Orbax metrics to troubleshoot checkpointing delays, slow write throughput, or stalls. Orbax exports these metrics on a Prometheus port, which you can read directly or send to Cloud Monitoring by using Google Cloud Managed Service for Prometheus. To enable these metrics in GKE, Cloud Monitoring, or ML Diagnostics, see Access Orbax and PyGrain metrics.
The following table lists the available Orbax metrics:
| Prometheus metric name | Description |
|---|---|
jax_checkpoint_write_async_total_duration_secs |
Total time for the entire asynchronous checkpoint write pipeline. The metric has the following properties:
|
jax_orbax_write_gbytes |
Total gigabytes written during a checkpoint save operation. The metric has the following properties:
|
jax_orbax_write_gbytes_per_sec |
Write throughput in gigabytes per second (GBps) across both the synchronous and asynchronous phases of a checkpoint save operation. The metric has the following properties:
|
jax_checkpoint_write_async_blocking_duration_secs |
Time spent in the blocking phase of an asynchronous checkpoint save before handing off to the background thread. The metric has the following properties:
|
jax_orbax_read_total_duration_secs |
Total end-to-end duration of a checkpoint restore operation. The metric has the following properties:
|
jax_orbax_read_blocking_duration_secs |
Duration of the synchronous (blocking) preparation phase of a checkpoint restore operation. The metric has the following properties:
|
jax_orbax_write_async_start_total |
Count of asynchronous checkpoint write operations started. The metric has the following properties:
|
PyGrain metrics
PyGrain (Grain) is a Python library for reading, preprocessing, and batching input data for machine learning training and evaluation workloads. PyGrain provides deterministic, reproducible data pipelines that prefetch and transform records on the host CPU so that TPU accelerators aren't stalled waiting for input batches.
If your workload uses PyGrain for data loading, you can collect PyGrain metrics to troubleshoot input-pipeline bottlenecks, slow storage reads, or depleted prefetch buffers. PyGrain exports these metrics on a Prometheus port, which you can read directly or send to Cloud Monitoring by using Google Cloud Managed Service for Prometheus. To enable these metrics in GKE, Cloud Monitoring, or ML Diagnostics, see Access Orbax and PyGrain metrics.
The following table lists the available PyGrain metrics:
| Prometheus metric name | Description |
|---|---|
grain_python_dataset_next_duration_ns |
Time spent in the The metric has the following properties:
|
grain_python_dataset_prefetch_buffer_ready_count |
Number of ready elements in the prefetch buffer. The metric has the following properties:
|
grain_python_data_sources_bytes_read_total |
Total bytes read from data sources. The metric has the following properties:
|
grain_python_dataset_source_read_time_ns |
Time spent reading from the data source. The metric has the following properties:
|
grain_python_data_loader_iterator_get_next |
Time spent in the The metric has the following properties:
|
Prerequisites
Before you collect Megascale, Orbax, or PyGrain metrics or use the associated ML Diagnostics analyzers, make sure that your environment meets the requirements in the following sections.
Megascale requirements
To collect Megascale metrics and view them in Cloud Monitoring, meet the following version requirements:
- Install LibTPU version
0.40.0or later. - To view these metrics in Cloud Monitoring, install GKE
version
1.36.4-gke.1247000or later.
Orbax and PyGrain requirements
To export Orbax and PyGrain metrics from your workload, meet the following package and container requirements:
- Python packages:
- Install Orbax (
orbax-checkpoint) version0.12.1or later. - Install PyGrain (
grain) version0.2.18or later. - Install the
prometheus-clientPython package in your container image. Theorbax-checkpointpackage installsprometheus-clientautomatically as a dependency, butgraindoesn't. If your workload uses only PyGrain, install the package by runningpip install prometheus-client.
- Install Orbax (
- Workload container configuration:
- Set the following environment variables:
ENABLE_ORBAX_PROMETHEUS_TELEMETRY="true"(enables Orbax metrics)ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY="true"(enables PyGrain metrics)
- Expose TCP port
9431on the workload container.
- Set the following environment variables:
The following example container specification sets the telemetry environment
variables and exposes TCP port 9431:
containers:
- name: WORKLOAD_CONTAINER
env:
- name: ENABLE_ORBAX_PROMETHEUS_TELEMETRY
value: "true"
- name: ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY
value: "true"
ports:
- name: framework-metrics
containerPort: 9431
protocol: TCP
Replace WORKLOAD_CONTAINER with the name of your workload
container.
ML Diagnostics analyzer requirements
To use the ML Diagnostics
analyzers
for Megascale, Orbax, and PyGrain metrics, install GKE version
1.36.4-gke.1247000 or later.
Access Megascale metrics
When your cluster runs a supported GKE version and LibTPU, GKE automatically collects Megascale metrics so that you can view them in Cloud Monitoring. You can also deploy a Prometheus collector of your choice to read Megascale metrics directly from the MXLA runtime's Prometheus port.
Access Orbax and PyGrain metrics
When you enable Prometheus telemetry on your workload container, Orbax and
PyGrain exports metrics on TCP port 9431. You can access these metrics in
either of the following ways:
- Read the Prometheus endpoint directly: Deploy a Prometheus collector of
your choice to read Orbax and PyGrain metrics from port
9431on the workload container. - Send metrics to Cloud Monitoring: Use
Google Cloud Managed Service for Prometheus to read port
9431on each node and send the metrics to Cloud Monitoring.
View metrics with Google Cloud Managed Service for Prometheus
To send Orbax and PyGrain metrics from port 9431 to Cloud Monitoring,
deploy a Google Cloud Managed Service for Prometheus PodMonitoring custom resource in
your GKE cluster. The PodMonitoring resource instructs the
Google Cloud Managed Service for Prometheus collector on each node to read port 9431 and
send the metrics to Cloud Monitoring. For more information, see Configure a
PodMonitoring
resource.
The prometheus.googleapis.com endpoint and the
Google Cloud Managed Service for Prometheus collector are enabled by default in
GKE clusters. Using Google Cloud Managed Service for Prometheus incurs
additional charges.
Enable managed collection
Managed collection for Google Cloud Managed Service for Prometheus is enabled by default in GKE Standard clusters running GKE version 1.27 or later. For more information, see Enable managed collection.
If managed collection is disabled on your cluster, enable it:
gcloud container clusters update CLUSTER_NAME \
--enable-managed-prometheus \
--region=REGION
Replace the following:
CLUSTER_NAME: the name of your GKE cluster.REGION: the region where your cluster is located.
If you modified the default permissions on the node service account, make sure
that it has the Monitoring Metric Writer (roles/monitoring.metricWriter)
role.
Configure the workload Pod template
To prepare your workload Pods for scraping, update the Pod template in your
JobSet, Job, or Deployment manifest:
- Add a Pod label, such as
ml-framework-metrics: "enabled", so that thePodMonitoringresource can select the workload Pods. - Set the
ENABLE_ORBAX_PROMETHEUS_TELEMETRYandENABLE_PYGRAIN_PROMETHEUS_TELEMETRYenvironment variables to"true". - Expose TCP port
9431and assign it a port name, such asframework-metrics.
The following example Pod template configures these fields:
template:
metadata:
labels:
ml-framework-metrics: "enabled" # used by the PodMonitoring selector
spec:
containers:
- name: WORKLOAD_CONTAINER
env:
- name: ENABLE_ORBAX_PROMETHEUS_TELEMETRY
value: "true"
- name: ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY
value: "true"
ports:
- name: framework-metrics
containerPort: 9431
protocol: TCP
Replace WORKLOAD_CONTAINER with the name of your workload
container.
Verify the metrics endpoint
Before you deploy the PodMonitoring resource, verify that your workload
container exports metrics on port 9431. Forward port 9431 from a running
workload Pod and query the /metrics endpoint:
kubectl -n NAMESPACE port-forward pod/WORKLOAD_POD 9431:9431
curl -s localhost:9431/metrics | grep -E '^(jax_orbax|jax_checkpoint|grain_python)'
Replace the following:
NAMESPACE: the Kubernetes namespace where your workload runs.WORKLOAD_POD: the name of a running workload Pod.
Orbax initializes each metric the first time it records a value, so Orbax metrics appear only after the first checkpoint save or restore operation.
Deploy the PodMonitoring resource
To configure Google Cloud Managed Service for Prometheus to read your workload Pods, complete the following steps:
Save the following manifest as
orbax-grain-podmonitoring.yaml:apiVersion: monitoring.googleapis.com/v1 kind: PodMonitoring metadata: name: orbax-grain-metrics namespace: WORKLOAD_NAMESPACE # must match the workload namespace spec: selector: matchLabels: ml-framework-metrics: "enabled" endpoints: - port: framework-metrics # the named containerPort (9431) path: /metrics interval: 30s # a longer interval reduces sample volume and cost metricRelabeling: # optional: keep only Orbax and PyGrain series to control cost - action: keep sourceLabels: [__name__] regex: (jax_orbax_|jax_checkpoint_|grain_python_).+ targetLabels: # optional: copy the JobSet name onto every series fromPod: - from: jobset.sigs.k8s.io/jobset-name to: jobset_nameReplace
WORKLOAD_NAMESPACEwith the Kubernetes namespace where your workload runs.Apply the manifest and confirm that the
PodMonitoringresource is created:kubectl apply -f orbax-grain-podmonitoring.yaml kubectl get podmonitoring -A
A PodMonitoring resource reads Pods only in its own namespace. If your
workloads run across multiple namespaces, use the same specification with kind:
ClusterPodMonitoring and omit the metadata.namespace field. A
ClusterPodMonitoring resource discovers matching Pods across all namespaces.
Optional: Verify that Google Cloud Managed Service for Prometheus is scraping the Pods
To verify that Google Cloud Managed Service for Prometheus is actively scraping your workload Pods, complete the following steps:
- In the
gmp-publicnamespace, edit theOperatorConfigresource and setfeatures.targetStatus.enabledtotrue. Inspect the
PodMonitoringresource and confirm that the targets reportHealth: up:kubectl -n NAMESPACE describe podmonitoring/orbax-grain-metricsReplace
NAMESPACEwith the Kubernetes namespace where your workload runs.After you finish troubleshooting, set
features.targetStatus.enabledback tofalsein theOperatorConfigresource. Enabling target status is recommended only when you are investigating active issues.
Query the metrics in Cloud Monitoring
To query the metrics sent to Cloud Monitoring, complete the following steps:
- In the Google Cloud console, go to Monitoring > Metrics Explorer.
Select PromQL, and then enter the following query to confirm that your targets are reporting:
up{namespace="NAMESPACE"}Replace
NAMESPACEwith your workload's Kubernetes namespace.
In the menu-based metric picker, metrics are stored using the format
prometheus.googleapis.com/METRIC_NAME/TYPE, where TYPE is counter, histogram, or gauge. Each
library constructs its Prometheus metric name from its metric path by replacing
/ with _. Counters append a _total suffix, and histograms are exported as
_bucket, _sum, and _count time series.
You can use the following example PromQL queries to analyze Orbax and PyGrain metrics:
# Asynchronous checkpoint saves started per Pod
sum by (pod) (increase(jax_orbax_write_async_start_total[1h]))
# Average time training is blocked by asynchronous checkpointing
sum(rate(jax_checkpoint_write_async_blocking_duration_secs_sum[10m]))
/ sum(rate(jax_checkpoint_write_async_blocking_duration_secs_count[10m]))
# PyGrain P99 dataset iterator next() latency (nanoseconds)
histogram_quantile(0.99, sum by (le) (rate(grain_python_dataset_next_duration_ns_bucket[5m])))
# PyGrain bytes read per second, by data source
sum by (source) (rate(grain_python_data_sources_bytes_read_total[5m]))
# Median prefetch-buffer readiness (low values can indicate an I/O bottleneck)
histogram_quantile(0.5, sum by (le) (rate(grain_python_dataset_prefetch_buffer_ready_count_bucket[5m])))
Cost considerations
Google Cloud Managed Service for Prometheus billing is based on the number of ingested
samples. To reduce costs, configure a longer scraping interval and use
metricRelabeling to keep only the time series that you need. You can view the
ingestion volume for each metric on the Metrics Management page in
Cloud Monitoring.
View analyzer outputs in ML Diagnostics
ML Diagnostics workload monitoring
and workload-contextualized system metrics are automatically enabled for
GKE TPU JobSet and Job workloads running on
GKE version 1.36.4-gke.1247000 and later. Workload monitoring
tracks the aggregated TPU duty cycle for your workload. In the event of a 15% or
greater drop in duty cycle (performance degradation), ML Diagnostics runs
analyzers across the infrastructure, orchestrator, and framework layers.
If your workload runs across multiple slices (using the MXLA layer) or uses the PyGrain or Orbax libraries, ML Diagnostics runs the following framework analyzers to determine whether the performance degradation was caused by network transfer latency, data loading, or asynchronous checkpointing:
ML Diagnostics analyzer reports are provided at no additional charge, and generating analyzer results doesn't require configuring Google Cloud Managed Service for Prometheus for Orbax and PyGrain metrics. In the System metrics tab of the ML Diagnostics Google Cloud console, Megascale metrics appear automatically when you use supported LibTPU and GKE versions.
You can view analyzer outputs in either of the following ways:
- Google Cloud console: In the ML Diagnostics Google Cloud console
(see View machine learning runs), open
the Monitoring overview tab, find the
performance degradationevent, and click View details to inspect the analyzer report. - Google Cloud CLI or REST API: Use the
monitored-eventscommands to list and describe events and their analyzer reports. For more information, see Get started with the ML Diagnostics CLI.