Known issues

This document describes known issues that you might encounter when you run AI-optimized Compute Engine instances or clusters. For other troubleshooting resources, see the following guides:

Issues

The following section lists known issues and workarounds for AI Hypercomputer.

Workload interruptions on A4 VMs due to firmware issues for NVIDIA B200 GPUs

NVIDIA has identified two firmware issues for B200 GPUs, which A4 virtual machine (VM) instances use. These issues can cause workload interruptions, such as the nvidia-smi command not responding. If you notice workload interruptions on A4 VMs, then confirm that one or more of the following conditions are true:

If one of these conditions is true, then you must reset the GPUs that are attached to your A4 VMs. To prevent future interruptions to your workload due to firmware issues, we recommend that you reset your GPUs at least once every 60 days. For more information, see Reset GPUs.

The metadata server might display old physicalHost metadata

After experiencing a host error or using the report faulty host API to move a compute instance to a new host, when you query the metadata server, it might display the physicalHost metadata of the compute instance's previous host.

To workaround this issue, do one of the following: