This document describes known issues that you might encounter when you run AI-optimized Compute Engine instances or clusters. For other troubleshooting resources, see the following guides:
- For issues with compute instances, see Compute Engine known issues.
- For
ZONE_RESOURCE_POOL_EXHAUSTEDor capacity errors when you create compute instances or clusters, see Troubleshoot resource availability errors. - For errors when you attempt to consume reservations, see Troubleshoot reservation consumption.
Issues
The following section lists known issues and workarounds for AI Hypercomputer.
Workload interruptions on A4 VMs due to firmware issues for NVIDIA B200 GPUs
NVIDIA has identified two firmware issues for B200 GPUs, which A4 virtual
machine (VM) instances use. These issues can cause workload interruptions, such
as the nvidia-smi command not responding. If you notice workload interruptions
on A4 VMs, then confirm that one or more of the following conditions are true:
- The VM's uptime (
lastStartTimestamp) exceeds 65 days. - Logs show an
Xid 149error message that mentions0x02a.
If one of these conditions is true, then you must reset the GPUs that are attached to your A4 VMs. To prevent future interruptions to your workload due to firmware issues, we recommend that you reset your GPUs at least once every 60 days. For more information, see Reset GPUs.
The metadata server might display old physicalHost metadata
After experiencing a
host error or
using the
report faulty host API to
move a compute instance to a new host, when you
query the metadata server,
it might display the physicalHost metadata of the compute instance's previous
host.
To workaround this issue, do one of the following:
- Use the
instances.getmethod or thegcloud compute instances describecommand to retrieve the correctphysicalHostinformation. - Stop and then start your
instance. This process updates the
physicalHostinformation in the metadata server. - Wait 24 hours for the impacted instance's
physicalHostinformation to be updated.