View instance distribution logs

When you create a regional managed instance group (MIG) or increase its target size, Compute Engine evaluates real-time resource availability to distribute new compute instances across zones and machine types. During this process, Compute Engine records an instance distribution log entry in Cloud Logging.

This document describes how to view and interpret these instance distribution logs. It also explains how to use these logs to investigate pending allocations and configure alerts.

To see the actual distribution of your running instances, see Monitor instance distribution in MIGs.

About instance distribution logs

Instance distribution logs provide a one-time snapshot of the MIG's planned allocations for compute instances. In these logs, an allocated instance means that the MIG found capacity and plans to create the compute instance on that machine type and in that zone. An allocated size doesn't mean that the compute instances are successfully created yet.

Because the logs provide a snapshot, Compute Engine doesn't update these logs when the MIG creates the remaining compute instances later. The log entry lets you view the following details:

Machine type distribution: the number of compute instances that the MIG has allocated per machine type. This information is present only for regional MIGs that use instance flexibility.
Zonal distribution: the number of compute instances that the MIG has allocated per zone.
Instances awaiting capacity: the number of instances for which the MIG couldn't allocate capacity immediately and is actively seeking capacity.

Limitations

Instance distribution logs are available only for regional MIGs.
For regional MIGs that use the EVEN target distribution shape without autoscaling, the log entry includes only the oldSize, newSize, deltaSize, and targetShape fields, without any instance distribution details.
Compute Engine logs instance distribution details only when you create a MIG or increase its target size. Compute Engine doesn't log these details when the target size decreases, or during instance repair, recreate, and update operations.

View instance distribution logs of a specific MIG

To view the instance distribution logs of a specific MIG, use one of the following options:

Console

You can view the logs for a specific MIG directly from the instance group details page.

In the Google Cloud console, go to the Instance groups page.

Go to Instance groups
Click the name of the MIG that you want to view.
Click the Monitoring tab.
In the left pane, select Logs.
In the log results filter, type instanceDistribution, and press Enter.
To view the instance distribution data, expand a log entry and then expand protoPayload.metadata.details.

gcloud

To view the instance distribution logs of a specific MIG, use the gcloud logging read command:

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
  protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND \
  resource.labels.instance_group_manager_name=\"MIG_NAME\""

Replace the following:

PROJECT_ID: the ID of your project.
MIG_NAME: the name of the MIG.

REST

To view the instance distribution logs of a specific MIG, make a POST request to the entries.list method.

POST https://logging.googleapis.com/v2/entries:list
{
"resourceNames": [
  "projects/PROJECT_ID"
],
"filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND resource.labels.instance_group_manager_name=\"MIG_NAME\""
}

Replace the following:

PROJECT_ID: the ID of your project.
MIG_NAME: the name of the MIG.

For more information about each field in a log, see Instance distribution log entry.

View all instance distribution logs in a project

To view all instance distribution logs in a project, select one of the following options:

Console

In the Google Cloud console, go to the Logs Explorer page.

Go to Logs Explorer

Enter a query. The following are some examples:

To view all instance distribution logs in the project:

logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.methodName="compute.instanceGroupManagers.instanceDistribution"

To view the instance distribution logs where part of the resize is still waiting for capacity:

logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.methodName="compute.instanceGroupManagers.instanceDistribution"
protoPayload.metadata.details.data.activeSeekDeltaSize > 0

Replace PROJECT_ID with the ID of your project.

Click Run query.
In the results, expand a log entry.
- To see the summary and the data, expand protoPayload.metadata.details.
- To see the summary on each row, click message and select Add field to summary line.

gcloud

To view the instance distribution logs, use the gcloud logging read command.

To view all instance distribution logs in the project:

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
  protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\""

To view all logs in the project from the last day, specify the --freshness=1d flag as follows:

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
    protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\"" \
    --freshness=1d

To view the instance distribution logs where part of the resize is still waiting for capacity:

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
    protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND \
    protoPayload.metadata.details.data.activeSeekDeltaSize > 0"

Replace PROJECT_ID with the ID of your project.

REST

To view the instance distribution logs, make a POST request to the entries.list method.

To view all instance distribution logs in the project:

POST https://logging.googleapis.com/v2/entries:list
{
  "resourceNames": [
    "projects/PROJECT_ID"
  ],
  "filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\""
}

To view all logs in the project from the last day, add a timestamp condition to the filter:

POST https://logging.googleapis.com/v2/entries:list
{
  "resourceNames": [
    "projects/PROJECT_ID"
  ],
  "filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND timestamp >= \"YYYY-MM-DDThh:mm:ssZ\""
}

To view the instance distribution logs where part of the resize is still waiting for capacity:

POST https://logging.googleapis.com/v2/entries:list
{
  "resourceNames": [
    "projects/PROJECT_ID"
  ],
  "filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND protoPayload.metadata.details.data.activeSeekDeltaSize > 0"
}

Replace the following:

PROJECT_ID: the ID of your project.
YYYY-MM-DDThh:mm:ssZ: the RFC 3339 timestamp for the start time—for example, 2024-10-22T00:00:00Z.

For more information about each field in a log, see Instance distribution log entry.

Instance distribution log entry

The following is a sample log entry (protoPayload.metadata.details). Because this event is a one-time snapshot, the log entry isn't updated when the remaining instances are created later. The log entry lets you view the details as follows:

details.data: shows the structured details about the instance distribution.
details.message: shows the summary of the instance distribution.

In this example, a regional MIG with instance flexibility is scaling out from 0 to 100. The MIG immediately allocates 80 of the 100 new instances across two ranked instance selections (20 c2-standard-4 instances in rank2-c2 selection and 60 n2d-standard-4 instances in rank3-n2d selection). The entry also shows that 40 compute instances are allocated in us-central1-b and us-central1-c each. For the remaining 20 instances (activeSeekDeltaSize), the MIG continues to seek capacity across all configured instance selections and zones. For regional MIGs without instance flexibility, the log entry omits the instanceSelections field and shows only the zonal distribution.

"metadata": {
  "details": {
    "data": {
      "oldSize": 0,
      "newSize": 100,
      "deltaSize": 100,
      "targetShape": "BALANCED",
      "activeSeekDeltaSize": 20,
      "instanceSelections": {
        "rank1-n2": {
          "rank": 1,
          "machineSelections": [
            {
              "machineType": "n2-standard-4",
              "allocatedSize": 0
            }
          ]
        },
        "rank2-c2": {
          "rank": 2,
          "machineSelections": [
            {
              "machineType": "c2-standard-4",
              "allocatedSize": 20
            }
          ]
        },
        "rank3-n2d": {
          "rank": 3,
          "machineSelections": [
            {
              "machineType": "n2d-standard-4",
              "allocatedSize": 60
            }
          ]
        }
      },
      "zonalDistribution": {
        "us-central1-a": {
          "allocatedSize": 0
        },
        "us-central1-b": {
          "allocatedSize": 40
        },
        "us-central1-c": {
          "allocatedSize": 40
        }
      }
    },
    "message": "Increasing MIG target size from 0 to 100 instances (+100).
    Allocated 80 of 100 new instances. Allocated instances across instance
    selections: rank2-c2 (rank 2; c2-standard-4: 20); rank3-n2d (rank 3;
    n2d-standard-4: 60). Allocated instances across zones based on target
    distribution shape BALANCED: us-central1-a: 0, us-central1-b: 40,
    us-central1-c: 40. MIG continues to allocate the remaining 20 new instances
    across all instance selections and zones. The final distribution across zones
    and machine types may differ as instances are created. View the group's
    instances for the current distribution."
  }
}

The following table describes each field in a log entry.

Field	Description
`details.data.oldSize`	The target size of the regional MIG before the resize operation.
`details.data.newSize`	The requested target size of the regional MIG.
`details.data.deltaSize`	The change in the target size (`newSize - oldSize`).
`details.data.targetShape`	The target distribution shape of the regional MIG.
`details.data.activeSeekDeltaSize`	The number of new instances for which the MIG is still seeking capacity across configured instance selections and zones. This field is omitted when its value is `0`.
`details.data.instanceSelections`	The configured instance selections evaluated during the allocation. This field is present only for MIGs that use instance flexibility. Each key is the name of an instance selection, and its value contains the following fields: `rank`: the priority rank configured for the instance selection. `machineSelections`: a list of machine type allocations within the selection, where each entry specifies the machine type (`machineType`) and the number of compute instances allocated using that machine type (`allocatedSize`).
`details.data.zonalDistribution`	The configured zones and their allocated instance distribution. Each key is a zone name, and its value contains the number of compute instances allocated in that zone (`allocatedSize`).
`details.message`	A summary message describing the instance distribution, including allocated counts and seeking capacity details.

Interpret instance distribution logs

The instance distribution log provides a snapshot of the MIG's plan for compute instance creation. Because the log indicates that the MIG allocates compute instances before they are successfully created, you should use the log data alongside compute instance status and error messages to understand your MIG's actual behavior.

Use the following table to interpret common log outputs and determine if you need to take action:

Log output	Explanation	Recommended action
`activeSeekDeltaSize` is greater than `0`	The MIG couldn't find enough capacity to fulfill the entire resize request immediately. The MIG keeps trying to create the remaining instances across all configured options.	Check the MIG's instance errors to see why instance creation has failed. Consider adding machine types from other machine families, adding zones, or checking your quotas.
A higher-preference (lower rank number) selection has an `allocatedSize` of `0`	At decision time, the MIG checked your most preferred machine types but didn't find enough capacity. The MIG used a lower-preference (higher rank number) selection instead. Compute Engine doesn't log instance errors for skipped selections.	If the MIG skips your most preferred selection often, then consider adding more machine types to your most preferred selection or accepting lower-preference options.
A configured zone has an `allocatedSize` of `0`	The MIG didn't allocate any new instances in this zone.	If your target shape is `ANY` or `ANY_SINGLE_ZONE`, this uneven distribution is normal. Otherwise, review your quota, reservations, and zone support for your instance template.
A machine type is missing from the `message` summary	The MIG didn't use that machine type for allocation.	Check the structured `details.data` block. The summary message leaves out selections and machine types that got `0` new instances, but the structured data lists all evaluated options.

Check pending instance allocations

If your MIG is still waiting for capacity (activeSeekDeltaSize > 0), then you can check the following to understand the reasons behind the pending allocations:

Check instance creation errors. To see if instance creation failed after the MIG allocated capacity, or if compute instances are still waiting for capacity, check the instance errors. For more information, see List instance errors.
Check instance status. To see the status and current action of each instance, list the instances in the MIG and check their instanceStatus field. For more information, see Check the status of managed instances.
Review your configuration. If you've configured an instance flexibility policy in the MIG, then review your regional quotas for the machine families in the policy. Also, verify that your selected zones support the requested machine types. For more information, see View and manage quotas and Available regions and zones.

Create alerting policies based on logs

To get notified when MIG resize operations wait for capacity, you can create a log-based alerting policy in Logs Explorer. For example, you can trigger an alert when activeSeekDeltaSize is greater than 0.

To create this alert, do the following:

In the Google Cloud console, go to the Logs Explorer page.

Go to Logs Explorer

Enter the following query:

logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.methodName="compute.instanceGroupManagers.instanceDistribution"
protoPayload.metadata.details.data.activeSeekDeltaSize > 0

Replace PROJECT_ID with the ID of your project.

Click Run query to validate the query.
Create the alerting policy by following the instructions in the create a log-based alerting policy section of the Cloud Logging documentation.

What's next

Learn how regional MIGs distribute compute instances across zones.
Learn how multiple machine type selection works in MIGs.
View instance flexibility configuration in a MIG.

View instance distribution logs Stay organized with collections Save and categorize content based on your preferences.

About instance distribution logs

Limitations

View instance distribution logs of a specific MIG

Console

gcloud

REST

View all instance distribution logs in a project

Console

gcloud

REST

Instance distribution log entry

Interpret instance distribution logs

Check pending instance allocations

Create alerting policies based on logs

What's next

View instance distribution logs