Troubleshoot feature flags

This guide provides steps to diagnose and resolve common issues with App Lifecycle Manager feature flags.

SDK and runtime evaluation

The OpenFeature API is designed so that flag evaluation calls don't throw unhandled exceptions and instead return the default value if an evaluation error occurs.

Symptom: Flag returning safe default value

If the flag service is unreachable, the provider is unavailable, or the flag is misconfigured, the SDK returns the required default value provided in your code.

Resolution:

  1. Verify the connection between your OpenFeature provider and the saasconfig.googleapis.com endpoint.
  2. Check for authentication errors. Ensure that your application's runtime service account has the roles/saasconfig.viewer role.
  3. Verify FLAGD_SOURCE_PROVIDER_ID is correct.

Symptom: Unexpected evaluation results

If a flag is not returning the expected variant:

Resolution:

  1. Check your CEL conditions and evaluation context.
  2. If a required attribute is missing from the context during a condition check, the evaluation engine skips that specific rule.
  3. If an attribute is missing during a percentage-based allocation, the evaluation returns INVALID_CONTEXT.

Troubleshoot App-Centric template sync and console rollouts

If you onboard to feature flags using Application Design Center composite templates and Google Cloud console, use the following steps to resolve template synchronization and rollout issues.

Symptom: Unit kinds aren't created after connecting an App Design Center composite template

If the Unit kinds tab in your SaaS offering remains empty after you connect a composite template:

Resolution:

  1. Verify that each embedded standard application template and the parent composite application template have been added to the catalog in Application Design Center before importing the composite template into your SaaS offering.
  2. Verify that the App Lifecycle Manager service agent (service-PROJECT_NUMBER@gcp-sa-saasservicemgmt.) has the roles/saasservicemgmt.serviceAgent role in your project.
  3. Check the SaaS offering details page in Google Cloud console or run gcloud beta app-lifecycle-manager saas describe SAAS_OFFERING_ID --location=global to inspect template sync status.

Replace the following:

  • PROJECT_NUMBER: your Google Cloud project number.
  • SAAS_OFFERING_ID: the ID of your SaaS offering.

Symptom: Flag rollout in the console doesn't update a unit created from a standard template

If a flag rollout completes in Google Cloud console, but a unit created from a standard template doesn't receive the updated flag configuration:

Resolution:

  1. Verify that the unit belongs to the targeted unit kind, is in a region included in the rollout, and matches any unitFilter configured on the rollout kind.
  2. Confirm that FLAGD_SOURCE_PROVIDER_ID in your application matches the unit's full feature flag configuration resource path (projects/PROJECT_ID/locations/LOCATION/featureFlagsConfigs/UNIT_ID).

Replace the following:

  • PROJECT_ID: your Google Cloud project ID.
  • LOCATION: the region of your unit.
  • UNIT_ID: the ID of your unit.

Monitor rollouts

Monitoring a feature flag rollout involves verifying both the orchestration state (did the rollout complete?) and the application health (did the binary pick up the flag?).

Rollout orchestration state

The system monitors the success rate of operations in real-time.

Automatic pausing

To prevent a faulty configuration from reaching your entire fleet, the system automatically pauses a rollout if the failure rate in a location exceeds the error budget configured on the rollout kind (errorBudget.allowedCount or errorBudget.allowedPercentage, such as a 10% per-location threshold).

Check summary stats

Use the describe command to see the aggregate failure count and determine if the rollout is PAUSED or FAILED.

gcloud beta app-lifecycle-manager rollouts describe ROLLOUT_ID --location=global

Filter failed unit updates

After you confirm failures exist, you must find the specific units that missed the update. Use this filter to isolate only the failed flag updates in a specific region:

gcloud beta app-lifecycle-manager unit-operations list \
    --location=LOCATION \
    --filter="state:UNIT_OPERATION_STATE_FAILED AND flag_update:*"

Diagnose the root cause

For each failed operation, you can retrieve a detailed error message provided by the actuation engine in the conditions field (message and reason) and errorCategory (or inspect stateMessage on the rollout resource).

gcloud beta app-lifecycle-manager unit-operations describe OPERATION_ID --location=LOCATION

Replace the following:

  • ROLLOUT_ID: the ID of the rollout.
  • LOCATION: the region of the unit operation.
  • OPERATION_ID: the ID of the unit operation.

Unit missing configs

This section describes how unit pinning can prevent a unit from receiving feature flag configurations and how to recover the unit.

Understand unit pinning

Pinning freezes a unit at its current release, preventing it from being updated by any automated or manual rollouts. A pinned unit is explicitly excluded from rollout operations until its pinning period expires or it is manually unpinned.

Missing config from pinned units

A critical scenario occurs when a unit is pinned during a feature flag rollout. Pinning blocks all updates, meaning the unit skips the FlagUpdate operation required to receive its configuration.

Symptom: After the unit is unpinned and upgraded to a binary that requires feature flags, the application fails to initialize or enters an infinite retry loop, leading to failed readiness probes and continuous container restarts.

Root cause: While the unit was pinned, rollouts updated the parent unit kind resource's defaultFlagRevisions field, but the pinned unit skipped the FlagUpdate operation and received no active configuration.

Diagnostic commands

Use the following commands to inspect failed flag update operations and unit status.

1. List failed flag operations

To find units that missed an update, filter the unit operations for the UNIT_OPERATION_STATE_FAILED state and the flag_update type.

gcloud beta app-lifecycle-manager unit-operations list \
    --location="LOCATION" \
    --filter="state:UNIT_OPERATION_STATE_FAILED AND flag_update:*"

2. Filter specifically for failed flag rollouts

To pinpoint failed flag updates across your fleet or for a specific unit, add a condition for the UNIT_OPERATION_STATE_FAILED state.

gcloud beta app-lifecycle-manager unit-operations list \
    --location="LOCATION" \
    --filter="unit:UNIT_ID AND state:UNIT_OPERATION_STATE_FAILED AND flag_update:*"

3. Inspect the failure reason

Use the describe command to inspect the conditions field (message and reason) and errorCategory.

gcloud beta app-lifecycle-manager unit-operations describe "OPERATION_ID" --location="LOCATION"

4. Check unit condition

You can also check the current state of a unit directly. If a flag rollout is failing or the last operation resulted in an error, the unit's conditions list contains an entry with type: TYPE_OPERATION_ERROR.

gcloud beta app-lifecycle-manager units describe "UNIT_ID" --location="LOCATION"

Replace the following:

  • LOCATION: the region of the unit or unit operation.
  • UNIT_ID: the ID of the unit.
  • OPERATION_ID: the ID of the unit operation.

Manual mitigation

To recover a pinned unit that missed flag updates, apply the default flag revisions from its parent unit kind.

Apply unit kind default flag revisions

If a unit is missing its configuration because it was pinned during one or more rollouts, you can recover it by manually resetting the unit to the current baseline defined in its unit kind.

  1. Unpin the unit (if pinned):

    gcloud beta app-lifecycle-manager units update "UNIT_ID" \
        --location="LOCATION" \
        --maintenance-pinned-until-time="1970-01-01T00:00:00Z"
    
  2. Retrieve default revisions from the parent unit kind:

    gcloud beta app-lifecycle-manager unit-kinds describe "UNIT_KIND_ID" \
        --location="LOCATION" \
        --format="value(defaultFlagRevisions.join(','))"
    
  3. Create a new FlagRelease containing those revisions:

    gcloud beta app-lifecycle-manager flags releases create "recovery-release-1" \
        --location="LOCATION" \
        --unit-kind="UNIT_KIND_ID" \
        --revisions="REVISIONS"
    
  4. Manually initiate a FlagUpdate unit operation to apply the recovery release:

    gcloud beta app-lifecycle-manager unit-operations create "recovery-op" \
        --unit="UNIT_ID" \
        --flag-update="projects/PROJECT_ID/locations/LOCATION/flagReleases/recovery-release-1" \
        --location="LOCATION"
    

Replace the following:

  • UNIT_ID: the ID of the unit to recover.
  • LOCATION: the region of the unit.
  • UNIT_KIND_ID: the ID of the parent unit kind.
  • REVISIONS: the comma-separated list of default flag revision resource names returned in step 2.
  • PROJECT_ID: your Google Cloud project ID.