Custom Linux OS image import for VMs reference architecture

This solution reference architecture details a customer-managed, "shift-left" approach to importing non-natively supported Linux operating systems into Google Distributed Cloud (GDC) air-gapped.

In a standard GDC deployment, automated tools attempt to convert uploaded images and inject the gdc-guest-agent. However, for unsupported distributions or OSs with immutable (read-only) filesystems, this automated translation fails, breaking the image import process and preventing network and SSH access. This architecture solves that problem by detailing how to perform offline image surgery to manually configure the OS, bypass the platform's automated translation engine, and optionally create a repeatable "Golden Image" for scalable deployments. This document is intended for Platform Administrators (PA) and Application Operators (AO).

Features and capabilities

  • Offline image surgery: Mount a virtual disk offline to satisfy GDC air-gapped requirements (such as internal NTP synchronization and root password configuration) before uploading the image.
  • Translation bypass (manual mode): Instruct the GDC platform to skip its standard automated configuration and agent-installation processes, preventing import failures on unsupported or immutable operating systems.
  • Direct SSH and serial console access: Replace automated guest agent key injection with serial console access and manual SSH public key injection into the OS.
  • Golden image automation: Capture a customized VM disk and convert it into a standard VirtualMachineImage template, enabling repeatable deployments that process startup scripts.

Architectural principles

  • Workload flexibility and sovereignty: Unblock the migration of legacy, specialized, or high-security workloads by deploying custom Linux distributions.
  • Separation of concerns: Keep OS image preparation separate from platform infrastructure. Shifting configuration to a local workstation avoids corrupting image imports with incompatible automated platform scripts.
  • Resilience and reliability: Air-gapped environments are sensitive to time drift. This architecture configures internal platform NTP servers directly in the image's core configuration so certificates and logging function immediately upon boot.
  • Scalability through golden images: Avoid performing manual image surgery on every deployed VM by using an initial preparation VM to generate a reusable artifact.

Architecture

Custom Linux OS image import architecture diagram.

1. Image preparation (local workstation)

  • Mount Image: Map the .qcow2 virtual disk to a physical block device using qemu-nbd.
  • Chroot Entry: Access the disk's internal filesystem via chroot to perform image changes.
  • Hardcode Configs: Manually set internal NTP servers in /etc/chrony.conf and establish a temporary root password.
  • Verify Drivers: Ensure necessary Virtio drivers are baked into the kernel for disk and network compatibility.
  • Install Agent if needed: Install the GDC guest agent to let GDC manage the guests SSH access.

2. GDC import (manual mode)

  • Translation bypass: Import the image by using the gdcloud CLI with the --os=garden-linux flag.
  • Skip automation: This flag acts as a manual mode switch, instructing the platform to skip automated configuration and guest agent injection that can fail on unsupported distributions.

3. Network and access management

  • IP address and firewall: Deploy the VM and configure VirtualMachineExternalAccess for an ingress IP address and a ProjectNetworkPolicy (PNP) to open SSH port 22.
  • Initial sign-in: Access the VM through the GDC serial console by using the root password set during preparation.
  • Manual SSH setup: Add the administrator's public SSH key to the ~/.ssh/authorized_keys file.

    4. Golden image capture pipeline

  • VM Capture: Customize the running "preparation VM" and shut it down cleanly to ensure data consistency.

  • Template Creation: Convert the boot disk into a standard VirtualMachineImage artifact.

  • Scalability: Use the resulting Golden Image for repeatable, large-scale deployments that natively support platform startup scripts and GDC-managed user/SSH access.

Concepts and technologies

This section breaks down the solution into distinct, functional components required to support custom image imports.

  • Image Preparation Unit (Local Workstation):
    • Responsibilities: Modifies the sparse virtual disk image (.qcow2) before it enters the GDC platform, ensuring all drivers and network requirements are met.
    • Interfaces: Utilizes QEMU utilities. Specifically, qemu-nbd acts as a bridge to map the virtual disk file to a physical block device (e.g., /dev/nbd0). The chroot command is then used to change the root directory, allowing administrators to log into the mounted image and execute configurations (like editing /etc/chrony.conf or setting passwords) exactly as if they were natively logged in.
  • GDC Import Interface:
    • Responsibilities: Safely ingests the modified image without triggering the GDC platform's breaking automation.
    • Interfaces: The gdcloud compute images import CLI tool. The critical component here is the --os=garden-linux flag. Because GDC recognizes "Garden Linux" as an immutable system, this flag acts as a "Manual Mode" switch, instructing the translation engine to skip automated agent installation and image translation entirely.
  • Network and Access Management:
    • Responsibilities: Facilitates secure communication and administrative access to the unsupported OS without reliance on the native gdc-guest-agent.
    • Interfaces: Relies on configuring a VirtualMachineExternalAccess resource to allocate an Ingress IP, and a ProjectNetworkPolicy (PNP) to open port 22 for SSH traffic through the project firewall. Initial access is achieved via the GDC Serial Console using a password set during offline surgery, allowing the administrator to manually paste their public SSH key into the ~/.ssh/authorized_keys file.
  • Golden Image Capture Pipeline:
    • Responsibilities: Converts a manually staged "preparation VM" into a reusable artifact.
    • Data Flow: Prepares the running VM (e.g., ensuring cloud-init is installed to handle KubeVirt cloudInitConfigDrive metadata), cleanly shuts the VM down to ensure disk consistency, and captures the boot disk into a VirtualMachineImageImport resource.

Considerations

  • Scalability and Performance: The initial offline preparation is a manual, highly technical process and is not horizontally scalable. However, converting the first successfully deployed VM into a Golden Image allows the solution to scale infinitely for subsequent deployments across the project.
  • Operational Complexity: This is strictly a customer-managed solution. The operational burden on the Platform Administrator is high during the initial configuration, requiring root privileges on a Linux dev workstation, manual dependency resolution for air-gapped package installations (e.g., wget, cloud-init), and an understanding of disk partition layouts.
  • Security and Compliance: The solution requires creating a temporary root password for emergency Serial Console access. Strict operational hygiene must be maintained to rotate or remove this password once public SSH keys are manually injected, ensuring the system transitions to key-based authentication.
  • Availability and Reliability (NTP): Air-gapped environments require precise time synchronization for core services (like certificates and authentication) to function. The architecture dictates that NTP (e.g., chrony) must be strictly configured to GDC's internal time servers (ntp1.org.internal, etc.) during the offline configuration phase.

Design decision

  • Context & Constraints: GDC air-gapped native import translators do not currently support immutable filesystems (like RHCOS 9) or certain Linux distributions (like SLES 16 or Debian builds), causing image import failures and preventing basic SSH access.
  • Rationale & Trade-offs: The decision was made to utilize a "shift-left" approach using qemu-nbd and chroot combined with the --os=garden-linux CLI flag.
  • Impact: This decision may require operators to manage their own SSH keys, OS patching, and network routing manually, but successfully unblocks the migration of mission-critical BYO operating systems for stringent compliance environments.

Assumptions and limitations

Assumptions

  • Virtio Driver Compatibility: The target operating system kernel must already include the required Virtio drivers (virtio_pci for the base bus, virtio_blk or virtio_scsi for disk access, and virtio_net for networking). If these drivers are not baked into the OS before import, the VM will fail to boot on GDC's underlying KubeVirt hypervisor because it will not be able to see its own hard drive or network card.
  • The administrator performing the import has an existing Linux workstation with qemu-utils installed, sudo privileges, and the ability to mount block devices.

Limitations

  • Guest Agent Functionality: Automated provisioning of SSH keys and users via the GDC control plane is unsupported out-of-the-box because the translation engine is bypassed. Full platform integration is only possible if the customer manually compiles and successfully installs the gdc-guest-agent .deb or .rpm packages during the chroot phase.
  • Specific Filesystems: Immutability layers like ostree (used by RHCOS) require specific bind-mounting logic inside the chroot environment that is not applicable to standard root filesystems like ext4 or Btrfs (used by Debian or SLES).

What's next