Debian 11 OS image import in GDC air-gapped

This solution reference implementation outlines a customer-managed, shift-left strategy to manually import a Debian 11 operating system into Google Distributed Cloud (GDC) air-gapped. Standard automated tools fail on unsupported OSes because the automated image translation and agent injection processes break.

This technical guide solves the issue by performing an offline image surgery on a local workstation using QEMU utilities. By modifying the .qcow2 virtual disk, configuring required settings like NTP, optionally installing the GDC guest agent and cloud-init inside the chroot environment, and using the --os=garden-linux flag during import, the platform's native translation engine is completely bypassed. Finally, the configured VM can be captured as a reusable Golden Image (VirtualMachineImage) to ensure scalable deployments with native startup script support.

Architecture

Debian 11 OS image import architecture diagram.

  • Image Preparation Unit: Offline modification of the .qcow2 disk via qemu-nbd and chroot on a local workstation.
  • GDC Import Interface: Using the gdcloud CLI with the --os=garden-linux flag to safely ingest the image into GDC by bypassing automated processes.
  • Network & Access Management: Utilizing the GDC Serial Console for initial VM access, allowing the setup of the GDC guest agent and SSH keys.
  • Golden Image Capture Pipeline (Optional): Converting the custom-staged preparation VM into a VirtualMachineImage artifact to provide a scalable foundation for future deployments.

Before you begin

Ensure all prerequisites are met before beginning the offline image surgery:

  1. Project roles: To get the permissions that you need to manage VMs and VM images, ask your Project IAM Admin to grant you the following roles in your project namespace:
    • Virtual Machine Project Admin (project-vm-admin)
    • Project Viewer (project-viewer)
    • Virtual Machine Image Project Admin (project-vm-image-admin)
  2. Workstation requirements: A Linux development workstation connected to the GDC environment network with the qemu-utils package installed.
  3. OS compatibility (Virtio drivers): The base image kernel must include Virtio drivers (virtio-net and virtio-scsi supporting virtio-spec 1.0, plus virtio_pci and virtio_blk). Without these drivers, the VM fails to recognize its network interface and disk on the underlying KubeVirt hypervisor.
  4. Base image: A downloaded Debian 11 generic .qcow2 cloud image, such as debian-11-generic-amd64.qcow2.

Procedure

Optional: Obtain the GDC agent from an Ubuntu VM

This step is needed only if GDC-managed SSH is required or wanted. If SSH access is managed manually by the customer, this step can be skipped.

Create a VM in the project you are using (YOUR_PROJECT_NAME) with an Ubuntu 20.04 image, and follow Connect to a Linux VM to connect to it. Once in it

sudo apt-get update

# this will show the agent version and metadata
apt-cache show gdc-guest-agent

The output is similar to

Package: gdc-guest-agent
Architecture: amd64
Maintainer: gdch-vmm-eng@google.com
Priority: optional
Section: misc
Filename: pool/gdch/g/gdc-guest-agent/gdc-guest-agent_1.15.0.21_amd64.deb
Size: 42353128
SHA256: a214b2832a5a756fde941bd0d36a10ef8b760830ad9b60952a1a60f32580f1ad
SHA1: c484c3d8bc7f8416c1085a2811ca8fd8d87280dd
MD5sum: f905dcb2d7c9c99254998fa15ddeb62e
Description: Google Distributed Cloud Guest Agent contains the guest
 agent binaries.
Description-md5:
Homepage: https://cloud.google.com/distributed-cloud

Then set the version (e.g. 1.15.0.21) and download the package.

# set the version from metadata above
apt download gdc-guest-agent=$AGENT_VERSION

As per the example above, the download should be similar to

apt download gdc-guest-agent=1.15.0.21

The output will be a package named gdc-guest-agent_VERSION_amd64.deb and this file should be transferred to the local workstation, for example by following Transfer Files from VM.

At this stage you should have on your local machine the downloaded Debian 11 base image in .qcow2 format and the GDC guest agent debian package.

Mount the downloaded image locally

To locally mount the image, use the qemu-nbd utility, which maps the virtual file into a physical block device path.

# Load the network block device kernel module
sudo modprobe nbd

# Connect the Debian 11 image to the block device
sudo qemu-nbd -c /dev/nbd0 debian-11-generic-amd64.qcow2

# Identify the correct partition (usually the largest ext4 partition,
# like /dev/nbd0p1)
sudo parted /dev/nbd0 print

# Create a mount point and mount the identified partition
sudo mkdir -p /mnt/debian
sudo mount /dev/nbd0p1 /mnt/debian

# [OPTIONAL] Move the GDC agent deb package to the mount path
sudo cp gdc-guest-agent_VERSION_amd64.deb /mnt/debian/tmp/

# Enter the chroot environment to act as the root user inside the image
sudo chroot /mnt/debian

After this command you are acting as root in the image and be able to make all the checks and needed changes.

Check dependencies (Virtio, chrony, cloud-init)

Verify that Virtio drivers are present in the image

INITRDFILE=$(ls /boot/initrd.img*)
lsinitramfs $INITRDFILE | grep virtio

The output should be similar to the following:

usr/lib/modules/5.10.0-39-amd64/kernel/drivers/block/virtio_blk.ko
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/net/virtio_net.ko
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/scsi/virtio_scsi.ko
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/virtio
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/virtio/virtio.ko
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/virtio/virtio_mmio.ko
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/virtio/virtio_pci.ko
usr/lib/modules/5.10.0-39-amd64/kernel/drivers/virtio/virtio_ring.ko

Then, check if chrony and cloud-init are installed and configured as expected

dpkg -l | grep chrony

# output like
## ii  chrony  4.0-8+deb11u2  amd64  Versatile implementation of the Network Time Protocol

dpkg -l | grep cloud-init

# output like
## ii  cloud-init                20.4.1-2+deb11u1  all  initialization system for infrastructure cloud instances
## ii  cloud-initramfs-growroot  0.18.debian8      all  automatically resize the root partition on first boot

Ensure that the nocloud datasource is not disabled. You can run the following command within the chroot to ensure the nocloud datasource is enabled:

cat <<EOF | sudo tee /etc/cloud/cloud.cfg.d/99_nocloud.cfg
datasource_list: [ NoCloud, ConfigDrive, None ]
EOF

Configure NTP, serial access and SSH

Configure the NTP server addresses with chrony (on Debian, the configuration file is located at /etc/chrony/chrony.conf):

cat <<EOF > /etc/chrony/chrony.conf
server ntp1.org.internal iburst
server ntp2.org.internal iburst
server ntp3.org.internal iburst
server ntp4.org.internal iburst
server ntp5.org.internal iburst
server ntp6.org.internal iburst
EOF

Setup the initial serial console access

echo "root:gdchpassword" | chpasswd

Optional: Install the GDC agent package:

apt-get install -y /tmp/gdc-guest-agent_VERSION_amd64.deb
rm /tmp/gdc-guest-agent_VERSION_amd64.deb

Optional: Manual SSH setup

If SSH is managed by the customer and the GDC agent is not installed, configure initial root SSH access as follows:

mkdir -p /root/.ssh
chmod 700 /root/.ssh
echo "[YOUR_PUBLIC_KEY]" > /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

This step can be done also from serial console when the VM is running for the first time.

Unmount and import the image

After the offline modifications are complete, safely detach the virtual disk. Import the image to GDC using the --os=garden-linux flag. This flag is mandatory; it prevents the GDC translation engine from attempting automated configuration that causes unsupported OSes to fail.

First, exit from chroot

exit

Then ensure that all changes are written to the disk and unmount.

sudo systemctl --root=/mnt/debian enable guest-agent.service
sync
sudo umount -l /mnt/debian
sudo qemu-nbd -d /dev/nbd0
sync

Last, log into GDC and import the custom image with the garden linux os flag.

gdcloud auth login

gdcloud compute images import debian-11-custom \
  --project YOUR_PROJECT_NAME \
  --source-file=./debian-11-generic-amd64.qcow2 \
  --os=garden-linux \
  --timeout=60m

Make sure to have all needed roles as in prerequisites.

This operation may take quite some time, depending on network upload speed, and image size.

Launch the VM with the custom image and connect

At this point you are already able to deploy a VM using the imported image. This can be done via UI or using gdcloud

gdcloud compute instances create debian-11-vm \
  --project YOUR_PROJECT_NAME \
  --image debian-11-custom \
  --boot-disk-size 20GB \
  --machine-type n3-standard-2-gdc \
  --enable-external-access

Wait until the VM gets into running state, then you can:

  • Access Serial Console, in case you setup the credentials
  • SSH using customer-managed keys, in case you set them up
  • SSH using GDC-managed guest, in case GDC agent has been installed

Serial console access

gdcloud compute connect-to-serial-port debian-11-vm \
  --project YOUR_PROJECT_NAME

When prompted, insert the credentials set in the chroot configuration.

Customer-managed SSH

ssh -i my_key root@VM_IP

This will work with the root user: if you need more users to access, create users in the VM and upload their public keys as authorized keys.

GDC-managed SSH

gdcloud compute ssh debian-11-vm --project YOUR_PROJECT_NAME

This will work only if the GDC agent is properly configured and active, and the login user will be the same used when authenticating in GDC.

What you have now

At this stage, you have a VM image with the desired OS.

If you configured the cloud-init and installed the GDC agent, you can benefit from GDC-managed SSH and startup scripts. If you choose not to use GDC agent, you can manage the SSH using the serial console access or the root access.

However, a manual offline setup is not horizontally scalable. Since this base VM may be insufficient for scaling purposes as you may need to onboard the same users on multiple VMs, or have the same packages installed on multiple VMs, or may want to reduce the SSH keys management, or similar common use cases, it may be a good idea to install all you need in the created VM, and configure it properly once, and then use it as a model for the following VMs with same OS.

If this is the goal, then follow the next section to create a Golden Image.

Optional: Customize and create a golden image

Now, you can customize the VM as you prefer, based on your requirements. Connect to the VM, and transfer the needed deb packages into it.

Once the VM has all the necessary configuration and packages, use it to create a new VM image to be used afterwards.

First, stop the VM: perform a clean shutdown to ensure complete disk consistency. You can do this from the VM terminal (sudo shutdown -h now) or using the GDC console.

Then, create the custom Golden Image using the GDC console or the Virtual Machine Manager API by capturing the stopped VM's boot disk (debian-11-vm-boot-disk) into a new VirtualMachineImage:

kubectl --kubeconfig MANAGEMENT_API_SERVER \
  -n YOUR_PROJECT_NAME \
  apply -f - <<EOF
apiVersion: virtualmachine.gdc.goog/v1
kind: VirtualMachineImageImport
metadata:
  name: debian-11-golden-import
  namespace: YOUR_PROJECT_NAME
spec:
  source:
    diskRef:
      name: debian-11-vm-boot-disk
  imageMetadata:
    name: debian-11-golden
    operatingSystem: garden-linux
    minimumDiskSize: 20G
EOF

This creates a new VirtualMachineImage named debian-11-golden to use when creating new VM instances. For example:

gdcloud compute instances create my-final-workload-vm \
  --project YOUR_PROJECT_NAME \
  --image debian-11-golden \
  --boot-disk-size 20GB \
  --machine-type n3-standard-2-gdc \
  --enable-external-access