GDC 网闸隔离配置上的 Gemma 模型与 vLLM 用户指南

本文档提供了在 Google Distributed Cloud (GDC) 网闸隔离环境中部署 Google DeepMind 的开放权重模型(特别是 Gemma 4 系列)的分步说明。其中涵盖了使用 vLLM 实现高吞吐量服务,以及利用 GDC 平台的功能(包括 Kubernetes、Harbor 和 GPU 资源)。在“模型”部分中选择模型和 GPU 选项;对于所有模型,其他每个步骤都相同。

支持的模型

模型 Hugging Face 仓库 参数 许可 GPU 选项
Gemma 4 E2B Instruct google/gemma-4-E2B-it 51 亿(20 亿有效) Gemma 使用条款 1 个任意 GPU
Gemma 4 E4B Instruct google/gemma-4-E4B-it 80 亿(40 亿有效) Gemma 使用条款 1 个任意 GPU
Gemma 4 12B Instruct google/gemma-4-12B-it 12B(密集) Gemma 使用条款 1 个任意 GPU
Gemma 4 26B-A4B Instruct google/gemma-4-26B-A4B-it 总参数量 258 亿/活跃参数量 40 亿(混合专家模型) Gemma 使用条款 1 x NVIDIA A100 80 GB
1 x NVIDIA H100 94 GB
1 x NVIDIA H200 141 GB 或 B300 288 GB
Gemma 4 31B Instruct google/gemma-4-31B-it 31.3B(稠密) Gemma 使用条款 1 x NVIDIA A100 80 GB
1 x NVIDIA H100 94 GB
1 x NVIDIA H200 141 GB 或 B300 288 GB

架构

该解决方案涉及在标准集群中将容器化服务后端 (vLLM) 作为 Deployment 进行部署。模型权重存储在 PersistentVolume 上,这些权重是从本地下载的文件中填充的,并使用加载器 Pod 上传。LoadBalancer 类型的 Kubernetes Service 会公开后端 API,而 NetworkPolicy/ProjectNetworkPolicy 对象会保护对该 API 的访问。

准备工作

在工作站上完成使用 vLLM 参考实现部署开放权重模型,并确保您有权访问 ${HOME}/gdcag-solutions 目录。

Identity and Access Management

确保已正确配置必要的 IAM 账号、角色和权限。

组织的 GDC 用户角色:

  • 可选:Standard Cluster Admin(通过控制台查看集群)

项目的 GDC User 角色:

  • Standard Cluster Admin (standard-cluster-admin)

标准集群上的 GDC 用户角色:

  • StandardClusterRoleBinding 到 StandardClusterRole cluster-admin(由 IAM 项目管理员创建,请参阅参考实现的身份和访问权限管理部分)

工作站

除非某个步骤另有说明,否则请在具有所需工具且可连接到环境和互联网的工作站上运行本指南中的所有命令。

环境配置是分层的:

  • ${HOME}/gdcag-solutions/env.sh 会加载平台和注册表设置。
  • ${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh 会添加 vLLM 设置。

  • 本用户指南添加了 env.d/model.sh(模型)和 env.d/accelerator.sh(GPU 选项和运行时设置),您可以在“模型”部分中编写这些内容;以及 env.d/model-common.sh(从中派生的值)。

  1. 创建用户指南目录结构:

    mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d
    
  2. 创建通用模型环境配置文件。它会从 GDCS_HF_MODEL 派生每个路径和资源名称,因此所有模型都不可替换:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!"
    # Model (derived from env.d/model.sh)
    export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models"
    export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}"
    export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
    export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model"
    
    GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}"
    export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}"
    
    # Workload
    export GDCS_WORKLOAD_NAMESPACE="vllm-gemma"
    export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}"
    export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}"
    EOF
    
  3. 创建用户指南环境加载器文件:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh && echo "Successfully created." || echo "Failed to create!"
    source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh"
    
    export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/gemma"
    echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}"
    
    # Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them
    source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh"
    source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh"
    source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh"
    echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})"
    EOF
    

模型

选择与标准集群的 GPU 节点匹配的模型和 GPU 选项。GDC 网闸隔离配置提供以下 GPU(请参阅集群节点机器类型);加速器资源类型是节点向 Kubernetes 公开 GPU 时使用的名称。

GPU 内存 机器类型 加速器资源类型
NVIDIA A100 40 GB a2-highgpu-1g-gdc nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB
NVIDIA A100 80 GB a2-ultragpu-1g-gdc,a2-ultragpu-2g-gdc nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE
NVIDIA H100 94 GB a3-highgpu-1g-gdc(预览版)、a3-highgpu-2g-gdc(预览版)、a3-highgpu-4g-gdc nvidia.com/gpu-pod-NVIDIA_H100_NVL
NVIDIA H200 141 GB a3-ultragpu-8g-gdc nvidia.com/gpu-pod-NVIDIA_H200
NVIDIA B300 288 GB a4-ultragpu-8g-gdc nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC

每个标签页都会列出模型事实、写入模型环境文件 env.d/model.sh 的命令,以及写入加速器环境文件 env.d/accelerator.sh 的嵌套标签页,其中包含所选 GPU 选项的预填充资源类型和 vLLM 设置。如需稍后切换模型或 GPU 选项,请编写不同的环境文件,再次获取环境,然后重复“实现”部分;多个模型的工作负载可以在同一命名空间中并排运行,因为每个资源名称都派生自模型名称。

  1. 为您要部署的模型和 GPU 选项创建模型和加速器环境文件。

    Gemma 4 E2B Instruct

    属性 值
    Hugging Face 代码库 google/gemma-4-E2B-it
    参数 51 亿(20 亿有效)
    许可 Gemma 使用条款
    vLLM 使用的权重 Hugging Face 格式 (model-*.safetensors),约 10.2 GB
    上下文长度 128k 个词元;本指南适用于 --max-model-len 32768

    创建模型环境文件:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 E2B Instruct
    export GDCS_HF_MODEL="google/gemma-4-E2B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    为您的 GPU 选项创建加速器环境文件:

    NVIDIA A100 40 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 40 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 E4B Instruct

    属性 值
    Hugging Face 代码库 google/gemma-4-E4B-it
    参数 80 亿(40 亿有效)
    许可 Gemma 使用条款
    vLLM 使用的权重 Hugging Face 格式 (model-*.safetensors),约 16.0 GB
    上下文长度 128k 个词元;本指南适用于 --max-model-len 32768

    创建模型环境文件:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 E4B Instruct
    export GDCS_HF_MODEL="google/gemma-4-E4B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    为您的 GPU 选项创建加速器环境文件:

    NVIDIA A100 40 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 40 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 12B Instruct

    属性 值
    Hugging Face 代码库 google/gemma-4-12B-it
    参数 12B(密集)
    许可 Gemma 使用条款
    vLLM 使用的权重 Hugging Face 格式 (model-*.safetensors),约 23.9 GB
    上下文长度 256k tokens; this guide serves --max-model-len 32768

    创建模型环境文件:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 12B Instruct
    export GDCS_HF_MODEL="google/gemma-4-12B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    为您的 GPU 选项创建加速器环境文件:

    NVIDIA A100 40 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 40 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 26B-A4B Instruct

    属性 值
    Hugging Face 代码库 google/gemma-4-26B-A4B-it
    参数 总参数量 258 亿/活跃参数量 40 亿(混合专家模型)
    许可 Gemma 使用条款
    vLLM 使用的权重 Hugging Face 格式 (model-*.safetensors),约 51.6 GB
    上下文长度 256k tokens; this guide serves --max-model-len 32768

    创建模型环境文件:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 26B-A4B Instruct
    export GDCS_HF_MODEL="google/gemma-4-26B-A4B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    为您的 GPU 选项创建加速器环境文件:

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 31B Instruct

    属性 值
    Hugging Face 代码库 google/gemma-4-31B-it
    参数 31.3B(稠密)
    许可 Gemma 使用条款
    vLLM 使用的权重 Hugging Face 格式 (model-*.safetensors),约 62.5 GB
    上下文长度 256k tokens; this guide serves --max-model-len 32768

    创建模型环境文件:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 31B Instruct
    export GDCS_HF_MODEL="google/gemma-4-31B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    为您的 GPU 选项创建加速器环境文件:

    NVIDIA A100 80 GB

    在 80 GB GPU 上提供服务时,KV 缓存几乎没有可用内存,因此此选项会设置 --max-model-len 16384。

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="16384"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    
  2. 获取环境文件:

    source ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh
    

    输出类似于以下内容:

    GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions
    GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm
    GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/gemma
    GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)
    

集群

  1. 配置 gdcloud:

    gdcloud config set core/account "default-user"
    gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}"
    gdcloud config set core/project "${GDC_PROJECT}"
    gdcloud config set core/zone "${GDC_ZONE}"
    
  2. 向 GDC 环境进行身份验证:

    gdcloud auth login
    
  3. 检索集群凭据:

    gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \
    --project="${GDC_PROJECT}" \
    --standard \
    --zone="${GDC_ZONE}"
    
  4. 验证与集群的连接:

    kubectl get nodes -L node.cluster.private.gdc.goog/machine-class
    
  5. 验证集群中是否提供了加速器环境文件的加速器资源类型。该命令会列出节点公开的加速器资源类型(名称取决于安装的 GPU 型号;在没有 GPU 节点的集群上,该名称为空)和配置的值:

    kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]'
    echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
    

实现

命名空间

  1. 创建此命名空间:

    kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \
    --dry-run=client \
    --output=yaml | kubectl apply -f -
    
  2. 添加 imagePullSecret:

    kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \
    --dry-run=client \
    --from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
    --output=yaml | kubectl apply -f -
    

模型权重

  1. 下载模型权重(仅下载 vLLM 所需的所选模型文件,GDCS_HF_DOWNLOAD_EXCLUDE 会跳过其余文件),并移除 .cache 目录:

    EXCLUDE_ARGS=()
    if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then
      read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}"
      EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}")
    fi
    hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \
    rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"
    
  2. 根据下载的模型权重计算所需的 PersistentVolumeClaim 大小,并加上 25% 的开销(向上取整到下一个整数 GiB):

    MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}')
    export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi"
    echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"
    
  3. 为模型权重创建 ReadWriteOnce PersistentVolumeClaim 的清单:

    cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml"
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: ${GDCS_WORKLOAD_NAME}-rwo
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: ${GDCS_MODEL_PVC_SIZE}
      storageClassName: standard-rwo
      volumeMode: Filesystem
    EOF
    
  4. 为模型权重应用 PersistentVolumeClaim 的清单:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    
  5. 等待 PersistentVolumeClaim 绑定并输出基础架构运维人员调整 I/O 服务质量 (QoS) 政策所需的值:

    kubectl wait \
    --for=jsonpath='{.status.phase}'=Bound \
    "pvc/${GDCS_WORKLOAD_NAME}-rwo" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
    --timeout=120s
    BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
    --output=jsonpath='{.spec.volumeName}')
    echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"
    
  6. 如果您的集群上没有 performance-rwo,请让基础架构运维人员将高性能 QoS 政策(system-performance-<ORG>-admin,每 GiB 30 IOPS,最低 6,148 IOPS)应用于支持 PersistentVolumeClaim 的卷。提供上一步输出的 BACKING_PVC 和 ORG 值。基础设施运维人员按照基础设施运维人员文档中的 FILE-R0097 Runbook 所述应用该政策;该政策会对实时卷生效,因此您可以继续执行后续步骤。

  7. 为模型加载器 Pod 创建清单:

    cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml"
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${GDCS_MODEL_LOADER_POD_NAME}
    spec:
      containers:
        - name: uploader
          image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG}
          command: ["sleep", "infinity"]
          volumeMounts:
            - name: model-data
              mountPath: ${GDCS_HF_PV_MOUNT_PATH}
      imagePullSecrets:
        - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
      volumes:
        - name: model-data
          persistentVolumeClaim:
            claimName: ${GDCS_WORKLOAD_NAME}-rwo
    EOF
    
  8. 应用模型加载器 Pod 的清单:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    
  9. 等待模型加载器 Pod 变为“Running”(正在运行)状态:

    watch --color --interval 5 --no-title \
    "kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \
    --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"
    
  10. 将模型权重上传到 PersistentVolume(该脚本会跳过已存在且大小相同的文件,因此可以重新运行):

    export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}"
    export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
    export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}"
    
    ${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh
    
  11. 删除模型加载器 Pod:

    kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    

推理服务器

  1. 为 vLLM Deployment 创建清单。ConfigMap 包含来自环境的特定于模型的 vLLM 设置,因此每个模型和 GPU 选项的清单都相同:

    cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml"
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ${GDCS_WORKLOAD_NAME}
    data:
      MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}"
      MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
      PORT: "8000"
      TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
      VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}"
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ${GDCS_WORKLOAD_NAME}
      labels:
        app: ${GDCS_WORKLOAD_NAME}
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: ${GDCS_WORKLOAD_NAME}
      strategy:
        type: Recreate
      template:
        metadata:
          labels:
            app: ${GDCS_WORKLOAD_NAME}
        spec:
          containers:
            - command:
                - /bin/sh
                - -c
              args:
                - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS
              envFrom:
                - configMapRef:
                    name: ${GDCS_WORKLOAD_NAME}
              image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips
              livenessProbe:
                failureThreshold: 3
                httpGet:
                  path: /health
                  port: 8000
                periodSeconds: 20
                timeoutSeconds: 5
              name: vllm
              ports:
                - containerPort: 8000
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /health
                  port: 8000
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 5
              resources:
                limits:
                  ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
                requests:
                  ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
              startupProbe:
                httpGet:
                  path: /health
                  port: 8000
                failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD}
                periodSeconds: 10
                timeoutSeconds: 5
              volumeMounts:
                - name: model-weights
                  mountPath: ${GDCS_HF_PV_MOUNT_PATH}
                - name: shm
                  mountPath: /dev/shm
          securityContext:
            seLinuxOptions:
              type: spc_t
          imagePullSecrets:
            - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
          volumes:
            - name: model-weights
              persistentVolumeClaim:
                claimName: ${GDCS_WORKLOAD_NAME}-rwo
            - name: shm
              emptyDir:
                medium: Memory
                sizeLimit: "8Gi"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: ${GDCS_WORKLOAD_NAME}
    spec:
      ports:
        - name: vllm-http
          port: 80
          protocol: TCP
          targetPort: 8000
      selector:
        app: ${GDCS_WORKLOAD_NAME}
      type: LoadBalancer
    EOF
    
  2. 应用 vLLM Deployment 的清单:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    
  3. 为 vLLM NetworkPolicy 创建清单:

    cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!"
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-vllm-ingress
      namespace: ${GDCS_WORKLOAD_NAMESPACE}
    spec:
      ingress:
      - from:
        - ipBlock:
            cidr: 0.0.0.0/0
        - namespaceSelector: {}
        ports:
        - port: 8000
          protocol: TCP
      podSelector: {}
      policyTypes:
      - Ingress
    EOF
    
  4. 应用 vLLM NetworkPolicy 的清单:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"
    
  1. 等待 Pod 变为“Running”(运行中)状态:

    watch --color --interval 5 --no-title \
    "kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \
    --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"
    
  2. 等待 Deployment 准备就绪:

    watch --color --interval 5 --no-title \
    "kubectl get deployment/${GDCS_WORKLOAD_NAME} \
    --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1     1            1'"
    
  3. 可选:流式传输容器日志以监控启动进度:

    kubectl logs deployment/${GDCS_WORKLOAD_NAME} \
    --follow \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    

验证

  1. 检索 vLLM 的 LoadBalancer IP 地址:

    export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}')
    echo "VLLM_LB_IP=${VLLM_LB_IP}"
    
  2. 向 vLLM 发送测试请求。max_tokens 值为 env.d/accelerator.sh 中的 GDCS_VLLM_TEST_MAX_TOKENS:

    curl http://${VLLM_LB_IP}/v1/models
    
    curl http://${VLLM_LB_IP}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'",
      "messages": [
        {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."}
      ],
      "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}'
    }'
    
  3. 如果无法从工作站访问 LoadBalancer IP 地址(例如,因为尚无 ProjectNetworkPolicy 允许访问),请通过端口转发发送请求:

    kubectl port-forward service/${GDCS_WORKLOAD_NAME} \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 &
    PF_PID=$!
    
    sleep 1
    
    curl http://127.0.0.1:8000/v1/models
    
    kill ${PF_PID}