本文档提供了在 Google Distributed Cloud (GDC) 网闸隔离环境中部署 Google DeepMind 的开放权重模型(特别是 Gemma 4 系列)的分步说明。其中涵盖了使用 vLLM 实现高吞吐量服务,以及利用 GDC 平台的功能(包括 Kubernetes、Harbor 和 GPU 资源)。在“模型”部分中选择模型和 GPU 选项;对于所有模型,其他每个步骤都相同。
支持的模型
| 模型 | Hugging Face 仓库 | 参数 | 许可 | GPU 选项 |
|---|---|---|---|---|
| Gemma 4 E2B Instruct | google/gemma-4-E2B-it |
51 亿(20 亿有效) | Gemma 使用条款 | 1 个任意 GPU |
| Gemma 4 E4B Instruct | google/gemma-4-E4B-it |
80 亿(40 亿有效) | Gemma 使用条款 | 1 个任意 GPU |
| Gemma 4 12B Instruct | google/gemma-4-12B-it |
12B(密集) | Gemma 使用条款 | 1 个任意 GPU |
| Gemma 4 26B-A4B Instruct | google/gemma-4-26B-A4B-it |
总参数量 258 亿/活跃参数量 40 亿(混合专家模型) | Gemma 使用条款 | 1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB 或 B300 288 GB |
| Gemma 4 31B Instruct | google/gemma-4-31B-it |
31.3B(稠密) | Gemma 使用条款 | 1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB 或 B300 288 GB |
架构
该解决方案涉及在标准集群中将容器化服务后端 (vLLM) 作为 Deployment 进行部署。模型权重存储在 PersistentVolume 上,这些权重是从本地下载的文件中填充的,并使用加载器 Pod 上传。LoadBalancer 类型的 Kubernetes Service 会公开后端 API,而 NetworkPolicy/ProjectNetworkPolicy 对象会保护对该 API 的访问。
准备工作
在工作站上完成使用 vLLM 参考实现部署开放权重模型,并确保您有权访问 ${HOME}/gdcag-solutions 目录。
Identity and Access Management
确保已正确配置必要的 IAM 账号、角色和权限。
组织的 GDC 用户角色:
- 可选:Standard Cluster Admin(通过控制台查看集群)
项目的 GDC User 角色:
- Standard Cluster Admin (
standard-cluster-admin)
标准集群上的 GDC 用户角色:
StandardClusterRoleBinding到StandardClusterRolecluster-admin(由 IAM 项目管理员创建,请参阅参考实现的身份和访问权限管理部分)
工作站
除非某个步骤另有说明,否则请在具有所需工具且可连接到环境和互联网的工作站上运行本指南中的所有命令。
环境配置是分层的:
${HOME}/gdcag-solutions/env.sh会加载平台和注册表设置。${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh会添加 vLLM 设置。本用户指南添加了
env.d/model.sh(模型)和env.d/accelerator.sh(GPU 选项和运行时设置),您可以在“模型”部分中编写这些内容;以及env.d/model-common.sh(从中派生的值)。
创建用户指南目录结构:
mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d创建通用模型环境配置文件。它会从
GDCS_HF_MODEL派生每个路径和资源名称,因此所有模型都不可替换:cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!" # Model (derived from env.d/model.sh) export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models" export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}" export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}" export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model" GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}" export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}" # Workload export GDCS_WORKLOAD_NAMESPACE="vllm-gemma" export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}" export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}" EOF创建用户指南环境加载器文件:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh && echo "Successfully created." || echo "Failed to create!" source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh" export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/gemma" echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}" # Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh" source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh" source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh" echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})" EOF
模型
选择与标准集群的 GPU 节点匹配的模型和 GPU 选项。GDC 网闸隔离配置提供以下 GPU(请参阅集群节点机器类型);加速器资源类型是节点向 Kubernetes 公开 GPU 时使用的名称。
| GPU | 内存 | 机器类型 | 加速器资源类型 |
|---|---|---|---|
| NVIDIA A100 | 40 GB | a2-highgpu-1g-gdc |
nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB |
| NVIDIA A100 | 80 GB | a2-ultragpu-1g-gdc,a2-ultragpu-2g-gdc |
nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE |
| NVIDIA H100 | 94 GB | a3-highgpu-1g-gdc(预览版)、a3-highgpu-2g-gdc(预览版)、a3-highgpu-4g-gdc |
nvidia.com/gpu-pod-NVIDIA_H100_NVL |
| NVIDIA H200 | 141 GB | a3-ultragpu-8g-gdc |
nvidia.com/gpu-pod-NVIDIA_H200 |
| NVIDIA B300 | 288 GB | a4-ultragpu-8g-gdc |
nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC |
每个标签页都会列出模型事实、写入模型环境文件 env.d/model.sh 的命令,以及写入加速器环境文件 env.d/accelerator.sh 的嵌套标签页,其中包含所选 GPU 选项的预填充资源类型和 vLLM 设置。如需稍后切换模型或 GPU 选项,请编写不同的环境文件,再次获取环境,然后重复“实现”部分;多个模型的工作负载可以在同一命名空间中并排运行,因为每个资源名称都派生自模型名称。
为您要部署的模型和 GPU 选项创建模型和加速器环境文件。
Gemma 4 E2B Instruct
属性 值 Hugging Face 代码库 google/gemma-4-E2B-it参数 51 亿(20 亿有效) 许可 Gemma 使用条款 vLLM 使用的权重 Hugging Face 格式 ( model-*.safetensors),约 10.2 GB上下文长度 128k 个词元;本指南适用于 --max-model-len 32768创建模型环境文件:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 E2B Instruct export GDCS_HF_MODEL="google/gemma-4-E2B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF为您的 GPU 选项创建加速器环境文件:
NVIDIA A100 40 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 40 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 E4B Instruct
属性 值 Hugging Face 代码库 google/gemma-4-E4B-it参数 80 亿(40 亿有效) 许可 Gemma 使用条款 vLLM 使用的权重 Hugging Face 格式 ( model-*.safetensors),约 16.0 GB上下文长度 128k 个词元;本指南适用于 --max-model-len 32768创建模型环境文件:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 E4B Instruct export GDCS_HF_MODEL="google/gemma-4-E4B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF为您的 GPU 选项创建加速器环境文件:
NVIDIA A100 40 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 40 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 12B Instruct
属性 值 Hugging Face 代码库 google/gemma-4-12B-it参数 12B(密集) 许可 Gemma 使用条款 vLLM 使用的权重 Hugging Face 格式 ( model-*.safetensors),约 23.9 GB上下文长度 256k tokens; this guide serves --max-model-len 32768创建模型环境文件:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 12B Instruct export GDCS_HF_MODEL="google/gemma-4-12B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF为您的 GPU 选项创建加速器环境文件:
NVIDIA A100 40 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 40 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 26B-A4B Instruct
属性 值 Hugging Face 代码库 google/gemma-4-26B-A4B-it参数 总参数量 258 亿/活跃参数量 40 亿(混合专家模型) 许可 Gemma 使用条款 vLLM 使用的权重 Hugging Face 格式 ( model-*.safetensors),约 51.6 GB上下文长度 256k tokens; this guide serves --max-model-len 32768创建模型环境文件:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 26B-A4B Instruct export GDCS_HF_MODEL="google/gemma-4-26B-A4B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF为您的 GPU 选项创建加速器环境文件:
NVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 31B Instruct
属性 值 Hugging Face 代码库 google/gemma-4-31B-it参数 31.3B(稠密) 许可 Gemma 使用条款 vLLM 使用的权重 Hugging Face 格式 ( model-*.safetensors),约 62.5 GB上下文长度 256k tokens; this guide serves --max-model-len 32768创建模型环境文件:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 31B Instruct export GDCS_HF_MODEL="google/gemma-4-31B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF为您的 GPU 选项创建加速器环境文件:
NVIDIA A100 80 GB
在 80 GB GPU 上提供服务时,KV 缓存几乎没有可用内存,因此此选项会设置
--max-model-len 16384。cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="16384" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOF获取环境文件:
source ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh输出类似于以下内容:
GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/gemma GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)
集群
配置
gdcloud:gdcloud config set core/account "default-user" gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}" gdcloud config set core/project "${GDC_PROJECT}" gdcloud config set core/zone "${GDC_ZONE}"向 GDC 环境进行身份验证:
gdcloud auth login检索集群凭据:
gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \ --project="${GDC_PROJECT}" \ --standard \ --zone="${GDC_ZONE}"验证与集群的连接:
kubectl get nodes -L node.cluster.private.gdc.goog/machine-class验证集群中是否提供了加速器环境文件的加速器资源类型。该命令会列出节点公开的加速器资源类型(名称取决于安装的 GPU 型号;在没有 GPU 节点的集群上,该名称为空)和配置的值:
kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]' echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
实现
命名空间
创建此命名空间:
kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \ --dry-run=client \ --output=yaml | kubectl apply -f -添加
imagePullSecret:kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \ --dry-run=client \ --from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ --output=yaml | kubectl apply -f -
模型权重
下载模型权重(仅下载 vLLM 所需的所选模型文件,
GDCS_HF_DOWNLOAD_EXCLUDE会跳过其余文件),并移除.cache目录:EXCLUDE_ARGS=() if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}" EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}") fi hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \ rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"根据下载的模型权重计算所需的
PersistentVolumeClaim大小,并加上 25% 的开销(向上取整到下一个整数 GiB):MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}') export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi" echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"为模型权重创建 ReadWriteOnce PersistentVolumeClaim 的清单:
cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ${GDCS_WORKLOAD_NAME}-rwo spec: accessModes: - ReadWriteOnce resources: requests: storage: ${GDCS_MODEL_PVC_SIZE} storageClassName: standard-rwo volumeMode: Filesystem EOF为模型权重应用 PersistentVolumeClaim 的清单:
kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"等待
PersistentVolumeClaim绑定并输出基础架构运维人员调整 I/O 服务质量 (QoS) 政策所需的值:kubectl wait \ --for=jsonpath='{.status.phase}'=Bound \ "pvc/${GDCS_WORKLOAD_NAME}-rwo" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ --timeout=120s BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ --output=jsonpath='{.spec.volumeName}') echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"如果您的集群上没有
performance-rwo,请让基础架构运维人员将高性能 QoS 政策(system-performance-<ORG>-admin,每 GiB 30 IOPS,最低 6,148 IOPS)应用于支持PersistentVolumeClaim的卷。提供上一步输出的BACKING_PVC和ORG值。基础设施运维人员按照基础设施运维人员文档中的 FILE-R0097 Runbook 所述应用该政策;该政策会对实时卷生效,因此您可以继续执行后续步骤。为模型加载器 Pod 创建清单:
cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" apiVersion: v1 kind: Pod metadata: name: ${GDCS_MODEL_LOADER_POD_NAME} spec: containers: - name: uploader image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG} command: ["sleep", "infinity"] volumeMounts: - name: model-data mountPath: ${GDCS_HF_PV_MOUNT_PATH} imagePullSecrets: - name: ${GDCS_HARBOR_K8S_PULL_SECRET} volumes: - name: model-data persistentVolumeClaim: claimName: ${GDCS_WORKLOAD_NAME}-rwo EOF应用模型加载器 Pod 的清单:
kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"等待模型加载器 Pod 变为“Running”(正在运行)状态:
watch --color --interval 5 --no-title \ "kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \ --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"将模型权重上传到
PersistentVolume(该脚本会跳过已存在且大小相同的文件,因此可以重新运行):export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}" export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}" export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}" ${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh删除模型加载器 Pod:
kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"
推理服务器
为 vLLM Deployment 创建清单。ConfigMap 包含来自环境的特定于模型的 vLLM 设置,因此每个模型和 GPU 选项的清单都相同:
cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" apiVersion: v1 kind: ConfigMap metadata: name: ${GDCS_WORKLOAD_NAME} data: MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}" MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}" PORT: "8000" TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}" VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}" --- apiVersion: apps/v1 kind: Deployment metadata: name: ${GDCS_WORKLOAD_NAME} labels: app: ${GDCS_WORKLOAD_NAME} spec: replicas: 1 selector: matchLabels: app: ${GDCS_WORKLOAD_NAME} strategy: type: Recreate template: metadata: labels: app: ${GDCS_WORKLOAD_NAME} spec: containers: - command: - /bin/sh - -c args: - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS envFrom: - configMapRef: name: ${GDCS_WORKLOAD_NAME} image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips livenessProbe: failureThreshold: 3 httpGet: path: /health port: 8000 periodSeconds: 20 timeoutSeconds: 5 name: vllm ports: - containerPort: 8000 readinessProbe: failureThreshold: 3 httpGet: path: /health port: 8000 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: limits: ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}" requests: ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}" startupProbe: httpGet: path: /health port: 8000 failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD} periodSeconds: 10 timeoutSeconds: 5 volumeMounts: - name: model-weights mountPath: ${GDCS_HF_PV_MOUNT_PATH} - name: shm mountPath: /dev/shm securityContext: seLinuxOptions: type: spc_t imagePullSecrets: - name: ${GDCS_HARBOR_K8S_PULL_SECRET} volumes: - name: model-weights persistentVolumeClaim: claimName: ${GDCS_WORKLOAD_NAME}-rwo - name: shm emptyDir: medium: Memory sizeLimit: "8Gi" --- apiVersion: v1 kind: Service metadata: name: ${GDCS_WORKLOAD_NAME} spec: ports: - name: vllm-http port: 80 protocol: TCP targetPort: 8000 selector: app: ${GDCS_WORKLOAD_NAME} type: LoadBalancer EOF应用 vLLM Deployment 的清单:
kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"为 vLLM
NetworkPolicy创建清单:cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!" apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-vllm-ingress namespace: ${GDCS_WORKLOAD_NAMESPACE} spec: ingress: - from: - ipBlock: cidr: 0.0.0.0/0 - namespaceSelector: {} ports: - port: 8000 protocol: TCP podSelector: {} policyTypes: - Ingress EOF应用 vLLM
NetworkPolicy的清单:kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"
等待 Pod 变为“Running”(运行中)状态:
watch --color --interval 5 --no-title \ "kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \ --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"等待 Deployment 准备就绪:
watch --color --interval 5 --no-title \ "kubectl get deployment/${GDCS_WORKLOAD_NAME} \ --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1 1 1'"可选:流式传输容器日志以监控启动进度:
kubectl logs deployment/${GDCS_WORKLOAD_NAME} \ --follow \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"
验证
检索 vLLM 的 LoadBalancer IP 地址:
export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}') echo "VLLM_LB_IP=${VLLM_LB_IP}"向 vLLM 发送测试请求。
max_tokens值为env.d/accelerator.sh中的GDCS_VLLM_TEST_MAX_TOKENS:curl http://${VLLM_LB_IP}/v1/models curl http://${VLLM_LB_IP}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'", "messages": [ {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."} ], "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}' }'如果无法从工作站访问 LoadBalancer IP 地址(例如,因为尚无
ProjectNetworkPolicy允许访问),请通过端口转发发送请求:kubectl port-forward service/${GDCS_WORKLOAD_NAME} \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 & PF_PID=$! sleep 1 curl http://127.0.0.1:8000/v1/models kill ${PF_PID}