Run and profile vLLM workloads with ML Diagnostics
vLLM is an open-source library for large language model (LLM) inference and serving. Integrating ML Diagnostics with vLLM lets you track serving configurations, monitor workload telemetry, and capture XProf profiles on Google Cloud. This document describes the ML Diagnostics wrapper command for vLLM and shows you how to containerize, run, and profile a vLLM workload.
Wrapper command-line arguments
ML Diagnostics provides a built-in wrapper command,
google_cloud_mldiagnostics.commands.run_vllm, to launch vLLM serving and
inference workloads and automatically configure telemetry and profiling. You can
set the run_vllm wrapper as the entrypoint in your vLLM Docker container or
run it directly from the command line on
Google Kubernetes Engine (GKE) or
Compute Engine.
When you start a workload with the run_vllm command, the wrapper performs the
following actions:
- Register the run: Creates and registers a managed
MLRunresource configured for vLLM serving workloads. - Configure profiling: Sets profiler environment variables and updates profiler output paths in Cloud Storage to organize traces in a directory for your ML run.
- Forward arguments: Passes all remaining command-line arguments directly
to the
vllmcommand.
The following table describes the command-line arguments that the run_vllm
wrapper accepts:
| Argument | Description |
|---|---|
--mlrun_name |
Required. Unique identifier for the ML run (MLRun) resource. |
--mlrun_gcs_path |
Required. Cloud Storage bucket path that stores profiler traces
and artifacts. For example,
gs://my-bucket/vllm-profiles. |
--project |
Optional. Your Google Cloud project ID. |
--region |
Optional. Region that stores the MLRun metadata.
Default: us-central1. |
--run_group |
Optional. Identifier that groups multiple related runs. |
--configs |
Optional. User-defined configurations that you pass as a JSON object
string. For example,
'{"user":"my-team", "running_on":"TPU"}'. |
--framework |
Optional. Underlying ML framework for the workload
(pytorch or jax). Default:
pytorch. |
--jax_profiler_port |
Optional. JAX profiler server port. Default:
9999. |
--phased_profiling_dir |
Optional. Base Cloud Storage directory for JAX phased profiling
traces. The wrapper appends the
RUN_NAME/plugins/profile/ path and exports the
result as the PHASED_PROFILING_DIR environment variable to
the vLLM process. Use this argument to enable
phased profiling. |
Phased profiling in multi-host setups
To enable JAX phased profiling in multi-host setups, such as a Ray backend on a
GKE LeaderWorkerSet, pass the --phased_profiling_dir argument
and omit the PHASED_PROFILING_DIR environment variable in your operating
system environment.
If you set the PHASED_PROFILING_DIR environment variable without passing the
--phased_profiling_dir argument, phased profiles might not appear in XProf in
the Google Cloud console for the following reasons:
- The
run_vllmwrapper detects thePHASED_PROFILING_DIRvariable in the operating system environment, appends the run-scoped suffix, forwards the path to thevllm servecommand, and logs a warning. - vLLM's Ray executor applies driver environment variables on remote workers
by using the
os.environ.setdefaultmethod. - Remote workers that already have the
PHASED_PROFILING_DIRvariable in their environment keep their existing path instead of adopting the run-scoped path, which writes their traces outside theMLRundirectory.
Run and profile vLLM workloads
To serve a model with vLLM and ML Diagnostics, complete the following steps:
- Build a Docker container image that packages vLLM with the ML Diagnostics
SDK and configures the
run_vllmwrapper as the entrypoint. - Run the container on GKE or Compute Engine by passing
your
MLRunarguments alongside yourvllm servearguments. - Capture performance traces while the server processes inference requests by using phased, manual, or on-demand profiling, and view the results in the Google Cloud console.
Prepare a Docker image for vLLM
To use the run_vllm wrapper in a container, build a Docker image that installs
the ML Diagnostics SDK, the xprof package, and
logging dependencies on top of the vLLM TPU base image. In the Dockerfile, set
the google_cloud_mldiagnostics.commands.run_vllm module as the container
entrypoint.
The following Dockerfile example packages a vLLM TPU image with the ML
Diagnostics SDK:
FROM vllm/vllm-tpu:latest
WORKDIR /app
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir \
google-cloud-mldiagnostics \
xprof \
google-cloud-logging \
packaging \
tensorflow-cpu
EXPOSE 8000 9999
ENTRYPOINT ["python3", "-m", "google_cloud_mldiagnostics.commands.run_vllm"]
CMD ["--help"]
Run vLLM with ML Diagnostics
After you build the Docker image, you can run the container on
GKE or
Compute Engine. When you start the
container, configure the profiler environment variables and pass the MLRun
arguments along with your standard vllm serve arguments.
Run the vLLM container with ML Diagnostics enabled:
docker run --name vllm_diagnostics -d --privileged --net=host \
-e HF_HUB_DISABLE_XET=1 \
-e USE_MOE_EP_KERNEL=1 \
-e PJRT_DEVICE=TPU \
-e ENABLE_GOOGLE_DIAGON_ML_DIAGNOSTICS=True \
-e VLLM_LOGGING_LEVEL=DEBUG \
-e VLLM_TORCH_PROFILER_DIR="gs://BUCKET_NAME/vllm_profiles" \
-e PHASED_PROFILING_DIR="gs://BUCKET_NAME/vllm_profiles" \
-e PYTHONUNBUFFERED=1 \
-e PHASED_PROFILER_NUM_STEPS_TO_PROFILE_FOR=100 \
-e HF_TOKEN="HF_TOKEN" \
-e HUGGING_FACE_HUB_TOKEN="HF_TOKEN" \
VLLM_IMAGE \
--mlrun_name="vllm-tpu-run" \
--run_group="vllm-benchmark" \
--configs='{"user":"ml-team", "running_on":"TPU"}' \
--mlrun_gcs_path="gs://BUCKET_NAME/vllm_profiles" \
--project="PROJECT_ID" \
--region="us-central1" \
-- serve Qwen/Qwen2-1.5B \
--kv-cache-dtype=fp8 \
--tensor-parallel-size=1 \
--data-parallel-size=1 \
--max-model-len=2048 \
--max-num-batched-tokens=2048 \
--max-num-seqs=64 \
--no-enable-prefix-caching \
--block-size=256 \
--gpu-memory-utilization=0.95 \
--port=8000 \
--profiler-config.profiler=torch \
--profiler-config.torch_profiler_dir="gs://BUCKET_NAME/vllm_profiles"
Replace the following:
BUCKET_NAME: the name of your Cloud Storage bucket for storing profiler traces and artifactsHF_TOKEN: your Hugging Face access tokenVLLM_IMAGE: the name or URI of your vLLM Docker imagePROJECT_ID: your Google Cloud project ID
Profile vLLM workloads
When you start your container with the run_vllm command, the wrapper
configures the profiler environment and saves all profiler
traces in Cloud Storage under the
MLRUN_GCS_PATH/RUN_NAME/plugins/profile/ directory:
MLRUN_GCS_PATH: the Cloud Storage bucket path that you specify with the--mlrun_gcs_pathargumentRUN_NAME: the unique identifier for yourMLRunresource
While your vLLM container serves inference requests, you can capture profiler traces by using phased, manual, or on-demand profiling.
Phased profiling
Phased profiling automatically captures traces across different inference
workload phases, such as PREFILL_ONLY, PREFILL_HEAVY, BALANCED, and
DECODE_HEAVY. When you provide a base Cloud Storage path, the run_vllm
wrapper appends the RUN_NAME/plugins/profile/ subdirectory and exports the
resulting path to the PHASED_PROFILING_DIR environment variable.
To enable phased profiling when you start the container, configure the following settings:
- Set the
ENABLE_GOOGLE_DIAGON_ML_DIAGNOSTICS=Trueenvironment variable. - Set the
PHASED_PROFILER_NUM_STEPS_TO_PROFILE_FORenvironment variable to the number of steps to profile. - Pass your base Cloud Storage path using the
--phased_profiling_dirwrapper argument (or set thePHASED_PROFILING_DIRenvironment variable in single-host setups).
Manual (API-triggered) profiling
Manual profiling lets you start and stop profiling sessions programmatically for
specific windows of inference traffic. When you start the container, the
run_vllm wrapper updates the VLLM_TORCH_PROFILER_DIR environment variable
and the --profiler-config.torch_profiler_dir argument to save traces in the
profile directory for your ML run.
To capture a manual profile:
- When you start the container, set the
VLLM_TORCH_PROFILER_DIRenvironment variable and pass the--profiler-config.profiler=torchand--profiler-config.torch_profiler_dirarguments to thevllm servecommand. - While generating load against the server, send HTTP requests to vLLM's
/start_profileand/stop_profileAPI endpoints to start and stop the profiling session.
On-demand profiling in the Google Cloud console
On-demand profiling lets you trigger live profile captures interactively from
the Google Cloud console without restarting your workload or sending API
requests. When you start the container, the run_vllm wrapper automatically
starts the JAX profiler server by setting the USE_JAX_PROFILER_SERVER=True and
JAX_PROFILER_SERVER_PORT environment variables.
To capture an on-demand profile while your workload is serving requests, complete the following steps:
- In the Google Cloud console, go to the Run
Diagnostics tab and select your
MLRunresource:- Cluster Director: Go to the Cluster Director page and click the Run Diagnostics tab.
- GKE: Go to the Kubernetes page, click AI/ML in the navigation menu, and click the Run Diagnostics tab.
- Select the Profiles tab.
- Start a capture session.