gcluster job submit

NAME

gcluster job submit - submits a workload to a cluster.

SYNOPSIS

gcluster job submit [FLAGS]

DESCRIPTION

The gcluster job submit command submits a workload to a cluster. On GKE clusters, Cluster Toolkit deploys workloads as Kubernetes JobSets integrated with Kueue.

FLAGS

  • -i, --image IMAGE: The name of the prebuilt container image to run, including the registry path. Specify either the --image flag or the --base-image flag, unless you submit a headless Pathways job with the --pathways-headless flag.
  • -B, --base-image IMAGE: The name of the base image used to build a new container image and push it to Artifact Registry by using Crane. This flag requires the --build-context flag and can't be combined with the --image flag.
  • -b, --build-context DIR: The path to the build context directory for Crane. This flag is required when you use the --base-image flag and also requires you to set the GCLUSTER_IMAGE_REPO environment variable to your Artifact Registry repository name.
  • -e, --command COMMAND: The command to run in the workload container, such as python3 train.py. This flag is required for standard job submissions, unless you submit a headless Pathways job with the --pathways-headless flag.
  • --compute-type TYPE: The type of compute to request, such as n2-standard-32, nvidia-l4, or v5litepod-32. This flag is required.
  • -o, --dry-run-out PATH: The path to output the generated Kubernetes manifest instead of applying it.
  • -f, --platform PLATFORM: The target platform for the image build, such as linux/amd64 or linux/arm64. Use this flag with the --base-image flag.
  • --mount VOLUME: The volumes to mount.
  • --env KEY=VALUE: Custom environment variables in KEY=VALUE format to pass to the workload container. You can specify this flag multiple times.
  • -n, --name NAME: The name of the workload to create. The name can't exceed 28 characters for standard jobs or 22 characters for Pathways jobs (--pathways). This flag is required.
  • -q, --queue QUEUE: The name of the Kueue LocalQueue.

  • --num-nodes N: The number of nodes to use per group or slice.

    Note: The `--num-nodes` flag can't be used with TPU jobs because the
  node count is calculated automatically from the `--topology` value.

  • --num-slices N: The number of independent groups or slices to use.

  • --restarts N: The maximum number of restarts for the JobSet.

  • --gke-ttl-after-finished DURATION: The time to retain the JobSet after it finishes.

  • --grace-period DURATION: The time to wait before forcefully terminating a pod.

  • --gke-disable-parallel-containers: Disables parallel containers for TPU7x on GKE.

  • --placement-policy POLICY: The name of the GKE placement policy to use.

  • --node-constraint KEY=VALUE: Key=value pairs for node labels to target specific nodes.

  • --cpu-affinity AFFINITY: CPU affinity rules.

  • --restart-on-exit-codes CODES: A list of exit codes that don't trigger a job failure.

  • --image-pull-secret SECRET: A comma-separated list of secrets for pulling images.

  • --service-account ACCOUNT: The Kubernetes service account name for the Pods. On TPU and Pathways jobs, setting this flag disables the hostNetwork field so that Pods authenticate by using Workload Identity Federation for GKE. GPU jobs use the hostNetwork: true setting, so their Pods authenticate as the GKE node pool's service account.

  • --topology TOPOLOGY: The TPU slice topology.

  • --gke-scheduler SCHEDULER: The Kubernetes Scheduler name.

  • --await-job-completion: Waits for the submitted job to complete.

  • --timeout TIMEOUT: The time to wait for the job.

  • --priority PRIORITY: A priority class name.

  • --verbose: Turns on verbose logging for the workload.

  • --gke-nap-provisioning MODEL: The compute provisioning model for GKE node auto-provisioning, such as on-demand, spot, or reservation.

  • --gke-nap-reservation RESERVATION: The name of the Google Cloud reservation for GKE node auto-provisioning. This field is required when you set the value --gke-nap-provisioning=reservation.

  • --pathways: Generates a manifest for a Pathways job.

  • --pathways-proxy-server-image PW_PROXY_IMAGE: The image for the Pathways proxy server.

  • --pathways-server-image PW_SERVER_IMAGE: The image for the Pathways server.

  • --pathways-worker-image IMAGE: The image for the Pathways worker.

  • --pathways-headless: Prevents deploying the user's workload container within the pathways-head job.

  • --pathways-gcs-location PW_GCS_LOCATION: The Cloud Storage location to store Pathways artifacts. This flag is required when the --pathways flag is set.

  • --pathways-elastic-slices N: Configures the number of elastic slices.

  • --pathways-max-slice-restarts N: The maximum times the workers in a slice can be restarted.

  • --pathways-proxy-args ARGS: Additional arguments for pathways-proxy.

  • --pathways-server-args ARGS: Additional arguments for pathways-rm.

  • --pathways-worker-args ARGS: Additional arguments for pathways-worker.

  • --pathways-proxy-env KEY=VALUE: Environment variables for the Pathways proxy.

  • --pathways-server-env KEY=VALUE: Environment variables for the Pathways server.

  • --pathways-worker-env KEY=VALUE: Environment variables for the Pathways worker.

  • --pathways-colocated-python-sidecar-image IMAGE: The image for the Python-based sidecar.

  • --pathways-head-np NP: The node pool to use for the Pathways head job.

  • --gke-mtc-enabled: Turns on Multi-Tier Checkpointing (MTC).

  • --gke-mtc-ramdisk-dir DIR: The ramdisk directory path for MTC. This flag is required when the --gke-mtc-enabled flag is set.

  • --gke-custom-templates-path PATH: The path to the local directory containing custom GKE templates.

PERSISTENT FLAGS

These flags are available to all gcluster job commands:

  • -c, --cluster CLUSTER_NAME: The name of the GKE cluster.
  • -l, --location LOCATION: The location, either a region or a zone, of the GKE cluster.
  • -p, --project PROJECT_ID: The Google Cloud project ID.
  • --gke-namespace NAMESPACE: The target GKE namespace for the operation.
  • --skip-prereqs: Skips local environment prerequisite checks, such as gcloud, kubectl, and docker authentication.

EXAMPLES

To submit a workload named train-job to a cluster:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=train-job \
    --image=IMAGE \
    --compute-type=TYPE \
    --command="python3 train.py"

To submit a multi-node workload:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=multi-node-job \
    --image=IMAGE \
    --command="python3 train.py" \
    --compute-type=TYPE \
    --num-nodes=N

To submit a multi-slice workload:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=multi-slice-job \
    --image=IMAGE \
    --command="python3 train.py" \
    --compute-type=TYPE \
    --num-slices=N

To submit a Pathways workload:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=pw-train-job \
    --compute-type=TYPE \
    --image=IMAGE \
    --pathways-server-image=PW_SERVER_IMAGE \
    --pathways-proxy-server-image=PW_PROXY_IMAGE \
    --command="python3 train.py" \
    --pathways \
    --pathways-gcs-location=PW_GCS_LOCATION