Menyesuaikan Gemma 3 di cluster Slurm A4

Tutorial ini menunjukkan cara menyesuaikan model bahasa besar (LLM) Gemma 3 pada cluster Slurm multi-node yang menggunakan dua instance virtual machine (VM) A4. Sebagai bagian dari tutorial ini, Anda akan melakukan hal berikut:

Tutorial ini ditujukan untuk engineer machine learning (ML), administrator dan operator platform, serta spesialis data dan AI yang tertarik menggunakan kemampuan penjadwalan tugas Slurm untuk menangani workload penyesuaian.

Tujuan

  1. Akses Gemma 3 menggunakan Hugging Face.

  2. Siapkan lingkungan Anda.

  3. Buat cluster Slurm A4.

  4. Siapkan workload Anda.

  5. Jalankan tugas penyesuaian.

  6. Pantau tugas Anda.

  7. Jalankan pembersihan.

Biaya

Dalam dokumen ini, Anda akan menggunakan komponen Google Cloudyang dapat ditagih berikut:

Untuk membuat perkiraan biaya berdasarkan proyeksi penggunaan Anda, gunakan kalkulator harga.

Pengguna Google Cloud baru mungkin memenuhi syarat untuk mendapatkan uji coba gratis.

Sebelum memulai

  1. Instal Google Cloud CLI.

  2. Konfigurasi gcloud CLI agar menggunakan identitas gabungan Anda.

    Untuk mengetahui informasi selengkapnya, lihat Login ke gcloud CLI dengan identitas gabungan Anda.

  3. Untuk melakukan inisialisasi gcloud CLI, jalankan perintah berikut: 

    gcloud init

  4. Buat atau pilih Google Cloud project.

    Peran yang diperlukan untuk memilih atau membuat project

    • Pilih project: Memilih project tidak memerlukan peran IAM tertentu—Anda dapat memilih project mana pun yang telah diberi peran.
    • Membuat project: Untuk membuat project, Anda memerlukan peran Project Creator (roles/resourcemanager.projectCreator), yang berisi izin resourcemanager.projects.create. Pelajari cara memberikan peran.

    • Buat Google Cloud project:

      gcloud projects create PROJECT_ID

      Ganti PROJECT_ID dengan nama untuk Google Cloud project yang Anda buat.

    • Pilih project Google Cloud yang Anda buat:

      gcloud config set project PROJECT_ID

      Ganti PROJECT_ID dengan nama project Google Cloud Anda.

  5. Verifikasi bahwa penagihan diaktifkan untuk project Google Cloud Anda.

  6. Aktifkan API yang diperlukan:

    Peran yang diperlukan untuk mengaktifkan API

    Untuk mengaktifkan API, Anda memerlukan izin serviceusage.services.enable. Jika Anda membuat project, kemungkinan Anda sudah memiliki izin ini melalui peran Pemilik (roles/owner). Jika tidak, Anda bisa mendapatkan izin ini melalui peran Admin Penggunaan Layanan (roles/serviceusage.serviceUsageAdmin). Pelajari cara memberikan peran. 

    gcloud services enable compute.googleapis.com file.googleapis.com logging.googleapis.com cloudresourcemanager.googleapis.com servicenetworking.googleapis.com

  7. Memberikan peran ke akun pengguna Anda. Jalankan perintah berikut satu kali untuk setiap peran IAM berikut: roles/compute.admin, roles/iam.serviceAccountUser, roles/file.editor, roles/storage.admin, roles/serviceusage.serviceUsageAdmin 

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

    Ganti kode berikut:

  8. Aktifkan akun layanan default untuk project Google Cloud Anda: 
    gcloud iam service-accounts enable PROJECT_NUMBER-compute@ \
    --project=PROJECT_ID

    Ganti PROJECT_NUMBER dengan nomor project Anda. Untuk meninjau nomor project Anda, lihat Mendapatkan project yang sudah ada.

  9. Berikan peran Editor (roles/editor) ke akun layanan default: 
    gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:PROJECT_NUMBER-compute@" \
    --role=roles/editor
  10. Buat kredensial autentikasi lokal untuk akun pengguna Anda: 
    gcloud auth application-default login
  11. Aktifkan Login OS untuk project Anda: 
    gcloud compute project-info add-metadata --metadata=enable-oslogin=TRUE
  12. Login atau buat akun Hugging Face.

Mengakses Gemma 3 menggunakan Hugging Face

Untuk menggunakan Hugging Face guna mengakses Gemma 3, ikuti langkah-langkah berikut:

  1. Tandatangani perjanjian izin untuk menggunakan Gemma 3 12B.

  2. Buat token akses read Hugging Face. Klik Profil Anda > Setelan > Token akses > +Buat token baru

  3. Salin dan simpan nilai token read access. Anda akan menggunakannya nanti dalam tutorial ini.

Instal Cluster Toolkit

Untuk mengetahui informasi selengkapnya tentang cara menggunakan gcluster dan mengelola cluster, lihat Ringkasan Cluster Toolkit.

  1. Siapkan versi Cluster Toolkit:

    # Find all available releases at: https://github.com/GoogleCloudPlatform/cluster-toolkit/releases
# Set the desired version TAG (e.g., v1.97.0)
export CLUSTER_TOOLKIT_TAG=v1.97.0

# Detect OS (linux or mac)
case "$(uname -s)" in
  Linux*)     OS="linux" ;;
  Darwin*)    OS="mac" ;;
  *)          echo "Error: Unsupported operating system: $(uname -s)" >&2; exit 1 ;; 
esac

# Detect Architecture (amd64 or arm64)
case "$(uname -m)" in
  x86_64)     ARCH="amd64" ;; 
  aarch64|arm64) ARCH="arm64" ;;
  *)          echo "Error: Unsupported architecture: $(uname -m)" >&2; exit 1 ;;
esac

  2. Download rilis:

    # Download and extract the platform-specific bundle
curl -LO "https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/download/${CLUSTER_TOOLKIT_TAG}/gcluster_bundle_${OS}_${ARCH}.zip"
unzip "gcluster_bundle_${OS}_${ARCH}.zip" -d cluster-toolkit/
rm -f "gcluster_bundle_${OS}_${ARCH}.zip"

  3. Tentukan jalur gcluster:

    export CLUSTER_TOOLKIT_PATH="$(pwd)/cluster-toolkit"
export PATH="${CLUSTER_TOOLKIT_PATH}:${PATH}"
gcluster --version

Menyiapkan lingkungan Anda

Untuk menyiapkan lingkungan Anda, ikuti langkah-langkah berikut:

  1. Tetapkan variabel lingkungan default:

    export PROJECT_ID="YOUR_PROJECT_ID"
export ZONE="YOUR_ZONE"
export REGION="YOUR_REGION"
export RESERVATION_URL="RESERVATION_NAME"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export BUCKET_NAME="YOUR_GCS_BUCKET"
export HF_TOKEN="YOUR_HF_TOKEN"

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"

    Ganti kode berikut:

    • YOUR_PROJECT_ID: ID projectGoogle Cloud tempat Anda ingin membuat bucket Cloud Storage.

    • YOUR_ZONE: zona tempat pemesanan Anda berada.

    • YOUR_REGION: region tempat pemesanan Anda berada.

    • RESERVATION_NAME: URL atau nama reservasi yang ingin Anda gunakan untuk membuat cluster Slurm.

    • YOUR_CLUSTER_NAME: nama cluster Slurm yang ingin Anda buat.

    • YOUR_GCS_BUCKET: nama untuk bucket Cloud Storage Anda yang mengikuti persyaratan penamaan bucket.

    • YOUR_HF_TOKEN: token akses Hugging Face yang Anda buat di bagian sebelumnya.

  2. Membuat bucket Cloud Storage: 

    gcloud storage buckets create "gs://${BUCKET_NAME}" \
  --project="${PROJECT_ID}"

Membuat cluster Slurm A4

Untuk membuat cluster Slurm A4, ikuti langkah-langkah berikut:

  1. Buat file a4high-slurm-deployment.yaml:

    MANIFEST_PATH="${CLUSTER_TOOLKIT_PATH}/examples/machine-learning/a4-highgpu-8g"
cat <<EOF > "${MANIFEST_PATH}/a4high-slurm-deployment.yaml"
terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: ${BUCKET_NAME}

vars:
  deployment_name: ${CLUSTER_NAME}
  project_id: ${PROJECT_ID}
  region: ${REGION}
  zone: ${ZONE}
  a4h_cluster_size: 2
  a4h_reservation_name: ${RESERVATION_URL}
EOF

  2. Siapkan manifes:

    1. Buat manifes Terraform:

      gcluster create \
  -d "${MANIFEST_PATH}/a4high-slurm-deployment.yaml" \
  "${MANIFEST_PATH}/a4high-slurm-blueprint.yaml" \
  -o "${WORK_DIR}"

    2. Patch manifes:

      echo "[$(date)] Patching Filestore deletion protection in ${CLUSTER_NAME}..."
sed -i '/deletion_protection = {/,/}/ { s/enabled = true/enabled = false/; /reason  = "Avoid data loss"/d; }' "${WORK_DIR}/${CLUSTER_NAME}/cluster-env/main.tf"

  3. Deploy cluster:

    gcluster deploy "${WORK_DIR}/${CLUSTER_NAME}" --auto-approve

    Perintah gcluster deploy adalah proses dua fase, yang dijelaskan sebagai berikut:

    • Fase pertama membuat image kustom dengan semua software yang sudah diinstal sebelumnya, yang dapat memerlukan waktu hingga 45 menit untuk diselesaikan.

    • Fase kedua men-deploy cluster menggunakan image kustom tersebut. Proses ini umumnya memerlukan waktu lebih singkat untuk diselesaikan daripada fase pertama.

    Jika fase pertama berhasil, tetapi fase kedua gagal, Anda dapat mencoba men-deploy cluster Slurm lagi dengan melewati fase pertama:

    gcluster deploy "${CLUSTER_NAME}" --auto-approve --skip "image" -w

Menyiapkan workload Anda

Untuk menyiapkan beban kerja Anda, ikuti langkah-langkah berikut:

  1. Buat skrip beban kerja.

  2. Upload skrip ke cluster Slurm.

  3. Hubungkan ke cluster Slurm.

  4. Instal framework dan alat.

Membuat skrip beban kerja

Untuk membuat skrip yang akan digunakan workload penyesuaian, ikuti langkah-langkah berikut:

  1. Untuk menyiapkan lingkungan virtual Python, buat file install_environment.sh dengan konten berikut:

    #!/bin/bash
# This script should be run ONCE on the login node to set up the
# shared Python virtual environment.

set -e
echo "--- Creating Python virtual environment in /home ---"
python3 -m venv ~/.venv
echo "--- Activating virtual environment ---"
source ~/.venv/bin/activate

echo "--- Installing build dependencies ---"
pip install --upgrade pip wheel packaging

echo "--- Installing PyTorch for CUDA 12.8 ---"
pip install torch --index-url https://download.pytorch.org/whl/cu128

echo "--- Installing application requirements ---"
pip install -r requirements.txt

echo "--- Environment setup complete. You can now submit jobs with sbatch. ---"

  2. Untuk menentukan konfigurasi tugas penyesuaian, buat file accelerate_config.yaml dengan konten berikut:

    # Default configuration for a 2-node, 8-GPU-per-node (16 total GPUs) FSDP training job.

compute_environment: "LOCAL_MACHINE"
distributed_type: "FSDP"
downcast_bf16: "no"
fsdp_config:
  fsdp_auto_wrap_policy: "TRANSFORMER_BASED_WRAP"
  fsdp_backward_prefetch: "BACKWARD_PRE"
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: false
  fsdp_offload_params: false
  fsdp_sharding_strategy: "FULL_SHARD"
  fsdp_state_dict_type: "FULL_STATE_DICT"
  fsdp_transformer_layer_cls_to_wrap: "Gemma3DecoderLayer"
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: "main"
mixed_precision: "bf16"
num_machines: 2
num_processes: 16
rdzv_backend: "static"
same_network: true
tpu_env: []
use_cpu: false

  3. Untuk menentukan tugas yang akan dijalankan pada cluster Slurm, buat file submit.slurm dengan konten berikut:

    #!/bin/bash
#SBATCH --job-name=gemma3-finetune
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1 # 1 task per node
#SBATCH --gpus-per-node=8   # 8 GPUs per node
#SBATCH --partition=a4high
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

set -e
echo "--- Slurm Job Started ---"

# --- STAGE 1: Copy Environment to Local SSD on all nodes ---
srun --ntasks=$SLURM_NNODES --ntasks-per-node=1 bash -c '
  echo "Setting up local environment on $(hostname)..."
  LOCAL_VENV="/mnt/localssd/venv_job_${SLURM_JOB_ID}"
  LOCAL_CACHE="/mnt/localssd/hf_cache_job_${SLURM_JOB_ID}"
  rsync -a --info=progress2 ~/./.venv/ ${LOCAL_VENV}/
  mkdir -p ${LOCAL_CACHE}
  echo "Setup on $(hostname) complete."
'

# --- STAGE 2: Run the Training Job using the Local Environment ---
echo "--- Starting Training ---"

LOCAL_VENV="/mnt/localssd/venv_job_${SLURM_JOB_ID}"
LOCAL_CACHE="/mnt/localssd/hf_cache_job_${SLURM_JOB_ID}"
LOCAL_OUTPUT_DIR="/mnt/localssd/outputs_${SLURM_JOB_ID}"
mkdir -p ${LOCAL_OUTPUT_DIR}

# This is the main training command.
srun --ntasks=$SLURM_NNODES --ntasks-per-node=1 bash -c '
  # Ensure nccl-gib is at version 1.1.2-1
  INSTALLED_VERSION=$(dpkg-query -W -f="\${Version}" nccl-gib 2>/dev/null || echo "not_installed")
  if [ "${INSTALLED_VERSION}" != "1.1.2-1" ]; then
    echo "Upgrading nccl-gib to 1.1.2-1..."
    sudo apt-get update && sudo apt-get install -y --allow-change-held-packages nccl-gib=1.1.2-1
  fi

  source '"${LOCAL_VENV}"'/bin/activate

  export HF_HOME='"${LOCAL_CACHE}"'
  export HF_DATASETS_CACHE='"${LOCAL_CACHE}"'

  if [ -f /usr/local/gib/scripts/set_nccl_env.sh ]; then
    echo "Sourcing set_nccl_env.sh"
    source /usr/local/gib/scripts/set_nccl_env.sh
  fi
  export LD_LIBRARY_PATH=/usr/local/gib/lib64:${LD_LIBRARY_PATH:-}

  export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
  export MASTER_PORT=29500

  # Use accelerate launch to properly run the distributed FSDP job
  accelerate launch \
    --config_file ~/accelerate_config.yaml \
    --machine_rank $SLURM_NODEID \
    --main_process_ip $MASTER_ADDR \
    --main_process_port $MASTER_PORT \
    --num_machines $SLURM_NNODES \
    --num_processes $((SLURM_NNODES * 8)) \
    ~/train.py \
    --model_id google/gemma-3-12b-pt \
    --output_dir '"${LOCAL_OUTPUT_DIR}"' \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 8 \
    --num_train_epochs 3 \
    --learning_rate 1e-5 \
    --save_strategy steps \
    --save_steps 100
'

# --- STAGE 3: Copy Final Model from Local SSD to Home Directory ---
echo "--- Copying final model from local SSD to /home ---"
# This command runs only on the first node of the job allocation
# and copies the final model back to the persistent shared directory.
srun --nodes=1 --ntasks=1 --ntasks-per-node=1 bash -c "
  rsync -a --info=progress2 ${LOCAL_OUTPUT_DIR}/ ~/gemma-12b-text-to-sql-finetuned/
"

echo "--- Slurm Job Finished ---"

  4. Untuk menentukan dependensi bagi tugas penyesuaian Anda, buat file requirements.txt dengan konten berikut:

    # Hugging Face Libraries (Pinned to recent, stable versions for reproducibility)
transformers==4.53.3
datasets==4.0.0
accelerate==1.9.0
evaluate==0.4.5
bitsandbytes==0.46.1
trl==0.19.1
peft==0.16.0

# Other dependencies
tensorboard==2.20.0
protobuf==6.31.1
sentencepiece==0.2.0

  5. Untuk menentukan petunjuk bagi tugas Anda, buat file train.py dengan konten berikut:

    import torch
import argparse
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, AutoConfig
from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model
from trl import SFTTrainer, SFTConfig
from huggingface_hub import login


def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_id", type=str, default="google/gemma-3-12b-pt", help="Hugging Face model ID")
    parser.add_argument("--hf_token", type=str, default=None, help="Hugging Face token for private models")
    parser.add_argument("--dataset_name", type=str, default="philschmid/gretel-synthetic-text-to-sql", help="Hugging Face dataset name")
    parser.add_argument("--output_dir", type=str, default="gemma-12b-text-to-sql", help="Directory to save model checkpoints")

    # LoRA arguments
    parser.add_argument("--lora_r", type=int, default=16, help="LoRA attention dimension")
    parser.add_argument("--lora_alpha", type=int, default=16, help="LoRA alpha scaling factor")
    parser.add_argument("--lora_dropout", type=float, default=0.05, help="LoRA dropout probability")

    # SFTConfig arguments
    parser.add_argument("--max_seq_length", type=int, default=512, help="Maximum sequence length")
    parser.add_argument("--num_train_epochs", type=int, default=3, help="Number of training epochs")
    parser.add_argument("--per_device_train_batch_size", type=int, default=8, help="Batch size per device during training")
    parser.add_argument("--gradient_accumulation_steps", type=int, default=1, help="Gradient accumulation steps")
    parser.add_argument("--learning_rate", type=float, default=1e-5, help="Learning rate")
    parser.add_argument("--logging_steps", type=int, default=10, help="Log every X steps")
    parser.add_argument("--save_strategy", type=str, default="steps", help="Checkpoint save strategy")
    parser.add_argument("--save_steps", type=int, default=100, help="Save checkpoint every X steps")

    return parser.parse_args()

def main():
    args = get_args()

    # --- 1. Setup and Login ---
    if args.hf_token:
        login(args.hf_token)

    # --- 2. Create and prepare the fine-tuning dataset ---
    # The SFTTrainer will use the `formatting_func` to apply the chat template.
    dataset = load_dataset(args.dataset_name, split="train")
    dataset = dataset.shuffle().select(range(12500))
    dataset = dataset.train_test_split(test_size=2500/12500)

    # --- 3. Configure Model and Tokenizer ---
    if torch.cuda.is_available() and torch.cuda.get_device_capability()[0] >= 8:
        torch_dtype_obj = torch.bfloat16
        torch_dtype_str = "bfloat16"
    else:
        torch_dtype_obj = torch.float16
        torch_dtype_str = "float16"

    tokenizer = AutoTokenizer.from_pretrained(args.model_id)
    tokenizer.pad_token = tokenizer.eos_token

    gemma_chat_template = (
        "{% for message in messages %}"
        "{% if message['role'] == 'user' %}"
        "{{ '<start_of_turn>user\n' + message['content'] + '<end_of_turn>\n' }}"
        "{% elif message['role'] == 'assistant' %}"
        "{{ '<start_of_turn>model\n' + message['content'] + '<end_of_turn>\n' }}"
        "{% endif %}"
        "{% endfor %}"
        "{% if add_generation_prompt %}"
        "{{ '<start_of_turn>model\n' }}"
        "{% endif %}"
    )
    tokenizer.chat_template = gemma_chat_template

    # --- 4. Define the Formatting Function ---
    # This function will be used by the SFTTrainer to format each sample
    # from the dataset into the correct chat template format.
    def formatting_func(example):
        # The create_conversation logic is now implicitly handled by this.
        # We need to construct the messages list here.
        system_message = "You are a text to SQL query translator. Users will ask you questions in English and you will generate a SQL query based on the provided SCHEMA."
        user_prompt = "Given the <USER_QUERY> and the <SCHEMA>, generate the corresponding SQL command to retrieve the desired data, considering the query's syntax, semantics, and schema constraints.\n\n<SCHEMA>\n{context}\n</SCHEMA>\n\n<USER_QUERY>\n{question}\n</USER_QUERY>\n"

        messages = [
            {"role": "system", "content": system_message},
            {"role": "user", "content": user_prompt.format(question=example["sql_prompt"], context=example["sql_context"])},
            {"role": "assistant", "content": example["sql"]}
        ]
        return tokenizer.apply_chat_template(messages, tokenize=False)

    # --- 5. Load Quantized Model and Apply PEFT ---

    # Define the quantization configuration
    config = AutoConfig.from_pretrained(args.model_id)
    config.use_cache = False

    # Load the base model in native precision
    print("Loading base model...")
    model = AutoModelForCausalLM.from_pretrained(
        args.model_id,
        config=config,
        attn_implementation="sdpa",
        torch_dtype=torch_dtype_obj,
    )

    # Configure LoRA.
    peft_config = LoraConfig(
        lora_alpha=args.lora_alpha,
        lora_dropout=args.lora_dropout,
        r=args.lora_r,
        bias="none",
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
        task_type="CAUSAL_LM",
    )

    # Apply the PEFT config to the model
    print("Applying PEFT configuration...")
    model = get_peft_model(model, peft_config)
    model.print_trainable_parameters()

    # --- 6. Configure Training Arguments ---
    training_args = SFTConfig(
        output_dir=args.output_dir,
        ddp_timeout=300.0,
        max_seq_length=args.max_seq_length,
        num_train_epochs=args.num_train_epochs,
        per_device_train_batch_size=args.per_device_train_batch_size,
        gradient_accumulation_steps=args.gradient_accumulation_steps,
        learning_rate=args.learning_rate,
        logging_steps=args.logging_steps,
        save_strategy=args.save_strategy,
        save_steps=args.save_steps,
        packing=False,
        gradient_checkpointing=True,
        gradient_checkpointing_kwargs={"use_reentrant": False},
        optim="adamw_torch",
        fp16=True if torch_dtype_obj == torch.float16 else False,
        bf16=True if torch_dtype_obj == torch.bfloat16 else False,
        max_grad_norm=0.3,
        warmup_ratio=0.03,
        lr_scheduler_type="constant",
        push_to_hub=False,
        report_to="tensorboard",
        dataset_kwargs={
            "add_special_tokens": False,
            "append_concat_token": True,
        }
    )

    # --- 7. Create Trainer and Start Training ---
    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset["train"],
        eval_dataset=dataset["test"],
        formatting_func=formatting_func,
    )

    print("Starting training...")
    trainer.train()
    print("Training finished.")

    # --- 8. Save the final model ---
    print(f"Saving final model to {args.output_dir}")
    trainer.save_model()

if __name__ == "__main__":
    main()

Mengupload skrip ke cluster Slurm

Untuk mengupload skrip yang Anda buat di bagian sebelumnya ke cluster Slurm, ikuti langkah-langkah berikut:

  1. Tetapkan variabel LOGIN_NODE dengan mengambil nama node login untuk cluster Anda:

    LOGIN_NODE="$(gcloud compute instances list \
                --project="${PROJECT_ID}" \
                --filter="labels.ghpc_deployment='${CLUSTER_NAME}' AND labels.slurm_instance_role='login'" \
                --format="value(name)" | head -n 1)"

    Variabel LOGIN_NODE menyimpan nilai yang mirip dengan ${CLUSTER_NAME}-login-001.

  2. Upload skrip Anda ke direktori utama node login:

    gcloud compute scp \
  --project="${PROJECT_ID}" \
  --zone="${ZONE}" \
  --tunnel-through-iap \
  ./install_environment.sh \
  ./requirements.txt \
  ./submit.slurm \
  ./accelerate_config.yaml \
  ./train.py \
  "${LOGIN_NODE}":~/

Menghubungkan ke cluster Slurm

Hubungkan ke node login, dan sebarkan token Hugging Face Anda ke sesi baru:

gcloud compute ssh "${LOGIN_NODE}" \
    --project="${PROJECT_ID}" \
    --tunnel-through-iap \
    --zone="${ZONE}" \
    -- -t "export HF_TOKEN='${HF_TOKEN}'; bash -l"

Menginstal framework dan alat

Setelah terhubung ke node login, siapkan lingkungan virtual Python dengan dependensi yang diperlukan:

chmod +x install_environment.sh
./install_environment.sh

Mulai workload penyesuaian

Untuk memulai workload penyesuaian, ikuti langkah-langkah berikut:

  1. Kirim tugas ke penjadwal Slurm dan kumpulkan ID pekerjaan:

    JOB_ID="$(sbatch submit.slurm 2>&1 \
  | tee /dev/tty \
  | grep -oP 'Submitted batch job \K\d+')"

  2. Di node login di cluster Slurm, Anda dapat memantau progres tugas dengan memeriksa file output yang dibuat di direktori home:

    tail -f "slurm-${JOB_ID}.out" "slurm-${JOB_ID}.err"

    Jika tugas Anda berhasil dimulai, file .err akan menampilkan status progres yang diperbarui seiring progres tugas Anda.

Memantau workload Anda

Anda dapat memantau penggunaan GPU di cluster Slurm untuk memverifikasi bahwa tugas penyesuaian Anda berjalan secara efisien. Untuk melakukannya, buka link berikut di browser Anda:

https://console.cloud.google.com/monitoring/metrics-explorer?project=PROJECT_ID&pageState=%7B%22xyChart%22%3A%7B%22dataSets%22%3A%5B%7B%22timeSeriesFilter%22%3A%7B%22filter%22%3A%22metric.type%3D%5C%22agent.googleapis.com%2Fgpu%2Futilization%5C%22%20resource.type%3D%5C%22gce_instance%5C%22%22%2C%22perSeriesAligner%22%3A%22ALIGN_MEAN%22%7D%2C%22plotType%22%3A%22LINE%22%7D%5D%7D%7D

Saat memantau workload, Anda dapat melihat hal berikut:

  • Penggunaan GPU: untuk tugas penyesuaian yang berjalan lancar, Anda dapat melihat penggunaan semua 16 GPU (delapan GPU untuk setiap VM dalam cluster) meningkat dan stabil ke tingkat tertentu selama pelatihan.

  • Durasi tugas: tugas akan memerlukan waktu sekitar satu jam untuk diselesaikan.

Pembersihan

Agar tidak perlu membayar biaya pada akun Google Cloud Anda untuk resource yang digunakan dalam tutorial ini, hapus project yang berisi resource tersebut, atau simpan project dan hapus setiap resource.

Menghapus cluster Slurm

Untuk menghapus cluster Slurm Anda:

gcluster destroy "${CLUSTER_NAME}" --auto-approve

Jika Anda perlu menghapus semua jaringan VPC, aturan firewall, router, IP, dan subnet yang terkait dengan project, lakukan hal berikut:

echo "========================================================================="
echo " STARTING AUTOMATED NETWORK CLEANUP FOR CLUSTER: ${CLUSTER_NAME}"
echo "========================================================================="

echo "Discovering all VPC networks linked to the cluster..."
NETWORKS=$(gcloud compute networks list --project="${PROJECT_ID}" --format="value(name)" | grep "^${CLUSTER_NAME}" || true)

if [ -z "${NETWORKS}" ]; then
    echo "No VPC networks found starting with ${CLUSTER_NAME}. Everything is already clean!"
    exit 0
fi

echo "Found the following networks to process:"
echo "${NETWORKS}"
echo "-------------------------------------------------------------------------"

echo "=== 1. Wiping Global Firewall Rules ==="
FIREWALL_RULES=$(gcloud compute firewall-rules list \
    --project="${PROJECT_ID}" \
    --filter="network ~ ^${CLUSTER_NAME} OR name ~ ^${CLUSTER_NAME}" \
    --format="value(name)" || echo "")

if [ -n "${FIREWALL_RULES}" ]; then
    echo "Deleting matching firewall rules:"
    echo "${FIREWALL_RULES}"
    echo "${FIREWALL_RULES}" | xargs -r gcloud compute firewall-rules delete --project="${PROJECT_ID}" --quiet
else
    echo "No matching firewall rules found."
fi

echo "=== 2. Tearing Down Network-Specific Infrastructure ==="
echo "${NETWORKS}" | while read -r net_name; do
    [ -z "${net_name}" ] && continue
    echo "Processing resources for network: ${net_name}"

    ROUTERS=$(gcloud compute routers list \
        --project="${PROJECT_ID}" \
        --regions="${REGION}" \
        --filter="network=${net_name}" \
        --format="value(name)" || echo "")

    if [ -n "${ROUTERS}" ]; then
        echo "  -> Deleting routers: ${ROUTERS}"
        echo "${ROUTERS}" | xargs -r gcloud compute routers delete --region="${REGION}" --project="${PROJECT_ID}" --quiet
    fi

    IPS=$(gcloud compute addresses list \
        --project="${PROJECT_ID}" \
        --regions="${REGION}" \
        --filter="name ~ ^${net_name}" \
        --format="value(name)" || echo "")

    if [ -n "${IPS}" ]; then
        echo "  -> Deleting IP reservations: ${IPS}"
        echo "${IPS}" | xargs -r gcloud compute addresses delete --region="${REGION}" --project="${PROJECT_ID}" --quiet
    fi

    SUBNETS=$(gcloud compute networks subnets list \
        --project="${PROJECT_ID}" \
        --regions="${REGION}" \
        --filter="network=${net_name}" \
        --format="value(name)" || echo "")

    if [ -n "${SUBNETS}" ]; then
        echo "  -> Deleting subnetworks:"
        echo "${SUBNETS}"
        echo "${SUBNETS}" | xargs -r gcloud compute networks subnets delete --region="${REGION}" --project="${PROJECT_ID}" --quiet
    fi
done

echo "-------------------------------------------------------------------------"
echo "Waiting 15 seconds for Google Cloud API dependencies to unlock..."
sleep 15

echo "=== 3. Final VPC Networks Destruction ==="
echo "${NETWORKS}" | while read -r net_name; do
    [ -z "${net_name}" ] && continue
    echo "Deleting core VPC network: ${net_name}..."
    gcloud compute networks delete "${net_name}" --project="${PROJECT_ID}" --quiet || \
    echo "Warning: Could not delete ${net_name} yet. If a lock occurred, please rerun in 1 minute."
done

echo "========================================================================="
echo " SUCCESS: All network resources for cluster ${CLUSTER_NAME} have been wiped!"
echo "========================================================================="

Menghapus project Anda

Menghapus Google Cloud project:

gcloud projects delete PROJECT_ID

Langkah berikutnya