Learn how to submit a batch workload on Managed Service for Apache Spark compute infrastructure that scales resources as needed.
Before you begin
Set up your project and, if needed, grant Identity and Access Management roles.
Set up your project
Perform one or more of the following steps as needed:
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
Enable the Dataproc API, if it is not already enabled.
Roles required to enable APIs
To enable APIs, you need the
serviceusage.services.enablepermission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.
Grant IAM roles if needed
Certain IAM roles are required to run the examples on this page. Depending on organization policies, these roles may have already been granted. To check role grants, see Do you need to grant roles?.
For more information about granting roles, see Manage access to projects,folders, and organizations.
User roles
To get the permissions that you need to submit a serverless batch workload, ask your administrator to grant you the following IAM roles:
- Dataproc Editor (
roles/dataproc.editor) on the project - Service Account User (
roles/iam.serviceAccountUser) on the Compute Engine default service account
Service account role
To ensure that the Compute Engine default service account has the necessary
permissions to submit a serverless batch workload,
ask your administrator to grant the
Dataproc Worker (roles/dataproc.worker) IAM role to the Compute Engine default service account on the project.
Submit a Spark batch workload
You can use the Google Cloud console, the Google Cloud CLI, or the REST API to create and submit a Managed Service for Apache Spark batch workload.
Console
In the Google Cloud console, go to the Managed Service for Apache Spark Jobs page.
Click Submit job to open the Submit a job page.
Submit a Spark batch workload that computes the approximate value of pi by selecting and filling in the following fields:
- Select compute type for job: Select Serverless Spark.
- Serverless job details:
- Job ID: Specify an ID for your batch workload (or accept the default
ID). This value must be 4 to 63 lowercase characters, start and end with
a lowercase letter or number, and contain only lowercase letters,
numbers, and hyphens (
/[a-z0-9-]/). - Runtime profile: Select a region in the Regions list and select Default service configuration (or a saved runtime profile) in the Runtime profiles list.
- Configure runtime profile (optional): In the Summary card, click
Configure (or click Create new profile in the Runtime
profile menu) to open the runtime profile configuration panel, where
you can customize the following settings and then click Use
configuration (or Save new profile):
- Tier and Executor configuration: Select the compute tier (Premium or Standard) and executor shape.
- Additional configuration:
- Runtime configuration: Click Edit to confirm or select
the
3.0Spark runtime version, or to specify a custom Spark image, Cloud Storage staging bucket, or Python package repository. - Metastore configuration: Click Edit to configure the workload to use a Lakehouse catalog or an external self-managed Dataproc Metastore.
- Network and security: Click Edit to select Service Account (by default, the workload runs using the Compute Engine default service account, or you can specify a custom service account with the Dataproc Worker role) and select a Subnetwork in the workload region (Managed Service for Apache Spark enables Private Google Access (PGA) on the specified subnet; see Managed Service for Apache Spark network configuration).
- Runtime configuration: Click Edit to confirm or select
the
- Job ID: Specify an ID for your batch workload (or accept the default
ID). This value must be 4 to 63 lowercase characters, start and end with
a lowercase letter or number, and contain only lowercase letters,
numbers, and hyphens (
- Files and artifacts:
- Job type: Select Spark (Scala / Java).
- Driver configuration: Select Main class.
- Main class:
org.apache.spark.examples.SparkPi
- Arguments: Enter
1000and pressEnter. - Jar files (this file is pre-installed in the Managed Service for Apache Spark execution environment):
file:///usr/lib/spark/examples/jars/spark-examples.jar
- Job execution and properties:
- Spark properties: Click Add property to enter the Key
(property name) and Value of
supported Spark properties
to set on your Spark batch workload. Note: Unlike Managed Service for Apache Spark
cluster properties,
Managed Service for Apache Spark workload properties don't include a
spark:prefix. - Labels: Optional: Click Add label to attach Key and Value label pairs to the workload.
- Spark properties: Click Add property to enter the Key
(property name) and Value of
supported Spark properties
to set on your Spark batch workload. Note: Unlike Managed Service for Apache Spark
cluster properties,
Managed Service for Apache Spark workload properties don't include a
Click Submit to run the Spark batch workload.
gcloud
To submit a Spark batch workload to compute the approximate value
of pi, run the following gcloud CLI
gcloud dataproc batches submit spark
command locally in a terminal window or in
Cloud Shell.
gcloud dataproc batches submit spark \ --region=REGION \ --version=3.0 \ --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \ --class=org.apache.spark.examples.SparkPi \ -- 1000
Replace the following:
- REGION: Specify the region where your workload will run.
- Other options: You can add
gcloud dataproc batches submit sparkflags to specify other workload options and Spark properties.--jars: The example JAR file is pre-installed in the Spark execution environment, The1000command argument passed to the SparkPi workload specifies 1000 iterations of the pi estimation logic (workload input arguments are included after the "-- ").--subnet: You can add this flag to specify the name of a subnet in the session region. If you don't specify a subnet, Managed Service for Apache Spark selects thedefaultsubnet in the session region. Managed Service for Apache Spark enables Private Google Access (PGA) on the subnet. For network connectivity requirements, see Managed Service for Apache Spark network configuration.--tags: You can add this flag to specify network tags for traffic control. Use network tags to limit connectivity. In production, the recommended practice is to limit firewall rules to the IP addresses used by your Spark workloads.--properties: You can add this flag to enter supported Spark properties for your Spark batch workload to use.--deps-bucket: You can add this flag to specify a Cloud Storage bucket where Managed Service for Apache Spark will upload workload dependencies. Thegs://URI prefix of the bucket is not required; you can specify the bucket path or bucket name. Managed Service for Apache Spark uploads the local file(s) to a/dependenciesfolder in the bucket before running the batch workload. Note: This flag is required if your batch workload references files on your local machine.--ttl: You can add the--ttlflag to specify the duration of the batch lifetime. When the workload exceeds this duration, it is unconditionally terminated without waiting for ongoing work to finish. Specify the duration using as,m,h, ord(seconds, minutes, hours, or days) suffix. The minimum value is 10 minutes (10m), and the maximum value is 14 days (14d).- 1.1 or 2.0 runtime batches: If
--ttlis not specified for a 1.1 or 2.0 runtime batch workload, the workload is allowed to run until it exits naturally (or run forever if it doesn't exit). - 2.1+ runtime batches: If
--ttlis not specified for a 2.1 or later runtime batch workload, it defaults to4h.
- 1.1 or 2.0 runtime batches: If
--service-account: You can specify a service account to use to run your workload. If you don't specify a service account, the workload runs under the Compute Engine default service account. Your service account must have the Dataproc Worker role.- Hive Metastore: The following command configures a batch workload to use an external
self-managed Hive Metastore
using a standard Spark configuration.
gcloud dataproc batches submit spark\ --properties=spark.sql.catalogImplementation=hive,spark.hive.metastore.uris=METASTORE_URI,spark.hive.metastore.warehouse.dir=WAREHOUSE_DIR> \ other args ... - Persistent History Server:
- The following command creates a PHS on a single-node Managed Service for Apache Spark
cluster. The PHS must be located in the region where you run batch workloads,
and the Cloud Storage bucket-name must
exist.
gcloud dataproc clusters create PHS_CLUSTER_NAME \ --region=REGION \ --single-node \ --enable-component-gateway \ --properties=spark:spark.history.fs.logDirectory=gs://bucket-name/phs/*/spark-job-history
- Submit a batch workload, specifying your running Persistent History Server.
gcloud dataproc batches submit spark \ --region=REGION \ --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \ --class=org.apache.spark.examples.SparkPi \ --history-server-cluster=projects/project-id/regions/region/clusters/PHS-cluster-name \ -- 1000
- The following command creates a PHS on a single-node Managed Service for Apache Spark
cluster. The PHS must be located in the region where you run batch workloads,
and the Cloud Storage bucket-name must
exist.
- Runtime version:
Use the
--versionflag to specify the Managed Service for Apache Spark runtime version for the workload.gcloud dataproc batches submit spark \ --region=REGION \ --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \ --class=org.apache.spark.examples.SparkPi \ --version=VERSION -- 1000
API
This section shows how to create a batch workload
to compute the approximate value
of pi using the Managed Service for Apache Spark
batches.create`
Before using any of the request data, make the following replacements:
- project-id: A Google Cloud project ID.
- region: A Compute Engine region where Managed Service for Apache Spark will run the workload.
- PROJECT_ID: Your Google Cloud project ID. Project IDs are listed in the Project info section on the Google Cloud console Dashboard.
- REGION: The session region.
Notes:
HTTP method and URL:
POST https://dataproc.googleapis.com/v1/projects/project-id/locations/region/batches
Request JSON body:
{
"sparkBatch":{
"args":[
"1000"
],
"runtimeConfig": {
"version": "2.3",
},
"jarFileUris":[
"file:///usr/lib/spark/examples/jars/spark-examples.jar"
],
"mainClass":"org.apache.spark.examples.SparkPi"
}
}
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{
"name":"projects/project-id/locations/region/batches/batch-id",
"uuid":",uuid",
"createTime":"2021-07-22T17:03:46.393957Z",
"sparkBatch":{
"mainClass":"org.apache.spark.examples.SparkPi",
"args":[
"1000"
],
"jarFileUris":[
"file:///usr/lib/spark/examples/jars/spark-examples.jar"
]
},
"runtimeInfo":{
"outputUri":"gs://dataproc-.../driveroutput"
},
"state":"SUCCEEDED",
"stateTime":"2021-07-22T17:06:30.301789Z",
"creator":"account-email-address",
"runtimeConfig":{
"version":"2.3",
"properties":{
"spark:spark.executor.instances":"2",
"spark:spark.driver.cores":"2",
"spark:spark.executor.cores":"2",
"spark:spark.app.name":"projects/project-id/locations/region/batches/batch-id"
}
},
"environmentConfig":{
"peripheralsConfig":{
"sparkHistoryServerConfig":{
}
}
},
"operation":"projects/project-id/regions/region/operation-id"
}
Estimate workload costs
Managed Service for Apache Spark workloads consume Data Compute Unit (DCU) and shuffle storage resources. For an example that outputs Managed Service for Apache Spark UsageMetrics to estimate workload resource consumption and costs, see Managed Service for Apache Spark pricing.
What's next
Learn about: