Set up a project

This document describes how to set up a Google Cloud project and a Compute Engine virtual machine (VM).

Set up a Google Cloud project

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Roles required to select or create a project

   • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
   • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.
   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Verify that billing is enabled for your Google Cloud project.

3. Enable the Google Cloud Dataproc API.

   Roles required to enable APIs

   To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

   Enable the API

4. Install the Google Cloud CLI.

5. Configure the gcloud CLI to use your federated identity.

   For more information, see Sign in to the gcloud CLI with your federated identity.

6. To initialize the gcloud CLI, run the following command:

   gcloud init

Note: When you enable the Managed Service for Apache Spark API, the Managed Service for Apache Spark Control API is also enabled. Disabling the Managed Service for Apache Spark API doesn't disable the Managed Service for Apache Spark Control API. You must separately disable the Managed Service for Apache Spark Control API.

Do you Need Credentials? You need credentials only if you plan to directly call Managed Service for Apache Spark APIs from your application or from a browser. If you plan on using the Google Cloud console or the Google Cloud CLI to create and manage clusters and to submit jobs, you only need to be logged in to the SDK with the gcloud init command.

Optional: Set up a VM

If you plan to run gcloud CLI from a Compute Engine virtual machine (VM) instance, the VM must be set up to access Google Cloud services.

To test whether your VM is set up correctly, run gcloud compute instances describe YOUR_VM_INSTANCE_NAME from the command line. A service account must be listed that includes the https://www.googleapis.com/auth/cloud-platform scope. For more information on setting up a VM, see Create a VM that uses a user-managed service account.

Use Cloud Shell to run gcloud CLI commands. As an alternative to setting up your own VM to run SDK commands, open a Cloud Shell session to automatically provision a VM with the necessary scopes, and then run gcloud CLI commands from Cloud Shell.

---

Next Step: Create a Cluster