To transfer data from an S3-compatible object storage provider (such as Ceph, MinIO, Cloudflare R2, or Wasabi), you must obtain API credentials from your storage provider and configure Storage Transfer Service agents with access to the source bucket.
Prerequisites
Before configuring your transfer, ensure you have set up the following:
- User permissions: Ensure your user account or user-managed service account has permission to configure transfers and agent pools. See Required user permissions.
- Agent-based project-level permissions: Ensure you have configured the required project-level roles for your service agent and transfer agent account. See Agent-based project-level permissions.
- Destination bucket permissions: Grant the required permissions on your destination Cloud Storage bucket. See Configure access to a sink: Cloud Storage.
Required source permissions
To authenticate to your S3-compatible storage provider, obtain an access key ID and secret access key from your provider's management console.
Ensure the access key credentials have the following minimum permissions on the source bucket:
| Permission | Description | Required |
|---|---|---|
s3:ListBucket |
Allows Storage Transfer Service to list objects in the source bucket. | Always required. |
s3:GetObject |
Allows Storage Transfer Service to read objects from the source bucket. | Always required. |
s3:DeleteObject |
Allows Storage Transfer Service to delete source objects after copying. | Required if deleteObjectsFromSourceAfterTransfer is enabled. |
You provide these credentials as environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) when starting your transfer agents.
What's next
- Learn how to Install transfer agents.
- Learn how to Manage agent pools.
- Learn how to Transfer from S3-compatible storage to Cloud Storage.
- Learn how to Configure access to a sink: Cloud Storage.