Configure access to a sink: file system

When transferring data to an on-premises or private cloud file system (such as POSIX, NFS, or SMB), you must configure the following:

  • Local file system permissions: Grant the destination transfer agents OS-level write access to your target directories.
  • Intermediate bucket permissions (file-to-file only): Grant IAM roles on the intermediate Cloud Storage bucket used to stage files between file systems.

Prerequisites

  • User permissions: Ensure your user account or user-managed service account has permission to configure transfer jobs and agent pools. See Required user permissions.
  • Agent-based project-level permissions: Ensure you have configured the required project-level roles for your service agent and transfer agent account. See Agent-based project-level permissions.

Local target file system permissions

Transfer agents write files directly to your host file system. To enable successful writes:

  1. OS user permissions: Ensure the operating system user running the container process has read, write, and execute permissions on the destination directory tree.

  2. Mount directories: When installing your destination agents, mount the target directory path into the container using the --mount-directories flag:

    gcloud transfer agents install --pool=DESTINATION_POOL_NAME \
 --mount-directories=/path/to/destination/dir

Intermediate bucket permissions (file-to-file transfers)

Transfers between two file systems use an intermediate Cloud Storage bucket to stage files during transfer.

The following predefined roles grant the required access on the intermediate bucket to the service agent:

  • Storage Object Creator (roles/storage.objectCreator)
  • Storage Object Viewer (roles/storage.objectViewer)
  • Storage Legacy Bucket Reader (roles/storage.legacyBucketReader)

The following role grants the required access on the intermediate bucket to the transfer agent account(s):

  • Storage Object User (roles/storage.objectUser)

Grant these roles on the intermediate bucket:

Console

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. Click the Bucket overflow menu () for the intermediate bucket and select Edit access.
  3. Click + Add principal.
  4. In New principals, enter your service agent email.
  5. Select Storage Object Creator, Storage Object Viewer, and Storage Legacy Bucket Reader from the Select a role drop-down menus.
  6. Click + Add principal again.
  7. Enter the email address of the transfer agent account(s) used for your source and destination agent pools.
  8. Select Storage Object User from the Select a role drop-down menu.
  9. Click Save.

gcloud

  1. Grant logging and metadata roles to the service agent:

    gcloud storage buckets add-iam-policy-binding gs://INTERMEDIATE_BUCKET_NAME \
    --member=serviceAccount:SERVICE_AGENT_EMAIL --role=roles/storage.objectCreator
     
    gcloud storage buckets add-iam-policy-binding gs://INTERMEDIATE_BUCKET_NAME \
    --member=serviceAccount:SERVICE_AGENT_EMAIL --role=roles/storage.objectViewer
     
    gcloud storage buckets add-iam-policy-binding gs://INTERMEDIATE_BUCKET_NAME \
    --member=serviceAccount:SERVICE_AGENT_EMAIL --role=roles/storage.legacyBucketReader

  2. Grant object management roles to the transfer agent account(s) for both the source and destination agent pools:

    gcloud storage buckets add-iam-policy-binding gs://INTERMEDIATE_BUCKET_NAME \
    --member=serviceAccount_OR_user:TRANSFER_AGENT_ACCOUNT --role=roles/storage.objectUser

    Replace:

    • INTERMEDIATE_BUCKET_NAME: The name of your intermediate staging bucket.
    • SERVICE_AGENT_EMAIL: The service agent email.
    • serviceAccount_OR_user: Specify either serviceAccount if your transfer agent account is a service account, or user if your agents were installed with a user account.
    • TRANSFER_AGENT_ACCOUNT: The email of the account running the transfer agents. Repeat this command if your source and destination agent pools use different accounts.

REST API

  1. Generate an access token to authenticate your request. If you are using the gcloud CLI, run:

    gcloud auth print-access-token

  2. Retrieve the existing bucket IAM policy (GET https://storage./storage/v1/b/<var>INTERMEDIATE_BUCKET_NAME</var>/iam), append the following bindings to its bindings array, and save the merged policy as intermediate_policy.json:

    {
  "bindings": [
    {
      "role": "roles/storage.objectCreator",
      "members": ["serviceAccount:SERVICE_AGENT_EMAIL"]
    },
    {
      "role": "roles/storage.objectViewer",
      "members": ["serviceAccount:SERVICE_AGENT_EMAIL"]
    },
    {
      "role": "roles/storage.legacyBucketReader",
      "members": ["serviceAccount:SERVICE_AGENT_EMAIL"]
    },
    {
      "role": "roles/storage.objectUser",
      "members": ["serviceAccount_OR_user:TRANSFER_AGENT_ACCOUNT"]
    }
  ]
}

    Replace:

    • SERVICE_AGENT_EMAIL: The service agent email.
    • serviceAccount_OR_user: Specify either serviceAccount if your transfer agent account is a service account, or user if your agents were installed with a user account.
    • TRANSFER_AGENT_ACCOUNT: The email of the account that installed your transfer agents. If your source and destination agent pools use different accounts, include both accounts in the members array for roles/storage.objectUser.

  3. Update the intermediate bucket policy via curl:

    curl -X PUT --data-binary @intermediate_policy.json \
    -H "Authorization: Bearer OAUTH2_TOKEN" \
    -H "Content-Type: application/json" \
    "https://storage./storage/v1/b/INTERMEDIATE_BUCKET_NAME/iam"

    Replace:

    • OAUTH2_TOKEN: The OAuth 2.0 access token generated in step 1.
    • INTERMEDIATE_BUCKET_NAME: The name of your intermediate bucket.

For more information on file-to-file architectures, see File system to file system transfers.

Performance and optimization

  • To maximize throughput when transferring to large file systems, install multiple agents in your destination agent pool (recommended minimum of 3). See Best practices for transfer agents.
  • For large files in file-to-file transfers, consider enabling multipart uploads to the intermediate bucket. When using Storage Object User (roles/storage.objectUser), required multipart upload permissions (storage.multipartUploads.*) are included automatically.

What's next