Configure access to a source: URL list

This page describes the requirements for transferring data from a publicly accessible list of HTTP/HTTPS URLs to Cloud Storage.

Transfers from a URL list are agentless transfers managed directly by Storage Transfer Service.

Prerequisites

Before creating a transfer from a URL list, ensure that you have configured the following:

  • User permissions: Ensure your user account or user-managed service account has permission to create transfer jobs. See Required user permissions.
  • Destination bucket permissions: Grant the required permissions on your destination Cloud Storage bucket to the Storage Transfer Service service agent. See Configure access to a sink: Cloud Storage.
  • URL list stored in Cloud Storage (optional): If you store your URL list file in a Cloud Storage bucket rather than a public HTTP/HTTPS URL, grant the Storage Transfer Service service agent roles/storage.objectViewer and roles/storage.legacyBucketReader on that bucket. See Configure access to a source: Cloud Storage.

Source URL requirements

To allow Storage Transfer Service to read your source data, ensure that the following requirements are met:

  • Public accessibility:

    • The URL list file (TSV format) must be accessible from a URL beginning with either http:// or https://, or stored in a Cloud Storage bucket.
    • Each object URL listed in the file must be publicly accessible via http:// or https://.

  • Web server configuration:

    • The web server hosting each object must support HTTP Range requests to allow parallel chunked transfers.
    • The web server must return a valid Content-Length header in each response.
    • The web server's robots.txt file must allow access to the URLs for the Google-Storage-Transfer-Service user agent or all user agents (*).

  • Object size and checksum validation:

    • If you specify object sizes or MD5 checksums in your URL list, ensure that the values match the actual object payloads returned by the server. If the size or checksum does not match, the transfer of that object fails.

What's next