We've reorganized our documentation navigation structure to align directly with your operational workflows. See the release notes and the walkthrough video for more information.

Upgrade cloud storage data feeds to the v2 connector framework

Supported in:

Google secops SIEM

To improve the reliability, scalability, and performance of data ingestion pipelines, we are upgrading our cloud storage data feeds to the new v2 connector framework. This new framework uses Google Cloud Storage Transfer Service (STS).

Benefits of the v2 connector framework

Upgrading to the v2 connector framework unlocks significant improvements for your data feeds, including the following:

Enhanced polling efficiency: Vastly improved efficiency when polling from sources like AWS.
Improved reliability and security: Real-time source credential validation makes sure that you have secure and reliable connections.
Expanded identity support: Built-in support for AWS federated identity.

Migration timelines

We are discontinuing the v1 connector framework for the following legacy connectors: Cloud Storage, Amazon S3, Amazon SQS, and Azure Blob Storage.

Take note of the following key milestones:

October 1, 2026 - End of support: No further updates are provided to v1 feeds, and only best-effort support is available. We highly encourage migrating and taking actions so that Google can migrate your feeds before this date to avoid disruptions.
March 15, 2027 - Permanent deactivation: Legacy connectors reach their end of life. Any data feeds still using v1 connectors after this date cease to function and return an error message.

Automatic migration and required actions

To facilitate a seamless transition, we provide automatic migration services for all active data feeds using legacy v1 connectors.

Depending on your source type, you must take the following actions to make sure that the automatic migration is successful:

Cloud Storage: Grant the specific permissions to your new service account as described in the grant access to the service account documentation.
Amazon S3: If your environment uses IP allowlisting, you must add the STS IP ranges to your bucket policy. Follow the steps to enable STS access for Amazon S3.
Amazon SQS: The migration only supports SQS queues that contain messages from a single Amazon S3 bucket. If your queue receives messages from multiple buckets, create a separate feed for each bucket. Verify that your SQS queue is configured correctly and that the access credentials for both the SQS queue and the Amazon S3 bucket are identical.
Microsoft Azure: If you use firewalls or virtual networks, allow the STS IP ranges by following the Azure STS access instructions.
Failed feeds: Review your current feeds and fix any that are failing due to incorrect credentials so that they can migrate successfully.

Safety and reliability

The automatic migration process aims to prevent data loss. The migration features strict, built-in guardrails. Minimal data duplication during the brief duration of the migration window is expected.

Need more help? Get answers from Community members and Google SecOps professionals.