Cloud Tasks lets you dispatch large volumes of asynchronous tasks while controlling the rate at which traffic reaches your queues and downstream services. By applying gradual ramp-up patterns and rate controls, you can scale workloads smoothly without triggering queue latency spikes or target overload when traffic climbs faster than infrastructure can adapt.
Overview
This document provides guidelines on the best practices for maintaining high
Cloud Tasks performance in high-traffic queues. A high-TPS
queue is a queue that has 500 created or dispatched tasks per second (TPS)
or more. A high-TPS queue group is a contiguous set of queues, for example
[queue0001, queue0002, …, queue0099], that have at least 2,000 tasks
created or dispatched per second in total. You can view the historical TPS of a
queue or group of queues by using the
Cloud Monitoring metrics,
api/request_count for CreateTask operations and queue/task_attempt_count
for task attempts. High-traffic queues and queue groups are prone to two
different broad classes of failures:
Queue overload occurs when task creation and dispatch to an individual queue or queue group increases faster than the queue infrastructure can adapt. Google's infrastructure uses adaptive layers that dynamically reallocate load based on traffic patterns, and this adaptation takes time. Similarly, target overload occurs when the task dispatch rate causes traffic spikes in the downstream target infrastructure. In both cases, follow a 500/50/5 pattern: when scaling beyond 500 TPS, increase traffic by no more than 50% every 5 minutes.
Separate from the 500/50/5 pattern for high-TPS workloads, initial queue ramp-up affects all queues when they start from a cold or idle state, including queues configured for moderate dispatch rates. This document reviews different scenarios that can introduce scaling risks and gives examples of how to manage queue ramp-up and apply the 500/50/5 pattern.
Initial queue ramp-up
The 500/50/5 pattern applies to high-TPS queues that already operate at a steady state. Initial queue ramp-up is a separate system mechanism that applies to all queues, including queues configured for moderate dispatch rates such as 20-100 TPS, as they transition from a cold or idle state to steady-state operation.
A queue enters a cold or idle state in any of the following situations:
The queue is newly created or resumed after being paused.
The queue has been inactive for even a few minutes without active dispatches.
The queue is recovering after backing off from downstream HTTP
429or5xxerror responses. For more information about how error backoff affects dispatch rates, see Backoff errors and enforced rates.
When tasks arrive at a cold or idle queue, Cloud Tasks doesn't
immediately dispatch tasks at the configured
max_dispatches_per_second
rate. To protect downstream targets from sudden traffic spikes,
Cloud Tasks starts dispatching at a low baseline rate, such as
around 1 TPS. It then progressively steps up the enforced dispatch rate across
periodic evaluation intervals. The enforced rate continues to step up as long
as queue demand remains saturated and target handlers return HTTP 2xx
responses. Depending on your target rate, reaching high
max_dispatches_per_second values can take several minutes.
If your workload sends spiky traffic separated by brief idle periods of a few minutes or more, the queue resets to a cold state between bursts and repeats this ramp-up process on each burst. To minimize ramp-up delays for burst-sensitive workloads, follow these best practices:
Maintain a steady flow of baseline traffic to keep latency-sensitive queues warm between bursts.
Reuse existing queues rather than frequently deleting and recreating queues.
Provision sufficient downstream target capacity so that handlers return HTTP
2xxresponses and avoid returning HTTP429or5xxerror responses that reset the enforced dispatch rate.
Diagnose queue throttling
If your queue dispatches tasks slower than your configured
max_dispatches_per_second
rate, check whether the bottleneck comes from cold or idle queue ramp-up,
target congestion backoff, or concurrency limits.
Cold or idle queue ramp-up
Initial queue ramp-up is likely the cause if all of the following conditions apply:
- The queue was recently created, resumed after being paused, or idle for a few minutes before traffic arrived.
- Task attempts succeed without target errors, showing
response_code="OK"in the Monitoring metriccloudtasks.googleapis.com/queue/task_attempt_count. - In-flight dispatches remain below
max_concurrent_dispatches, whilecloudtasks.googleapis.com/queue/depthorcloudtasks.googleapis.com/queue/task_attempt_delaysis elevated. At the same time, the dispatch rate incloudtasks.googleapis.com/queue/task_attempt_countsteps up progressively across periodic evaluation intervals.
For more information about Monitoring metrics, see Send metrics to Monitoring.
Target overload and congestion backoff
Cloud Tasks dynamically reduces dispatch rates if your downstream
target returns HTTP 429 Too Many Requests or HTTP 5xx error responses, such
as HTTP 503 Service Unavailable. This congestion backoff prevents target
overload.
To check whether target errors are causing Cloud Tasks to
throttle dispatches, view the Monitoring metric
cloudtasks.googleapis.com/queue/task_attempt_count grouped by the
response_code label. If queue logging is enabled, you can also inspect the
attempt response logs in Logs Explorer:
resource.type="cloud_tasks_queue"
jsonPayload.attemptResponseLog.status!="OK"
In both response_code and jsonPayload.attemptResponseLog.status,
Cloud Tasks records canonical gRPC status codes instead of
numeric HTTP status codes. For example, HTTP 429 maps to
RESOURCE_EXHAUSTED, HTTP 503 maps to UNAVAILABLE, HTTP 500 maps to
INTERNAL, and HTTP 504 or a timeout maps to DEADLINE_EXCEEDED.
If you observe RESOURCE_EXHAUSTED, UNAVAILABLE, or other server error
statuses from the target, scale up your target service instances or increase
target concurrency before increasing queue dispatch limits.
For more information about enabling queue logs, see Use Cloud Logging with queues.
Concurrency bottleneck
According to Little's law, downstream target latency bounds your maximum dispatch throughput:
Max Throughput (tasks/sec) = min(
max_dispatches_per_second,
max_concurrent_dispatches / Downstream Latency in seconds
)
For example, if downstream execution latency is 2 seconds and
max_concurrent_dispatches is set to 100,
throughput is bounded to 50 tasks per second, even if
max_dispatches_per_second is configured to
500.
To diagnose whether concurrency is the bottleneck, divide
max_concurrent_dispatches by the target
response latency recorded in the
jsonPayload.attemptResponseLog.attemptDuration field in
Logging. Compare this calculated threshold against your actual
dispatch rate in the Monitoring metric
cloudtasks.googleapis.com/queue/task_attempt_count. If actual dispatches
plateau near this threshold while cloudtasks.googleapis.com/queue/depth or
cloudtasks.googleapis.com/queue/task_attempt_delays grows, take one of the
following actions:
- If your downstream target has spare capacity, such as when tasks spend most
of their duration waiting on external I/O, increase
max_concurrent_dispatches. - If your downstream target is already under high resource utilization,
increasing
max_concurrent_dispatchescan worsen target overload. Instead, scale out your target instances or reduce handler execution latency.
Queue overload
Queues or queue groups can become overloaded any time traffic increases suddenly. As a result, these queues can experience:
- Increased task creation latency
- Increased task creation error rate
- Reduced dispatch rate
To defend against this, establish controls in any situation where the create or dispatch rate of a queue or queue group can spike suddenly. Start with a maximum of 500 operations per second to a cold queue or queue group, and then increase traffic by 50% every 5 minutes. In theory, you can grow to 740K operations per second after 90 minutes using this ramp-up schedule. There are a number of circumstances in which this can occur.
For example:
- Launching new features that make heavy use of Cloud Tasks
- Moving traffic between queues
- Rebalancing traffic across more or fewer queues
- Running batch jobs that inject large numbers of tasks
In these cases and others, follow the 500/50/5 pattern.
Use App Engine traffic splitting
If an App Engine app creates the tasks, you can take advantage of App Engine traffic splitting (App Engine standard environment or App Engine flexible environment) to smooth traffic increases. By splitting traffic between versions (standard environment or flexible environment), you can ramp up rate-managed requests over time to protect queue health. As an example, consider the case of ramping up traffic to a newly expanded queue group: let [queue0000, queue0199] be a sequence of high-TPS queues that receive 100,000 TPS creations in total at peak.
Let [queue0200, queue0399] be a sequence of new queues. After you shift all traffic, the number of queues in the sequence doubles and the new queue range receives 50% of the sequence's total traffic.
When deploying the version that increases the number of queues, gradually ramp up traffic to the new version, and thus the new queues, using traffic splitting:
- Start shifting 1% of the traffic to the new release. For example, 50% of 1% of 100,000 TPS yields 500 TPS to the set of new queues.
- Every 5 minutes, increase the traffic sent to the new release by 50%, as detailed in the following table:
| Minutes since start of the deployment | % of total traffic shifted to the new version | % of total traffic to the new queues | % of total traffic to the old queues |
|---|---|---|---|
| 0 | 1.0 | 0.5 | 99.5 |
| 5 | 1.5 | 0.75 | 99.25 |
| 10 | 2.3 | 1.15 | 98.85 |
| 15 | 3.4 | 1.7 | 98.3 |
| 20 | 5.1 | 2.55 | 97.45 |
| 25 | 7.6 | 3.8 | 96.2 |
| 30 | 11.4 | 5.7 | 94.3 |
| 35 | 17.1 | 8.55 | 91.45 |
| 40 | 25.6 | 12.8 | 87.2 |
| 45 | 38.4 | 19.2 | 80.8 |
| 50 | 57.7 | 28.85 | 71.15 |
| 55 | 86.5 | 43.25 | 56.75 |
| 60 | 100 | 50 | 50 |
Release-driven traffic spikes
When launching a release that significantly increases traffic to a queue or queue group, gradual rollout is, again, an important mechanism for smoothing the increases. Gradually roll out your instances so that the initial launch doesn't exceed 500 total operations per second to the new queues, increasing by no more than 50% every 5 minutes.
New high-TPS queues or queue groups
Newly created queues are especially vulnerable. Groups of queues, for example [queue0000, queue0001, …, queue0199], are just as sensitive as single queues during the initial rollout stages. For these queues, gradual rollout is an important strategy. Launch new or updated services that create high-TPS queues or queue groups in stages. Keep the initial load below 500 TPS, and stage increases of 50% or less at least 5 minutes apart.
Newly expanded queue groups
When increasing the total capacity of a queue group, for example expanding [queue0000-queue0199 to queue0000-queue0399], follow the 500/50/5 pattern. For rollout procedures, new queue groups behave no differently than individual queues. Apply the 500/50/5 pattern to the new group as a whole, not just to individual queues within the group.
For these queue group expansions, gradual rollout is an important strategy. If the source of your traffic is App Engine, you can use traffic splitting. When migrating your service to add tasks to the increased number of queues, gradually roll out your instances so that the initial launch doesn't exceed 500 total operations to the new queues, and increase traffic by no more than 50% every 5 minutes.
Emergency queue group expansion
On occasion, you might want to expand an existing queue group, for example because you expect tasks to be added to the queue group faster than the group can dispatch them. If you distribute the names of the new queues evenly among your existing queue names in lexicographical order, you can send traffic immediately to those queues. Ensure that there are no more than 50% new interleaved queues and that the traffic to each queue is less than 500 TPS. This method is an alternative to using traffic splitting and gradual rollout.
To achieve this type of interleaved naming, append a suffix to queues ending in even numbers. For example, if you have 200 existing queues [queue0000-queue0199] and want to create 100 new queues, choose [queue0000a, queue0002a, queue0004a, …, queue0198a] as the new queue names, instead of [queue0200-queue0299].
If you need a further increase, you can still interleave up to 50% more queues every 5 minutes.
Large-scale or batch task enqueues
When you need to add a large number of tasks, for example millions or billions, a double-injection pattern can be useful. Instead of creating tasks from a single job, use an injector queue. Each task added to the injector queue fans out and adds 100 tasks to the relevant queue or queue group. You can speed up the injector queue over time—for example, start at 5 TPS to generate 500 TPS in the target queue or queue group, and then increase the rate by 50% every 5 minutes.
Named tasks
When you create a new task, Cloud Tasks assigns the task a unique
name by default. You can assign your own name to a task by using the name
parameter. However, this introduces significant performance overhead, resulting
in increased latencies and potentially increased error rates associated with
named tasks. Sequential task names, such as timestamps, can magnify this
overhead significantly.
If you assign your own names, use a well-distributed prefix for task names, such
as a hash of the contents. For more information about naming a task, see the
Task.name field reference.
Target overload
Cloud Tasks can overload downstream services and resources that you use, such as App Engine, Firestore in Datastore mode (Datastore), and your network, if queue dispatches spike suddenly. If a backlog of tasks has accumulated, then unpausing those queues can potentially overload these services. To defend against target overload, follow the 500/50/5 pattern: if a queue dispatches more than 500 TPS, increase traffic triggered by a queue by no more than 50% every 5 minutes. To monitor your traffic increases proactively, use Monitoring metrics. You can use Monitoring alerts to detect potentially dangerous situations.
Unpause or resume high-TPS queues
When you resume a paused queue, Cloud Tasks applies
initial queue ramp-up and gradually steps up the dispatch rate from
a low baseline toward the queue's configured
max_dispatches_per_second
rate. However, if a large backlog of tasks accumulates in a high-TPS queue or
across multiple queues that you resume at the same time, the dispatch rate can
still climb faster than downstream target services can scale. To prevent target
overload when resuming queues with a large backlog, stagger when you resume
multiple queues. Alternatively, temporarily lower the
max_dispatches_per_second
rate and increase it gradually by following the 500/50/5 pattern. For more
information about how Cloud Tasks throttles and ramps up
dispatches when queues are resumed, see Backoff errors and enforced
rates.
Bulk scheduled tasks
Large numbers of tasks that are scheduled to dispatch at the same time can also introduce a risk of target overloading. If you need to start a large number of tasks at once, consider using queue rate controls to increase the dispatch rate gradually or provisioning target capacity in advance.
Increased fan-out
When updating services that Cloud Tasks calls, increasing the
number of remote calls can create production risks. For example, say the tasks
in a high-TPS queue call the handler /task-foo. A new release can
significantly increase the cost of calling /task-foo if, for example, that new
release adds several expensive Datastore calls to the handler.
The net result of such a release is a massive increase in
Datastore traffic that's immediately related to changes in user
traffic. Use gradual rollout or traffic splitting to manage ramp-up.
Retries
Your code can retry on failure when making Cloud Tasks API calls. However, when a significant proportion of requests fail with server-side errors, a high retry rate can further overload your queues and slow recovery. Cap the amount of outgoing traffic if your client detects that a significant proportion of requests are failing with server-side errors. For example, you can use the Adaptive Throttling algorithm described in the Handling Overload chapter of the Site Reliability Engineering book. Google's gRPC client libraries implement a variation of this algorithm.