Cloud Infrastructure for High-Load APIs: Rate Limiting, Queues, Caching, and Peak Traffic Protection

Anyone responsible for APIs under load knows the unpleasant scenario: traffic suddenly spikes, the application seems to scale, but the system still starts running into bottlenecks in the database, payment service, external integration, or task queue. The problem is often not the number of requests itself, but how many resources each one requires.

Ten thousand simple reference data requests may be handled smoothly if the response is served from cache. A few hundred heavy operations, however, can tie up the database, handlers, and external services. That is why high-load APIs need to be protected not with one large limit, but with multiple layers.

To help an API withstand traffic peaks, protection should be designed as follows:

• Filter out excess and suspicious traffic at the CDN/WAF and API Gateway layers;
• Serve repeated responses from cache;
• Move expensive operations to a queue and process them at a controlled rate;
• Protect the database and external services with timeouts, limits, and graceful degradation;
• Calculate limits not only by request count, but also by operation cost, clients, plans, and priorities.

The main goal is not to accept an unlimited number of requests, but to make API behavior predictable under load. Some traffic should be limited, some served from cache, some deferred, and the most expensive dependencies should not receive the full peak directly.

First, let’s look at exactly where excess traffic should be stopped. Without this protection map, limits, caching, and queues become a set of separate tools rather than a unified API resilience strategy.

Layered API Protection: Where Unwanted Traffic Should Be Stopped

If an API is viewed only as an entry point into an application, protection inevitably moves closer to the code and the database. This is expensive: by the time a request reaches the business logic, it has already consumed a connection, an execution thread, memory, log space, and may have already started a transaction.

Resilience is therefore built as a chain of filters. Early layers make coarser but cheaper decisions. Later layers operate with greater precision because they have more context, but every request incorrectly allowed through at that stage is already more costly.

Why filtering needs to happen earlier

A useful analogy here is an airport. Some people never get inside because of a basic check at the entrance, some quickly pass standard screening, and some are sent for additional inspection. In an API, these zones are represented by the edge layer, the API Gateway, the application, the cache, the queue, workers, and downstream services.

The chain looks like this:

• CDN/WAF — filters out obvious junk and some suspicious spikes.
• API Gateway — validates keys, routes, methods, and basic limits.
• Application — makes decisions based on the user, plan, client, and operation type.
• Cache — takes repeated work off the application and database.
• Queue — moves heavy operations into controlled background processing.
• Workers — execute tasks with limited concurrency.
• Database and external services — remain the final and most expensive part of the system.

The point of this chain is not to ensure that every request eventually reaches the database. On the contrary, most decisions should be made earlier: what to block, what to serve from cache, what to throttle, and what to enqueue.

For this model to work, the layers must not blindly duplicate one another. Each layer has its own role, its own level of context, and its own cost of error.

How responsibilities are distributed across layers

At the CDN and WAF layers, the system filters out bots, suspicious patterns, some DDoS-like traffic spikes, and malformed requests. Cacheable GET requests can also be served there without contacting the origin server. Decisions at this layer may be coarse-grained, but they are inexpensive: it is better to block clearly unnecessary traffic than to pass it through to the backend.

The API Gateway operates closer to the services. It validates keys, routes, methods, environments, and basic access rules. The Gateway does not yet know all the details of a customer's plan or current state, but it is well suited to enforcing technical controls at the entry point: who is allowed to call a route, with which method, and at what volume.

The application makes more precise decisions. At this point, it knows who the user is, what plan they are on, which customer environment they belong to, what operation is being performed, and whether it can be performed right now. This is where business context comes in, but the cost of an error is also higher: the request has already reached the backend.

Further down the stack, caching reduces repeated load on the application and the database, queues move heavy operations into controlled processing, and handlers, connection pools, timeouts, and circuit breakers protect the database and external APIs from cascading failure. A circuit breaker is a mechanism that temporarily stops calls to an unstable dependency when errors or latency exceed a defined threshold.

This layer map shows where excess traffic should be stopped. However, it does not define the admission rules themselves. That is why it is important to examine rate limiting separately: limits at the CDN, API Gateway, and application layers do not replace one another; they address different risks.

Rate limiting at the CDN, API Gateway, and application layers: why you need multiple limits, not just one

With the overall protection map in place, it is easier to treat inbound limits not as a single “N requests per second” rule, but as several filters with different levels of precision. One layer filters out obvious junk traffic, another enforces the API’s technical constraints, and a third accounts for the plan, the customer, and the cost of a specific operation.

The core principle is simple: the closer a layer is to the network edge, the cheaper it is to reject a request. The closer a layer is to the business logic, the more accurate the decision can be. For this reason, limits at the CDN/WAF, API Gateway, and application layers do not replace one another; they address different risks.

For example, the edge layer can quickly limit a suspicious spike by IP address, country, path, or headers. But it usually does not know which plan the customer is on, how many resources the operation costs, or whether a specific user is allowed to perform it right now. Those decisions are made closer to the application.

This is why limits are best distributed across layers. The table below shows the role each layer plays in overall API protection:

The takeaway from this comparison is straightforward: earlier layers protect the infrastructure from unnecessary traffic, while later layers protect business rules and the resources of specific customers. If you rely only on the CDN, the system will have a poor understanding of plans, quotas, and the cost of individual operations. If you rely only on the application, unnecessary traffic will already have consumed some server resources.

Consider a partner that suddenly increases the number of calls made with its API key. The API Gateway detects that the technical limit has been exceeded and returns 429 Too Many Requests along with Retry-After, so the client knows when to retry the request. But this is not enough for full protection: the application still needs to account separately for the plan, the customer context, and the operation type.

This is where the difference between a cheap and an expensive request becomes important. For example, frequent status checks are usually lighter for the system: it can quickly return a short response without touching critical dependencies. Bulk report generation is a completely different case. That kind of operation can consume database capacity, workers, memory, and external services, so it is better to limit it, defer it, or move it to a queue.

This is why rate limiting cannot be built only around requests per second. Rate limits control admission: who is allowed in, how many requests they can make, and under what conditions. But they do not replace caching, queues, timeouts, or graceful degradation. Limits determine which traffic stream enters the system, while the subsequent mechanisms determine how that stream is served without overload.

Caching: Taking Repeated Work Off the API and Database

Rate limiting determines which requests are allowed into the system in the first place. But a request that is allowed through does not have to reach the application and database every time. If, after an email campaign, thousands of users open the same catalog, offer list, or status page, then without caching the system repeats the same work again and again: routing, business logic, database queries, and response generation.

Caching reduces not only user-facing latency, but also the actual cost of serving traffic. The more repeated responses the system can return without accessing costly layers, the lower the load on the application, database, and internal services.

Where cache hits should occur

It is useful to classify caching by where it stops the request. The closer to the network edge a cache hit occurs, the less work reaches the application and database. The closer it is to the application, the more context it can take into account, but the more expensive a cache miss becomes.

This approach helps avoid debates about “where to cache everything.” Different layers solve different problems. A CDN is effective at absorbing large volumes of repeated reads, an API Gateway helps with standard responses at the API level, and an application cache is suitable where business context is required.

However, caching works only when the result can be safely reused. If a user starts a unique or resource-intensive operation—for example, report generation, a bulk export, data recalculation, or payment processing—caching no longer reduces the load. This type of work should not be repeatedly served from memory; it should be executed at a controlled rate. This is what queues and asynchronous processing are used for.

Queues and asynchronous processing: smoothing out a spike instead of passing it on to the database

A queue is useful when an operation does not have to finish immediately within a single HTTP request. If a task takes seconds or minutes, it is inefficient to keep the client, the application thread, the database connection, and external services waiting for the result.

Consider an online bookstore. A user opens a book details page, checks an order status, or views a list of recommendations — these operations should be fast. But generating a large PDF report, performing a bulk catalog export, or processing a batch of payments can be moved to background processing.

In this model, the API accepts the task, places it in a queue, and immediately tells the client that the work has been accepted. The result becomes available later: by task ID, through a notification, or via a webhook. To make the flow below easier to read, let’s define the main responses and elements of an asynchronous API:

During peak periods, a queue protects the most expensive parts of the system: the database, the payment provider, the report generator, or an external integration. Without a queue, the spike immediately puts pressure on these dependencies. With a queue, the incoming rate can be higher than the processing rate, while tasks are still executed gradually and with limited parallelism.

A typical operating model includes:

• Task queue — accepts work that cannot or should not be performed immediately;
• Workers — take tasks from the queue and execute them;
• Queue backlog — the number of tasks waiting to be processed;
• Processing latency — the time between enqueueing a task and producing the result;
• Dead-letter queue — a place for tasks that could not be processed after several attempts.

But a queue does not give the system unlimited capacity. If the bookstore accepts ten thousand report generation tasks and the workers can complete only one thousand within a reasonable time, the problem has not been solved — it has simply been hidden.

This is why workers cannot be scaled indefinitely either. Too many workers can overwhelm the database or an external API with parallel calls. Scaling must account not only for queue length, but also for dependency limits: database connections, the allowed rate of external calls, the cost of the operation, and the client’s priority.

For fast operations, a synchronous response can remain in place: retrieving a status, reading cacheable data, or creating a simple entity. Heavier tasks are better accepted asynchronously: return 202 Accepted, a task ID, the current state, and a way to check the result.

If the queue is full or latency has become too high, a clear 429 or 503 with Retry-After is better than silently accumulating tasks that will not be processed on time anyway.

Peak Load Handling Flow

After limits, caching, and queues, API behavior during a traffic spike can be brought together into a single flow. But this should not be a diagram for its own sake; it should be a clear sequence of decisions: what is filtered out at the entry point, what is served cheaply, what is deferred, and where the system deliberately degrades the response to avoid failing entirely.

EU Cloud Infrastructure You Control

Run production workloads on dedicated resources across EU data centres. Transparent pricing, no hidden costs. Full control over compute, storage, and networking.

Explore Infrastructure

Predictable Virtual Machines for Production

Dedicated vCPUs, consistent performance, no noisy neighbours. Deploy and scale VMs in minutes. Full root access, fixed pricing.

View VM Plans

Up to €22,000 to Migrate Your Workloads

Cut OPEX when migrating to EU infrastructure. Get migration support, cost assessment, and infrastructure credits with no upfront commitment.

Apply for Bonus

Kubernetes with Full Control

Deploy production-ready clusters with control over node configuration, storage, and networking. Predictable costs and direct access to infrastructure.

Deploy Cluster

Consider the same online bookstore. After an email campaign or a seasonal promotion, users open book detail pages in large numbers, check availability, add items to the cart, start exports, and update order statuses. If this entire flow is sent directly to the application and the database, the traffic spike quickly becomes chaotic. The load therefore needs to be handled in stages.

Step 1. Filter out unnecessary traffic at the entry point

First, the system should stop anything that should not reach the application at all: bots, suspicious request patterns, gross limit violations, malformed requests, and clearly unnecessary traffic. This is handled by CDN/WAF and API Gateway.

At this stage, the decision can be coarse but inexpensive. For example, if a single source requests the same pages too frequently or enumerates API routes, it is better to throttle it before it consumes application resources. If a client exceeds the allowed request rate, API Gateway can return 429 Too Many Requests and Retry-After.

The main goal of the first step is not to “save” all legitimate traffic, but to keep obviously unnecessary traffic out. The earlier the system filters out junk, the lower the load on expensive layers.

Step 2. Serve repeatable data without querying the database

After filtering, normal user traffic remains. But even this traffic does not need to reach the database every time. In a bookstore, many users open the same pages for popular books, collections, reference materials, promotions, or recommendation lists.

If this data can be safely reused, it is better to serve it from the cache. This way, the application and the database do not repeat the same work thousands of times. Users get responses faster, and the system preserves resources for operations that genuinely require fresh data.

At this step, it is important not to cache everything indiscriminately. Personal data, order status, access permissions, and rapidly changing inventory levels require caution. The cache should eliminate repetitive work, not create a risk of stale or someone else’s data being served.

Step 3. Defer resource-intensive operations

Some requests cannot simply be served from cache. For example, a user might start a large order export, generate a PDF report, perform a bulk data update, or run an operation that depends on an external service. These tasks should not be executed synchronously as part of a single request.

The API accepts the task, returns 202 Accepted and an identifier, and workers perform the work later with limited concurrency. This does not eliminate the spike, but it does make it manageable: the queue shows how many tasks have accumulated, how quickly they are being processed, and when the delay becomes risky.

If the queue is already overloaded, an explicit rejection is better than silently letting work accumulate. In this case, the API can return 429 or 503 with Retry-After so the client does not generate even more retries.

Step 4. Protect dependencies and define graceful degradation in advance

Even with limits, caching, and queues in place, the most expensive dependencies remain: the database, payment provider, search service, external APIs, and internal handlers. They must be protected with timeouts, concurrency limits, connection pools, and circuit breakers.

If a dependency starts responding slowly or returning errors, the system should not wait indefinitely and accumulate stalled requests. It is better to temporarily restrict some functionality: disable expensive sorting, return aggregates for the previous minute, hide non-priority fields, put a report into a queue instead of generating it synchronously, or throttle low-priority client segments.

In this model, the spike does not disappear, but it stops being chaotic: some requests are rejected, some are served from cache, heavy operations are moved to a queue, and the most expensive dependencies receive limited and predictable load.

However, having a CDN, cache, queue, and handlers does not guarantee resilience. The API can still fail if components scale without a shared cost model, limits, and priorities. Therefore, it is important to examine the common mistakes that cause even a formally “correct” architecture to propagate the spike to the database, external services, or the queue.

Common Mistakes When Scaling APIs

Even if the architecture already includes a CDN, an API Gateway, a cache, a queue, and workers, the system can still fail under peak load. In most cases, the problem is not the lack of a specific tool, but the fact that each component was scaled in isolation: the application separately, the queue separately, the database separately, and the limits separately.

For a high-load API, the key is not simply to “add more resources,” but to understand where the load will shift after each decision. If an online bookstore increases the number of application containers but does not limit database calls, the load spike will not disappear. It will simply reach the weakest point faster.

Mistake 1. Scaling the application while forgetting about its dependencies

The easiest temptation is to add more containers or application instances. It seems logical: if there are more users, you need more servers. The cloud can indeed spin up an additional application tier quickly, but the database, external API, message broker, payment service, and connection pools do not become infinite.

Imagine an engineer named John sees traffic grow after a promotion at a bookstore and triples the number of application instances. For the first few minutes, everything looks better: the application accepts requests faster. But then each new instance starts hitting the same database, payment provider, and recommendation service more aggressively. As a result, the application has scaled, while the bottleneck is now receiving even more concurrent calls.

This kind of scaling does not solve the problem and can sometimes accelerate a failure. Before scaling the application tier, you need to understand the limits of its dependencies: how many connections the database can handle, how many requests the external API accepts, how many tasks the workers can actually process, and where requests should fail or be queued.

Mistake 2. Treating all requests as equal

A limit of “100 requests per second” looks neat only on paper. In practice, one request may simply check an order status, while another may trigger a heavy aggregation, generate a report, query the database, and call an external service.

If the system treats them as equivalent, an active client may technically stay within the limit while still creating a disproportionate load. For example, a bookstore partner may not send thousands of requests per second, but may run bulk exports of sales statistics at scale. By request count, everything looks acceptable; by operation cost, the system is already overloaded.

For limits to be useful, requests need to be compared not only by count, but also by their approximate cost to the system:

Therefore, limits should account not only for frequency, but also for the operation type, route, client, plan, and the approximate cost of the request. A cheap read and heavy report generation should not be governed by the same rule.

Mistake 3. Keeping expensive operations synchronous

Another common mistake is trying to perform a long-running operation directly within a single HTTP request. The user clicks a button, the application starts generating a report, keeps the connection open, waits on the database, calls external services, and hopes to finish before the timeout.

Under peak load, this pattern breaks down quickly. The client does not receive a response, retries the request, the task starts again, load increases, and the system begins duplicating work. For a bookstore, this can mean regenerating the same report, creating duplicate catalog update tasks, or making unnecessary calls to a payment service.

These operations are better moved to an asynchronous contract:

• Accept the task and return an identifier to the client;
• Show the current processing status;
• Provide a way to retrieve the result later;
• Limit repeated starts of the same operation;
• Return a clear rejection if the queue is already overloaded.

This does not make an expensive operation free, but it decouples the user request from the internal processing time.

Mistake 4. Hiding the Problem Behind a Queue

A queue helps smooth out spikes, but it should not become a holding area for tasks the system will not be able to process anyway. Queue length alone says very little without the processing rate and the acceptable waiting time.

For example, a bookstore accepts ten thousand report export tasks. The queue grows, but the processors can handle only a small portion of them. Technically, the API continues to accept tasks, but users wait hours for the result instead of minutes. At that point, the queue is no longer protecting the system; it is hiding promises that cannot be fulfilled.

To understand whether a queue is helping or has already started to mask a problem, you need to look at several metrics at the same time:

If latency exceeds the acceptable threshold, the API should limit the intake of new tasks, return an explicit rejection, or change the contract. Continuing to accept work indefinitely means turning the queue into a store of future problems.

Mistake 5. Caching without consistency rules

Caching can significantly reduce load, but only when it is clear which data can be reused and how long it remains valid. Without such a model, the cache becomes a source of errors.

For example, a bookstore caches product pages for a long time but fails to account for the fact that inventory levels change quickly. A user sees that a book is in stock, adds it to the cart, and then discovers at checkout that the item is no longer available. Caching personalized responses without a strict cache key is even more dangerous: it can accidentally expose one customer's data to another.

Problems also arise during a cache stampede: a popular entry expires, and hundreds of requests simultaneously start recalculating the same object. This is why a cache needs well-defined keys, TTLs, invalidation, and rules for handling popular data under peak load.

Mistake 6. Not setting time and failure budgets

High load is dangerous not only because of the number of requests, but also because of slow dependencies. A single external service may start responding more slowly than usual, and the application will patiently wait. As more requests pile up waiting, worker threads will be exhausted, and a problem in one service will drag down the entire API.

In a bookstore, this could be a recommendation service or a payment provider. If recommendations are unavailable, it is better to temporarily hide the “similar books” section than to delay the entire product page. If the payment service is unstable, it is better to explicitly limit operations than to keep all connections waiting.

This requires basic dependency protection rules:

• Timeouts, so the application does not wait indefinitely for an external service;
• Concurrency limits, so a single dependency does not consume all worker threads;
• Controlled retries, so they do not create a second spike;
• A circuit breaker, so calls to an unstable service can be stopped temporarily;
• A fallback response path, if part of the functionality can be safely degraded.

This way, the API does not try to wait for a problematic service at any cost; it switches to a controlled mode instead.

Mistake 7. Relying on retries without idempotency

When errors occur, clients and internal services will retry requests. That is normal. The problem starts when retrying the same action creates a new order, a new payment, or a new task.

For example, a user places an order for a book, gets an error because of a timeout, and clicks the button again. If the operation is not protected, the system may create two orders or submit the payment operation twice. Under peak load, these situations can occur at scale.

Critical actions require idempotency keys: the system must understand that a repeated request refers to the same operation, not a new one. This is especially important for orders, payments, bulk tasks, and integrations.

Mistake 8. Failing to separate priorities for clients and operations

In an API with multiple clients, a single active client can consume a disproportionate share of resources. This is especially risky if everyone has the same limits and there is no separation by pricing plan, customer environment, or operation type.

For example, a major partner of a bookstore starts a large-scale catalog synchronization. If the system has no quotas or priorities, this task can crowd out regular users: pages load more slowly, order statuses are delayed, and the queue grows.

Fair load distribution is not just a commercial issue. It is a resilience mechanism. The API must be able to determine which clients and operations are critical, what can be deferred, what should be limited, and what must be served first.

The main point of this chapter is that API scaling must account for the cost of operations and the relationships between components. A CDN, cache, queue, handlers, and autoscaling work only when they follow a common model: which requests to admit, which to serve from cache, which to defer, which to limit, and which dependencies to protect first.

Conclusion

A resilient high-load API is not built around the idea of “accepting as many requests as possible,” but around load control. Repeated requests should be served from cache, unnecessary traffic should be filtered out at the earliest layers, and heavy operations should be moved to a queue so they do not put direct pressure on the database and external services.

The cloud helps scale an API, but it does not make a system resilient on its own. If limits, timeouts, queueing rules, caching, and degradation behavior are not defined in advance, a traffic spike will still reach the weakest point. Good architecture does not promise unlimited throughput; it makes system behavior under load predictable.

FAQ

Where is the best place to apply rate limiting: at the CDN, API Gateway, or application level?

At all three levels, but for different purposes. The CDN/WAF filters out obvious and suspicious traffic, the API Gateway limits requests by keys and routes, and the application enforces business limits by user, plan, customer environment, and operation type.

When should an API return "429 Too Many Requests"?

When the client has exceeded the allowed request rate, quota, or business limit, while the system itself remains operational. It is advisable to return a "Retry-After" header so the client can correctly defer the retry. If the entire infrastructure is overloaded or a dependency is temporarily unavailable, "503 Service Unavailable" with the same "Retry-After" header is often more appropriate.

Do heavy operations always need to be sent to a queue?

No. If an operation consistently fits within the response time budget, does not block critical resources, and does not create a risk of cascading failure, it can remain synchronous. A queue is needed when an operation is costly, long-running, depends on external systems, or may be repeated at scale during peak periods.

Which is more important for a high-load API: caching or horizontal scaling?

They are different tools. Horizontal scaling helps increase the number of running application instances, but it does not eliminate the cost of database queries and calls to external services. Caching reduces the amount of repeated work itself. In a real-world architecture, both mechanisms are usually needed, but caching often has a greater impact on repeated "GET" requests.

How can you tell when a queue is no longer helping and is instead hiding the problem?

Look not only at the queue size, but also at processing latency, the percentage of tasks in the dead-letter queue, the arrival rate, and the completion rate. If latency keeps growing steadily while new tasks continue to be accepted without limits, the queue has stopped smoothing out spikes and has started accumulating commitments that cannot be fulfilled.

Sources

1. Cloudflare Docs — Rate limiting rules

2. Microsoft Azure Architecture Center — Queue-Based Load Leveling pattern

3. AWS Well-Architected Framework — REL05-BP02 Throttle requests

4. Google Cloud — Rate Limiting | Service Infrastructure

5. AWS Builders’ Library — Making retries safe with idempotent APIs