Message Queues in Cloud Infrastructure: RabbitMQ, Kafka, Redis Streams, or a Managed Queue

A message queue should be chosen based on the exchange model, not the name of the technology. First, you need to understand what the system is actually sending: a task for a single worker, an event for multiple independent consumers, or a message in a managed cloud service.

The decision logic is as follows:

• RabbitMQ is suitable for task queues, routing, processing acknowledgments, and worker monitoring;
  
• Kafka is the better choice when you need an event log, high throughput, independent consumer groups, and the ability to reread older events;
  
• Redis Streams is suitable for lighter internal streams, especially if Redis is already in the infrastructure;
  
• Managed queue is beneficial when reducing operational overhead is more important and you are willing to accept the standard limitations of the cloud service.

The key question is not “which broker is better,” but what should happen to the message after it has been processed: should it be deleted, should the task be acknowledged, should the read position be committed, or should the event be retained for reuse?

Nor does reliability appear automatically simply because a broker has been chosen. It depends on processing acknowledgments, retries, DLQs, consumer idempotency, latency monitoring, and a clear response to queue growth. If a worker can receive the same message more than once, it must not charge money twice, create two orders, or send the customer multiple identical emails.

Use Case First, Technology Second

When services start communicating asynchronously, choosing a broker quickly stops being a matter of preference. You need to decouple components, handle load spikes, avoid losing a payment event, prevent sending a customer two identical emails, and know who will be paged if a queue starts growing overnight.

RabbitMQ, Kafka, Redis Streams, and managed queues overlap in their use cases, but they address different constraints. One tool is better suited to distributing tasks to workers, another to storing an event stream, a third to lightweight internal queues, and a fourth to reducing operational overhead.

In practice, the decision depends on several questions:

• Whether old events need to be replayed or whether it is enough to process a task once;
• Whether complex message routing between services is required;
• How critical ordering, throughput, and redelivery are;
• Whether the team is ready to operate the broker itself or whether it is more efficient to shift the infrastructure to a cloud provider.

From there, it is better to make the choice based not on the technology name, but on the communication model: a task queue, an event log, or a managed service. First, you need to distinguish these models from one another, then validate their behavior under failure conditions, and only after that narrow the options down to a practical choice.

Three exchange models: task, event, or managed queue

Before comparing RabbitMQ, Kafka, Redis Streams, and a managed queue, you need to define the exchange model, not the product. Otherwise, you end up putting different architectural roles on the same line: a tool for distributing tasks to workers, a store for an ordered sequence of events, and a cloud delivery service with delegated operations.

The same phrase, “send a message,” can mean different things. In one case, the message must be delivered to a single handler and disappear after successful processing. In another, the event must be retained so that billing, analytics, and anti-fraud can read it independently. In yet another, the main goal is not to operate a broker manually, but to get ready-to-use cloud message delivery with minimal operational overhead.

At a basic level, three models should be distinguished:

That is why the choice starts not with “RabbitMQ or Kafka,” but with “what should happen to the message.” If a task needs to be passed to one handler and removed after success, a task queue is the closer fit. If you need to retain a history of actions and allow multiple services to read it independently, that is an event log. If the key issue is the cost of maintaining infrastructure, on-call shifts, and upgrades, a managed queue becomes part of the decision.

After choosing the model, you can move on to the processing mechanics: exactly where the system records that a message has been processed, and what happens if the handler fails before acknowledging it. This is where the most unpleasant failures begin.

How message processing is recorded

In normal operation, everything looks simple: the producer sends a message, and the consumer processes it. But the real point of risk lies between doing the work and recording the result. This is where the system decides what to do next: mark the task complete, return it to another handler, commit the read position, or leave the record in the log.

The key mechanisms differ:

• ack/nack — acknowledgment of successful processing or rejection of it;
• offset commit — committing the read position in Kafka;
• pending entries — unacknowledged entries in Redis Streams;
• visibility timeout — the period during which a message is hidden from other consumers in the managed queue;
• DLQ — an error queue for messages that could not be processed normally.

The main message path and the point where processing is recorded look like this:

The same word, “message,” behaves differently in each case. In RabbitMQ, it is like a task that must be delivered to a single handler and marked complete after an ack. In Kafka, it is a record in a log: the consumer does not take it away permanently, but reads it and advances its position. In Redis Streams, a record can remain stuck among pending entries until ACK. In an SQS-like managed queue, after being read, a message is temporarily hidden and is deleted only after successful processing.

This acknowledgement point needs to be understood before discussing failures. Most problems arise precisely at the boundary between “the work is done” and “the system has recorded that the work is done”: the handler may have crashed, the acknowledgement may not have arrived, and the broker may deliver the message again.

Where RabbitMQ, Kafka, Redis Streams, and managed queues excel

The point at which processing is acknowledged directly affects a broker’s strengths. If a message needs to be delivered to a single worker and marked complete after successful processing, that is one scenario. If an event must be stored in a log and read independently by different services, that is another. If the team does not want to operate the broker itself, a managed queue becomes a candidate.

For this reason, it is better to compare the options not by asking which technology is more powerful, but by looking at the role the broker plays in the system: task queue, event log, lightweight stream, or managed delivery.

RabbitMQ: Tasks, Routing, and Worker Control

RabbitMQ is best suited for tasks that need to be handed off to one of the workers and marked complete after successful processing. Typical examples include generating a PDF invoice, sending an email, processing a product image, and calling an external API.

RabbitMQ’s strengths include exchange-based routing, competing consumers, prefetch for limiting the number of messages in progress, processing acknowledgments, and error queues. It is useful when delivery, worker control, and routing problematic messages are important.

RabbitMQ is generally not a good choice as a long-term event log. Systems where event storage and replay are the core model are better suited for that purpose.

Kafka: Event Log and Replay

Kafka is a strong fit when a message represents an event rather than a one-off task. Payment events, order changes, or user actions can be stored in a log and consumed by different groups, such as billing, analytics, auditing, and anti-fraud.

The offset model provides high throughput, independent consumption, and the ability to replay events. This is useful when different services need to process the same event history at their own pace.

However, Kafka is often overkill for simple background jobs. If all you need is to send an email or recalculate a small report, the infrastructure and processing model may be more complex than the task itself.

EU Cloud Infrastructure You Control

Run production workloads on dedicated resources across EU data centres. Transparent pricing, no hidden costs. Full control over compute, storage, and networking.

Explore Infrastructure

Predictable Virtual Machines for Production

Dedicated vCPUs, consistent performance, no noisy neighbours. Deploy and scale VMs in minutes. Full root access, fixed pricing.

View VM Plans

Up to €22,000 to Migrate Your Workloads

Cut OPEX when migrating to EU infrastructure. Get migration support, cost assessment, and infrastructure credits with no upfront commitment.

Apply for Bonus

Kubernetes with Full Control

Deploy production-ready clusters with control over node configuration, storage, and networking. Predictable costs and direct access to infrastructure.

Deploy Cluster

Redis Streams: lightweight internal streams

Redis Streams is a good fit as a lighter-weight streaming mechanism, especially if Redis is already part of the infrastructure. It helps organize a moderate internal event stream, consumer groups, and tracking of unacknowledged entries without introducing a separate heavyweight broker.

The main limitation is the cleanup policy. The stream should not be allowed to grow indefinitely; otherwise, Redis starts being used as long-term event storage, which is not always a safe architectural choice.

Managed queue: less operational overhead, more service constraints

A managed queue is a good fit when reducing operational overhead matters more than fine-tuning the broker. Serverless handlers, irregular workloads, a small team, standard retries, and a dead-letter queue are typical managed queue use cases.

The team gets built-in availability, updates, and some scaling handled by the provider. In return, it accepts quotas, service-specific behavior, limits on retention periods, operation costs, and a reduced degree of control.

This map shows the normal operating mode. But the choice cannot be considered complete without checking failure scenarios: what happens if a handler crashes, an acknowledgment is lost, a message is redelivered, the queue grows, or the stream overflows.

What Happens When Failures Occur

Failures in queue-based systems often appear as an indeterminate state: the business operation has already been completed, but the acknowledgment never arrived; the message has been handed to a consumer, but the consumer is unavailable; the stream is growing faster than it is being read.

Most practical designs operate under the at-least-once, i.e., “at least once,” model. The broker tries not to lose the message, but it may deliver it more than once. The implication is simple: consumers must be idempotent. Reprocessing the same message must not charge money twice, create a second order, or send the customer multiple identical emails.

Consumer crashed before acknowledgment

If the consumer crashes during processing, the broker usually treats the work as incomplete. In RabbitMQ, the message is returned to the queue if the connection is lost before the ack. In a managed queue, it becomes visible again after the visibility timeout expires. In Redis Streams, the entry remains among the pending entries until it is claimed through the recovery mechanism. In Kafka, the outcome depends on the offset commit: if the position was not committed, the event will be read again.

The main risk here is duplicate processing. The handler must therefore be able to safely retry the operation or determine that it has already been completed.

The work was completed, but the confirmation never arrived

The most problematic scenario is when the business operation has already been completed, but the broker never learns about it. For example, a service debited funds, created a database record, or sent a document, and then crashed before sending an ack, committing the offset, or deleting the message.

From the broker’s perspective, the task appears unfinished, so the message may be delivered again. The safeguards are idempotent processing, deduplication keys, and carefully defined transactional boundaries where they are available.

The queue is growing faster than it is being processed

If consumers cannot keep up, latency increases. In RabbitMQ, this is visible from the queue depth and the number of unacknowledged messages. In Kafka, it is reflected in how far the consumer group lags behind the end of the log. In Redis Streams, it appears as stream growth and an increase in pending entries. In a managed queue, it is visible from the age of the oldest message, queue depth, and the number of redeliveries.

It is important to monitor not only queue size, but also message age. A large queue that drains quickly may simply be a brief spike. An increase in the age of old messages means processing is already falling behind business needs.

Storage, Quotas, and DLQ

Each option has its own accumulation limit. In Kafka, the risk is tied to disk usage and retention policy. In Redis Streams, a stream with no length limit can grow indefinitely. In RabbitMQ, queue backlogs put pressure on the broker’s memory and disk. With a managed queue, provider quotas, retention limits, or rising operation costs typically come into play.

A DLQ is useful as a diagnostic mechanism, but not as a place to store permanent problems. If messages are being sent to the dead-letter queue in large volumes, you need to investigate the cause: an incompatible format, a failing external API, an incorrect handler version, or a retry limit being exceeded.

These scenarios are what connect broker selection with operations. If the team is not prepared to monitor latency, retries, unacknowledged messages, consumer group lag, disk utilization, and DLQ growth, the mere use of RabbitMQ, Kafka, Redis Streams, or a managed queue will not make the system resilient.

Broker Selection Table for Different Use Cases

This table is not intended to rank technologies from “best” to “worst”; it is meant to support practical selection. First, define the scenario: a task, an event, an internal stream, or managed delivery. Then check the constraints: ordering, replay, operations, latency, and quotas.

For common use cases, the choice can be summarized as follows:

If you look at the constraint rather than the scenario, the picture is as follows:

In summary, RabbitMQ is more often chosen for controlled task distribution, Kafka for an event log and rereading events, Redis Streams for lighter internal streams, and a managed queue when standard delivery and reduced operational overhead are important.

All of these options can support borderline scenarios, but that is exactly where the cost of choosing the wrong model rises quickly. For example, Kafka can process tasks, RabbitMQ can deliver events, and a managed queue can work with topics and subscriptions. But if the architecture goes against the tool’s core model, the team pays for it in complexity, workarounds, and more expensive operations.

When a managed queue is more advantageous than self-managed RabbitMQ or Kafka

A managed queue should be viewed not as a “simplified RabbitMQ” or a “small Kafka,” but as a different responsibility model. With a self-hosted approach, the team is responsible for the cluster, upgrades, redundancy, monitoring, security, disk capacity, failure recovery, and on-call operations. With a managed queue, a significant share of this work shifts to the cloud provider, while the team manages the service settings, message schema, handlers, and operating costs.

A managed queue is usually the better choice when the broker is not core to the product and is needed as an infrastructure layer for delivering tasks. Common indicators of this scenario include:

• The team is small and does not want to maintain dedicated RabbitMQ or Kafka expertise;
• The workload is irregular: spikes, background jobs, and periodic processing;
• Standard retries, visibility timeout, a DLQ, and a limited retention period are sufficient;
• An external SLA for the broker platform is important;
• The team does not want to manually maintain the cluster, upgrades, disks, and fault tolerance;
• Ready-made integrations are needed with cloud functions, object storage, schedulers, and audit events.

However, a managed queue is not always cheaper or simpler. If you need fine-grained routing, a nonstandard acknowledgment policy, long-term retention of large event streams, control over data placement, or minimal latency within your own network, self-managed RabbitMQ or Kafka may be justified.

This is especially important for Kafka. If the event stream becomes a central part of the architecture—analytics, audit, integrations, replay, and high throughput—its operational complexity may be justified. In that case, a managed queue does not replace an event platform; it only covers some delivery scenarios.

The practical rule is simple: if the queue is needed as a supporting layer for tasks, a managed queue often wins on total cost of ownership. If the broker becomes the core of an event-driven architecture or requires fine-tuned behavior, it is more reasonable to consider a self-managed or specialized managed Kafka/RabbitMQ deployment.

After that, you can move on to the conclusion: the choice has already been analyzed in terms of models, failures, scenarios, and operational responsibility.

Conclusion

Choosing a broker does not start with the name of a technology, but with the messaging model. If tasks need to be distributed to workers, RabbitMQ or a managed queue is a logical choice. If you need an event log, replayability, and high throughput, Kafka is the right fit. If you need a lightweight internal stream and Redis is already part of the infrastructure, Redis Streams is worth considering.

However, resilience is determined not by the broker itself, but by its operational model: processing acknowledgements, retries, DLQs, deduplication, consumer idempotency, and monitoring. The final choice should therefore answer not only the question of “how a message flows under normal conditions,” but also “what happens when a failure occurs.”

FAQ

Can Kafka be used as a regular task queue?

Yes, but it is often overkill. Kafka excels as an event log: messages are persisted, read by offset, and can be reread by different consumer groups. For simple background tasks such as sending emails or generating files, RabbitMQ or a managed queue is usually simpler.

When is a managed queue more cost-effective than running your own RabbitMQ or Kafka?

A managed queue is more cost-effective when the team does not have the resources to maintain a cluster, the load is irregular, standard retries and a dead-letter queue are required, and fine-tuning the broker is not a key requirement. This is especially true in serverless scenarios and for small teams.

Do you need idempotency if the broker promises reliable delivery?

Yes. In many real-world designs, delivery follows an at-least-once model: a message may be delivered again after a failure, timeout, or retry. Therefore, the handler must handle duplicates safely—for example, it must not charge twice or create a duplicate order.

When Are Redis Streams Sufficient Instead of Kafka?

Redis Streams can be sufficient for moderate internal streams if Redis is already part of the infrastructure and long-term storage of large volumes of events is not required. However, it should not be treated as a direct replacement for Kafka for high-throughput event streaming, long-term retention, or large-scale replay.

Do queues guarantee strict message ordering?

Not always. Kafka preserves order within a single partition, RabbitMQ preserves order within a queue when parallel processing is constrained, Redis Streams uses ordered IDs, and a managed queue may offer different modes, including dedicated FIFO variants. Retries, multiple consumers, and redelivery can change the actual processing order.

What is important to monitor after selecting a broker?

At a minimum, monitor queue depth, processing latency, the number of unacknowledged messages, consumer errors, DLQ growth, commit/ack/delete latency, storage usage, and the message ingress rate. For Kafka, consumer group lag and disk usage are also important; for Redis Streams, monitor stream growth and pending entries.

Sources

1. RabbitMQ — Consumer Acknowledgements and Publisher Confirms

2. Apache Kafka — Design

3. Redis — Streams documentation

4. Amazon SQS Developer Guide — Visibility Timeout