What is the difference among SQS, SNS and Kafka?

SQS is a point-to-point queue: each message is processed by one consumer and then deleted. SNS is pub/sub: one published message is pushed to many subscribers, generally without storing it. Kafka is a durable, partitioned log: events are retained, and many consumer groups read them independently and can replay them.

Choose SQS for background jobs and task distribution where each message needs one worker and no replay is needed. It is fully managed, scales without capacity planning, has built-in retries and DLQs, and is billed per request. Kafka adds partitions, brokers and offset management that a simple job queue does not need.

You publish once to an SNS topic and subscribe several SQS queues to it. Each queue gets its own copy and buffers messages durably, so each consumer service processes at its own pace, retries independently and survives downtime. SNS gives fan-out, and SQS adds the storage and retry layer that SNS lacks.

Kafka keeps events on disk for a configured retention period regardless of consumption. Consumers only track an offset, so you can reset it and reread history. SQS deletes a message once the consumer acknowledges it, so processed messages are gone. Only unprocessed messages remain, for up to 14 days.

SQS standard queues and SNS standard topics give best-effort ordering only. SQS FIFO queues and SNS FIFO topics guarantee order within a message group ID. Kafka guarantees order within a partition, and events with the same key go to the same partition. None of them gives global ordering at scale without limiting parallelism.

SQS standard and SNS standard are at-least-once, so duplicates can occur. SQS FIFO deduplicates within a 5-minute window for exactly-once processing on the queue side. Kafka is at-least-once by default. Idempotent producers and transactions give exactly-once for read-process-write flows within Kafka. External side effects still need idempotent consumers.

In SQS you add more consumers, and they all pull from the same queue. There is no partition limit, so scaling is nearly linear. In a Kafka consumer group, each partition is read by only one consumer. Parallelism is therefore capped by the partition count, and you must plan partitions in advance.

On AWS with modest volume, use SNS fanned out to three SQS queues. Each service gets an isolated, retryable copy. If analytics needs replay, high throughput or stream processing, publish to Kafka instead and give each service its own consumer group. That also lets you add new consumers later and backfill them from history.

SNS retries delivery according to a retry policy that depends on the protocol. For HTTP endpoints this policy is configurable. If all retries fail, the message is dropped, unless a dead-letter queue is set on that subscription. SNS standard topics do not store messages for later reading. This is why SQS queues are usually placed behind SNS for durability.

SNS standard topics do not retain messages. SNS FIFO topics support message archiving with a retention period, and subscribers can request a replay from that archive. That is a narrow feature, not a general replayable log like Kafka with consumer offsets.