Inbox Deduplication

Inbox deduplication uses a durable message identifier to recognize deliveries that have already been processed. A consumer can then reject or safely ignore duplicates, making message handling effectively idempotent even when delivery is repeated.

Message brokers can redeliver the same logical message, especially when a consumer processes the message but fails before acknowledging it. Inbox deduplication handles this by associating each incoming message with a stable unique identifier.

A consumer records that identifier in durable storage as part of processing. When another delivery arrives with the same identifier, the consumer can detect that it has already been handled and skip the business operation.

The important correctness property is atomicity between recording the message and applying its business effect. A common design puts the inbox record and the business update in the same database transaction, so a crash cannot leave one committed while the other is absent.

The identifier must represent the logical message, not merely a delivery attempt. If the broker assigns a new identifier to every retry, deduplication will not recognize those retries as duplicates.

Inbox deduplication reduces duplicate effects but does not by itself guarantee exactly-once behavior across arbitrary systems. When processing spans multiple resources, transactional boundaries, idempotent operations, or other coordination mechanisms may still be required.

Key points

• Use a stable message ID to identify the logical delivery.
• Persist processed IDs in durable storage.
• Prefer an inbox record and business mutation in the same database transaction.
• Treat repeated IDs as already processed and avoid repeating the side effect.
• Exactly-once effects across independent systems require more than an inbox table.

Interview questions

What makes a message identifier suitable for deduplication?

It must be stable across retries of the same logical message and sufficiently unique to avoid treating different messages as the same one. The identifier should be generated or assigned at the logical-message level rather than regenerated for each delivery attempt. The consumer persists that identifier so the duplicate check survives process restarts. If the identifier changes on retry, the inbox cannot reliably detect the duplicate.

Why is inbox deduplication needed when using a message broker?

Message delivery is often at-least-once, so a consumer may receive the same logical message more than once. This can happen when processing succeeds but the acknowledgment is lost or the consumer crashes before acknowledging. A durable message ID lets the consumer recognize the later delivery as a duplicate. The duplicate can then be ignored instead of applying the business operation again.

How would you make inbox recording and business processing safe against a crash?

For database-backed processing, I would normally insert the message ID into an inbox table and apply the business change in the same database transaction. If the transaction commits, both the inbox record and business effect become durable together. If it rolls back, neither is committed, so a redelivery can retry the operation. A uniqueness constraint on the message ID also provides a strong database-level duplicate check.

Can inbox deduplication guarantee exactly-once processing?

Not in the general distributed-systems sense. It can provide exactly-once effects within a transaction boundary when the inbox record and business mutation are atomically committed together. If processing also calls an external service, that external side effect may still be repeated or may succeed even when the local transaction fails. Such workflows need additional mechanisms such as idempotency keys, transactional messaging, or compensating actions.

How do you handle two consumers processing the same message concurrently?

The database should enforce uniqueness on the message ID rather than relying only on an application-level check. Concurrent transactions can then race to create the inbox record, with only one succeeding. The winning transaction performs the business mutation, while the other treats the uniqueness conflict as evidence that the message has already been claimed or processed, according to the transaction design. Isolation and retry behavior should be chosen carefully for the specific database.