Consumer Idempotency

Consumer idempotency makes repeated handling of the same logical event produce the same business result as handling it once. It is a key defense against retries and duplicate delivery, but it does not mean the consumer executes only once or provide global exactly-once behavior.

Event brokers commonly deliver a message again when a consumer crashes, a network response is lost, or acknowledgement does not complete. Consumer idempotency means repeated handling of the same logical event leaves the business state as though it had been handled once. It is a practical defense against at-least-once delivery, not a promise that the handler executes only once.

A common approach is to assign each event a stable identifier and record processed identifiers in durable storage. In a database-backed consumer, insert the identifier into an inbox or deduplication table and apply the business update in the same transaction. A uniqueness constraint on the event identifier lets concurrent deliveries race safely: only one transaction can claim it.

The transaction boundary matters. If the business update commits but the deduplication record does not, a retry may apply the update again. If the record commits first but the update fails, a retry may be incorrectly discarded. Keeping both changes atomic avoids this split for effects within the same database.

For effects outside that transaction, such as sending an email or charging a payment provider, use that system's idempotency key where available, or record intent in a transactional outbox and make downstream handling deduplicated too. Choose retention for deduplication records carefully: removing them too soon can make an old replay effective again. Also distinguish duplicate delivery of one event from two different events that happen to describe similar actions.

Key points

• Assume duplicate delivery can happen; make the business effect safe to repeat.
• Use stable event identifiers and durable deduplication state.
• Commit the deduplication claim and database effect atomically when possible.
• For remote effects, propagate idempotency keys or use an outbox and downstream deduplication.
• Set deduplication retention to cover retry and replay windows.

Interview questions

What does consumer idempotency mean, and why is it useful?

It means processing the same logical event repeatedly has the same externally visible result as processing it once. This is useful because brokers and consumers can retry after failures, causing duplicate deliveries. A stable event identifier and durable deduplication state are common implementation tools. It does not mean the handler runs only once.

How would you implement idempotency for a consumer that updates a database?

I would store a processed-event identifier in an inbox table with a unique constraint, and apply the business update in the same database transaction. If inserting the identifier conflicts with an existing row, the delivery is a duplicate and its business effect is skipped. The atomic transaction prevents a crash from leaving the update committed without its deduplication record, or vice versa. The consumer should acknowledge only after the transaction succeeds.

How do you preserve idempotency when processing triggers an external side effect?

First identify which effects are inside one transactional resource and which are outside it. An inbox record and database update can usually share a transaction, but a database commit cannot atomically include an arbitrary external API call. For an external effect, pass a stable idempotency key to the provider if supported; otherwise use an outbox or workflow with retries and reconciliation, while ensuring downstream operations are also safe to repeat. State the remaining failure modes rather than claiming global exactly-once behavior.

How long should a consumer keep deduplication records?

The deduplication retention period must cover the maximum period in which a duplicate or replay could arrive. That depends on broker retention, retry policies, disaster recovery, and whether operators can replay historical events. If records are deleted earlier, an old event may be treated as new and apply its effect again. Retention has storage costs, so it should be an explicit operational policy rather than an arbitrary short TTL.