Duplicate Event Detection

Duplicate event detection lets a consumer recognize that an event has already been processed and avoid repeating its side effects. It is a key technique for making event-driven consumers safer when messages are replayed or delivered more than once.

A consumer usually cannot assume that each delivered event is unique. Brokers, retries, consumer restarts, redelivery, and replay can cause the same logical event to reach a consumer multiple times. Duplicate detection adds state that records which event identities have already been successfully handled.

The consumer needs a stable identifier such as an event ID. Before applying a non-idempotent side effect, it checks whether that ID has already been processed. If it has, the consumer skips the work; otherwise it records the processing state and performs the required action.

The difficult part is coordinating the duplicate check, side effect, and acknowledgement. A process can crash after the side effect but before recording the event as processed, or after recording state but before completing the side effect. A robust design therefore considers atomicity, transactions, idempotent operations, and the failure guarantees of the surrounding infrastructure.

A common implementation stores event IDs in a database with a unique constraint. The insert can act as the ownership check: the first consumer succeeds, while a repeated event encounters a uniqueness conflict and can be treated as already processed. For stronger guarantees, the deduplication record and business-state update can often be committed in the same database transaction.

Deduplication does not make delivery exactly once by itself. It is a consumer-side strategy for limiting repeated effects, and its correctness depends on event identity, retention, storage durability, transaction boundaries, and what happens when processing fails.

Key points

• Use a stable event ID to recognize repeated deliveries.
• Persist processed-event state rather than relying only on in-memory collections.
• A unique database constraint can make duplicate detection atomic among concurrent consumers.
• Coordinate deduplication with business updates and message acknowledgement.
• Deduplication reduces repeated side effects but does not automatically provide end-to-end exactly-once processing.

Interview questions

Why do event consumers need duplicate detection?

A message can be delivered more than once because of retries, redelivery, replay, or consumer failures. If processing has side effects such as creating a payment or sending a notification, repeating the operation can cause an incorrect result. The consumer can use a stable event ID to recognize a previously handled event. It then skips the duplicate instead of applying the side effect again.

How would you implement duplicate detection with a relational database?

I would create a table containing the event ID and a unique constraint on that ID. When processing starts, the consumer attempts to insert the event ID. If the insert succeeds, it owns the first processing attempt; if the unique constraint rejects it, the event is already known and can be skipped. When possible, I would put the deduplication record and the related business-state update in the same database transaction.

What failure can occur if you record the event before performing the side effect?

The consumer could record the event as processed and then crash before completing the side effect. A later delivery would look like a duplicate and be skipped, leaving the business operation incomplete. This is why the deduplication record and the side effect should be coordinated when possible. If they cannot share a transaction, the design may require idempotent operations, an outbox or inbox pattern, retries, or another recovery mechanism.

Does duplicate detection provide exactly-once semantics?

Not by itself. It can prevent repeated processing of the same event when the identity and durable state are reliable, but the complete system still has failure windows between consuming a message, updating state, performing external effects, and acknowledging the message. Exactly-once behavior requires the relevant operations to be coordinated so retries cannot create additional observable effects. In practice, systems often combine durable deduplication with idempotent operations and transactional patterns.

How do you handle duplicate events arriving concurrently?

The duplicate check must itself be concurrency-safe. A database unique constraint is a common solution because concurrent consumers cannot both successfully create the same unique event record. The losing transaction can treat the conflict as evidence that another consumer has already claimed or processed that event, depending on the transaction design. I would also define what happens if the first transaction fails or rolls back so that the event remains recoverable.