At-Least-Once Processing with Consumer Groups

At-least-once processing means a message is delivered for processing one or more times, so a failure can cause redelivery. With consumer groups, this commonly happens when processing succeeds but the consumer fails before its offset is committed.

In a consumer group, each partition is assigned to one consumer at a time. The consumer reads records and tracks its position with an offset. Committing an offset tells the messaging system that the consumer has completed processing through that position.

The key failure window is between processing a record and committing its offset. If the application processes the record successfully but crashes before the commit, the broker still considers that offset uncommitted. After the consumer restarts or the partition is reassigned, the record can be read again.

This behavior favors reliability over avoiding duplicates. A message is not silently lost merely because the consumer failed after processing it but before acknowledging progress. The application therefore needs to tolerate repeated processing when duplicates are possible.

A common approach is to make the processing operation idempotent. For example, a database write can use a unique event ID or an upsert so that processing the same event again does not create an incorrect second effect. Offset commits should represent successfully completed processing, rather than being treated as proof that the business operation can never be repeated.

Consumer-group rebalancing can also expose the same principle: when partition ownership moves to another consumer, the new owner resumes from the committed position. Records processed after that position but before a commit may therefore be processed again.

Key points

• Processing can happen more than once when a failure occurs before offset commitment.
• The important failure window is after business processing but before the offset commit.
• Redelivery provides stronger protection against losing successfully fetched records.
• Consumers should make important side effects idempotent or otherwise duplicate-safe.
• A committed offset represents consumer progress, not a universal guarantee that a business effect occurred exactly once.

Interview questions

What does at-least-once processing mean?

It means a message can be processed one or more times. A consumer may successfully handle a record and then fail before recording its new offset. When processing resumes, that record can be delivered again. Therefore, consumers should be prepared for duplicates.

Why can a consumer process the same message twice?

The consumer's business operation and offset commit are separate steps in the basic at-least-once model. If the business operation succeeds but the process crashes before the offset is committed, the broker still has the older committed position. The replacement or restarted consumer can therefore read the same record again.

How would you make a Java consumer safe against duplicate messages?

I would give each event a stable identifier and make the business operation idempotent. For database effects, this can be implemented with a unique constraint, deduplication record, or an upsert, depending on the use case. The consumer should only acknowledge or commit progress after the required business operation succeeds. This way, a redelivery does not produce an additional unintended business effect.

What is the main difference between at-least-once and at-most-once processing?

At-least-once prioritizes avoiding message loss and can process a record more than once. At-most-once advances the consumer position before processing, so a failure during processing can cause the record to be skipped. The trade-off is therefore primarily between possible duplicates and possible loss. The appropriate choice depends on the required delivery semantics and the application's tolerance for each failure mode.

What happens during a consumer-group rebalance under at-least-once processing?

A partition can move from one consumer to another, and the new owner resumes from the committed offset. If the previous consumer processed records after that offset but did not commit them before losing ownership, the new consumer may process those records again. This is another reason that external side effects must tolerate duplicates. Rebalancing therefore does not by itself eliminate at-least-once redelivery.