Consumer Offset Independence

Consumer offset independence means a broker can delete messages according to its retention policy without waiting for consumers to process them. A consumer that falls behind can therefore encounter a gap when its committed offset is older than the earliest retained offset.

Message retention and consumer progress are separate mechanisms. A broker typically removes records based on configured limits such as age or storage usage, while a consumer records its progress by committing offsets.

If a consumer is slow, stopped, or disconnected long enough for retention to remove messages it has not processed, its committed offset can point to data that no longer exists. When it resumes, the broker may report that the requested offset is outside the currently retained range.

In Apache Kafka, this situation can result in an `OffsetOutOfRangeException` for a consumer using an affected offset. The application must then follow its configured recovery behavior, such as resetting to the earliest or latest available offset, depending on the consumer configuration and processing requirements.

This distinction is important for Java backend systems because increasing retention does not itself guarantee processing progress, and committing offsets does not prevent retention from deleting records. Durable recovery therefore requires considering retention duration, consumer downtime, offset management, and replay requirements together.

Key points

• Retention can delete records even when consumers have not processed them.
• Consumer offsets represent progress; they do not normally pin records in the broker.
• A lagging consumer can request an offset that has already expired from retention.
• Kafka consumers need an explicit recovery policy for offsets that are no longer available.

Interview questions

What does consumer offset independence mean in Kafka?

It means the broker's decision to retain or delete records is separate from a consumer's committed progress. A consumer can commit an offset, stop processing, and later find that records at or before that offset are no longer retained. Retention policies can therefore cause a lagging consumer to lose access to unprocessed history.

What happens if a Kafka consumer's committed offset is older than the earliest retained offset?

The consumer cannot read records that have already been deleted. Kafka can report an out-of-range offset condition, and the consumer needs a configured or application-level policy for choosing a valid position. For example, resetting to `earliest` starts at the earliest record still available, not at records that retention has removed.

Does committing a consumer offset prevent Kafka from deleting the corresponding messages?

No. Consumer offset commits track processing progress and are not generally a retention lock. Kafka retention can remove records even when a consumer group has not reached them. If replay of old data is a requirement, the retention policy must preserve that data for the required period or the data must be stored elsewhere.

How would you design a Java service to tolerate consumers falling behind retention?

First, I would establish the maximum expected outage or processing delay and make retention long enough to cover the required recovery window. I would monitor consumer lag and the distance between committed offsets and the earliest retained offset. The service should also define what happens when data is unavailable, such as resetting, rebuilding state from another source, or failing visibly rather than silently skipping required events. For critical replay requirements, I would consider an independent durable archive or another source of truth.