Consumer Offset Management

Consumer offset management records how far a consumer has processed a partition in an ordered event stream. In systems such as Apache Kafka, offsets let consumers resume after restarts and control delivery semantics such as at-least-once processing.

A partitioned event stream assigns each record a monotonically increasing offset within its partition. A consumer's position identifies the next record it intends to read, while a committed offset is durable progress recorded for recovery. Offsets are scoped to a partition, so a consumer group maintains progress separately for every partition it owns.

In Kafka, consumers typically poll records, process them, and commit offsets. With automatic commits, the client periodically commits positions; with manual commits, application code decides when progress is durable. Manual management is useful when processing must finish successfully before the corresponding offset is committed.

The key failure scenario is committing too early. If a consumer commits an offset before its side effect succeeds and then crashes, the record may not be processed again. Conversely, processing successfully but crashing before the commit can cause the record to be delivered again after restart. This is why Kafka consumers commonly provide at-least-once behavior when processing precedes committing.

An offset is a position, not a globally unique event identifier. The same numeric offset can exist in different partitions, and offsets are not necessarily consecutive across partitions. Consumers can also seek to a specific offset or reset according to configured policies when no valid committed position exists.

Key points

• Offsets are tracked independently for each partition.
• A committed offset is recovery state; it is not the same thing as the consumer's current in-memory position.
• Commit after successful processing to avoid losing work through premature commits.
• Processing before committing can produce duplicate processing after a crash.
• Consumer groups distribute partitions and maintain committed progress for each partition.

Interview questions

What is a consumer offset?

A consumer offset represents a position in a partitioned log. In Kafka, each partition has its own offset sequence, so progress must be tracked per partition. A committed offset allows a consumer to recover its position after a restart. The offset identifies a position rather than a globally unique event.

Why would you manually commit offsets instead of using automatic commits?

Manual commits give the application control over when processing progress becomes durable. A common pattern is to process a batch successfully and then commit its offsets. This reduces the chance of acknowledging work that has not actually completed. The trade-off is that the application must handle commit timing and failures correctly.

What happens if a consumer crashes after processing a record but before committing its offset?

After recovery, the consumer can resume from the previously committed offset. That means the already-processed record may be read again. This is a normal consequence of at-least-once processing. The application therefore needs idempotent processing or another mechanism for handling duplicates safely.

What is the difference between a consumer's position and its committed offset?

The consumer position is the next record the consumer will request from a partition. The committed offset is durable group state that can be used to restore progress. They can temporarily differ while records are being processed or while commits are pending. Understanding that distinction is important when diagnosing lag and restart behavior.

How would you design offset handling for a Kafka consumer that writes to a database?

I would first define the required delivery semantics and identify where the database transaction and offset commit boundaries occur. For at-least-once processing, I would complete the database operation successfully before committing the Kafka offset. Because a crash can still occur between those two actions, I would make the database operation idempotent or use a deduplication key. If stronger atomicity is required, I would evaluate an architecture that coordinates the relevant transaction boundaries rather than assuming an offset commit alone provides exactly-once effects.