Kafka Consumer Group

A Kafka consumer group is a set of consumers that cooperate to read topic partitions, with each partition assigned to at most one consumer in the group at a time. Consumer groups provide horizontal scaling and fault recovery while allowing different groups to independently consume the same topic.

A consumer group is identified by a `group.id`. When multiple consumers use the same group ID, Kafka treats them as members of one logical group and distributes the topic's partitions among them. Each partition is consumed by only one member of that group at a given time, preserving the partition's ordered sequence for that consumer.

Consumer groups are useful for scaling message processing. If a topic has six partitions, up to six consumers in one group can actively consume those partitions in parallel. Adding more consumers than available partitions does not increase parallelism for that topic because some consumers will have no partition assigned.

Kafka tracks a group's consumption position using offsets. Consumers periodically commit offsets so that, after a restart or failure, the group can resume from a known position. The exact delivery behavior depends on when offsets are committed relative to processing, so applications must choose an appropriate processing and commit strategy.

When group membership or partition availability changes, Kafka performs a rebalance. Partitions are reassigned among the current members. A rebalance can temporarily interrupt consumption, so production applications should avoid unnecessary membership churn and use appropriate consumer configuration.

Different consumer groups maintain independent offsets. Therefore, two groups can read the same topic independently: for example, one group can process orders while another performs analytics. This is different from two consumers belonging to the same group, where they share the work rather than each receiving every partition's records.

Key points

• Consumers sharing a `group.id` form one consumer group.
• Within a group, a partition is assigned to at most one consumer at a time.
• A topic's partition count limits parallel consumption within a single group.
• Committed offsets let a group resume consumption after restarts or failures.
• Membership or partition changes can trigger a rebalance.
• Separate groups consume the same topic independently.

Interview questions

What is a Kafka consumer group?

A consumer group is a set of Kafka consumers identified by the same `group.id`. Kafka assigns topic partitions among members of that group so that one partition is handled by at most one member at a time. This allows multiple consumers to process different partitions in parallel. The group also maintains offsets that represent its progress through the partitions.

What happens if a topic has three partitions but you start five consumers in one group?

At most three consumers can actively consume those three partitions at the same time. The remaining two consumers will have no partition assigned and therefore will not receive records from that topic. If more partitions become available later, assignments can change during a rebalance. This is why partition count is an important limit on consumer parallelism.

Why would you use multiple consumer groups for the same Kafka topic?

Different groups can consume the same topic independently because each group tracks its own offsets. For example, one group could process orders and another could build analytics data. Consumers within each group share the work, while the groups themselves do not share consumption progress. This lets multiple applications react independently to the same event stream.

What is a Kafka rebalance?

A rebalance is the process of redistributing partitions among the active consumers in a group. It can occur when consumers join or leave, or when the set of subscribed partitions changes. The purpose is to maintain a valid assignment as group membership changes. During a rebalance, consumption can be interrupted depending on the consumer configuration and assignment protocol.

How do offsets relate to a consumer group?

An offset identifies a position within a Kafka partition, and a consumer group records its progress by committing offsets. When a consumer restarts or another member takes over a partition, the group can use the committed position as a starting point. The timing of offset commits matters because committing before processing finishes can lead to records being skipped after a failure, while committing afterward can cause records to be processed more than once. Therefore, offset management is closely connected to delivery semantics and failure handling.