Consumer Groups

A consumer group is a set of consumers that cooperatively processes partitions of a stream. Within a group, each partition is assigned to at most one active consumer at a time, allowing work to be distributed across instances.

A consumer group lets multiple consumer instances share the work of reading a partitioned stream. Each consumer belongs to a named group, and the broker tracks which consumers are active members of that group.

For a topic with multiple partitions, partitions can be distributed among consumers in the same group. A consumer normally processes the records from the partitions assigned to it. If there are more consumers than partitions, some consumers can have no partition assigned.

Membership can change when consumers start, stop, fail, or otherwise leave the group. The system then performs a rebalance so partitions are reassigned among the remaining members. This allows a group to continue processing after an instance failure.

Consumers in different groups process the stream independently. For example, one group can process orders for billing while another group processes the same records for analytics. The group identity therefore determines which consumers share work and which consumers receive their own independent stream processing.

Key points

• Consumers in the same group share partition processing.
• A partition is assigned to at most one active consumer within a group at a time.
• Adding or removing consumers can trigger a rebalance.
• More consumers than partitions can leave some consumers idle.
• Different groups can independently consume the same stream.

Interview questions

What is a consumer group?

A consumer group is a collection of consumers that cooperate to process a partitioned stream. Consumers in the same group divide the available partitions between them. This allows multiple application instances to process the stream concurrently. Consumers belonging to different groups process the stream independently.

What happens if there are more consumers than partitions?

Some consumers will not receive a partition assignment. They remain members of the group but have no partition to process until the assignment changes. Therefore, simply adding consumers does not guarantee higher parallelism. The number of partitions places an upper bound on partition-level parallelism within the group.

What is a rebalance?

A rebalance is the process of redistributing partition assignments among the current members of a consumer group. It can occur when membership changes or when assignment-related state changes. During a rebalance, ownership of partitions may move between consumers. The goal is to establish a valid assignment for the new group membership.

Why use multiple consumer groups for the same topic?

Separate groups allow different applications or processing pipelines to consume the same stream independently. For example, billing and analytics can each receive all relevant records without competing for the same partition assignments. Each group maintains its own consumption state. This separates the progress and scaling behavior of the applications.

How does consumer-group membership affect scalability?

A group can scale processing across multiple consumers because partitions can be assigned to different instances. However, adding consumers beyond the available partitions does not create additional partition-level parallelism. The practical throughput also depends on processing speed, partition distribution, and downstream resources. Scaling therefore requires enough partitions as well as sufficient consumer capacity.