Consumer Groups

Consumer groups let multiple consumer instances cooperate on processing a stream by assigning each partition to at most one consumer within the group at a time. This provides horizontal scalability while allowing different groups to independently consume the same events.

A consumer group is a logical set of consumers that share a group identifier. In Kafka-style systems, partitions are the unit of parallelism: within one consumer group, a partition is assigned to one consumer at a time, so multiple consumers can process different partitions concurrently.

If a topic has more partitions than consumers, some consumers can receive multiple partitions. If there are more consumers than partitions, some consumers remain idle because a partition cannot be actively assigned to multiple consumers in the same group. Therefore, partition count places an upper bound on active consumer parallelism within a group.

Consumer groups also provide a useful isolation mechanism. Two different groups can independently consume the same topic and maintain separate progress, commonly represented by separate offsets. For example, an order-processing group and an analytics group can both receive the same events without competing with each other.

When group membership or partition assignments change, the system performs a rebalance so ownership is redistributed. A Java backend engineer should consider the effects of rebalances, processing time, offset management, duplicate processing, and idempotency when designing consumers.

The model is therefore both a scalability mechanism and a work-sharing mechanism: partitions are divided among members of one group, while separate groups represent independent consumers of the event stream.

Key points

• One partition is assigned to at most one active consumer in a group.
• Partition count limits the maximum parallel consumer activity for a group.
• Consumers in different groups can independently process the same topic.
• Membership or partition changes can trigger a rebalance.
• Offset handling and idempotent processing matter because failures can cause records to be processed again.

Interview questions

How do consumer groups provide scalability?

A consumer group allows multiple consumer instances to process different partitions concurrently. Each partition is assigned to only one consumer within that group at a time. Increasing the number of consumers can increase parallelism until the available partition count becomes the limiting factor. This makes partitioning an important part of capacity planning.

What happens when two consumers belong to the same consumer group?

They cooperate rather than independently receiving every record. The group coordinator assigns partitions among the consumers, with a partition having at most one active owner in that group. If membership changes, assignments can be redistributed through a rebalance. This prevents two members of the same group from normally processing the same partition concurrently.

Why would you use two different consumer groups for the same topic?

Different groups represent independent applications or processing pipelines. Each group can consume the same topic without competing for the other group's partition assignments. Their progress is tracked independently, so one application's consumption does not prevent another from reading the events. For example, billing and analytics can process the same order events independently.

What limits consumer-group throughput, and how would you scale it?

The partition count is a fundamental limit on concurrent partition processing within a group. If consumers are fewer than partitions, adding consumers may increase parallelism; once there are at least as many consumers as partitions, additional consumers generally cannot increase partition-level concurrency. Actual throughput can also be constrained by downstream services, serialization, network capacity, processing latency, and consumer configuration. Scaling therefore requires measuring the bottleneck rather than simply increasing consumer count.

How do you design a consumer to handle duplicate processing safely?

A consumer should assume that a record can be processed again when failures occur around processing and offset commitment. A common approach is to make the business operation idempotent, such as using a unique event or operation identifier with a database constraint or deduplication record. Offset commits should be coordinated carefully with successful processing according to the required delivery semantics. If stronger guarantees are required, the design may use transactional mechanisms, but the exact solution depends on the broker and application architecture.