Kafka Consumer Group Coordinator

The consumer group coordinator is the broker-side component that manages membership and coordination for a Kafka consumer group. It handles group membership, heartbeats, rebalancing, and the distribution of partitions among group members.

A Kafka consumer group is a set of consumers sharing work from one or more topics. Within a group, each partition is assigned to at most one consumer at a time. The group coordinator is the broker responsible for coordinating that group and maintaining its membership state.

Consumers discover their coordinator and send group-management requests to it. Members join the group, receive a generation and assignment, and periodically send heartbeats. If a member leaves or stops heartbeating, the coordinator detects the membership change and triggers a rebalance.

During a rebalance, the group determines which consumers are members and how partitions should be assigned. Depending on the configured assignor and rebalance protocol, assignment work can involve the group leader or newer cooperative mechanisms. The coordinator then communicates the resulting assignments to consumers.

The coordinator is also involved in tracking committed offsets for the group through Kafka's internal offset storage. It is important to distinguish this coordination role from the consumer's actual record processing: the coordinator does not normally fetch application records on behalf of consumers; consumers fetch records directly from the brokers that host the relevant partitions.

Key points

• One broker acts as the coordinator for a given consumer group.
• Consumers join the group and maintain membership with heartbeats.
• Membership changes can trigger a partition rebalance.
• The coordinator communicates group state and assignments to consumers.
• Consumer record fetching is performed by consumers, not by the coordinator.

Interview questions

What is a Kafka consumer group coordinator?

It is the broker responsible for coordinating a particular consumer group. It tracks group membership and handles requests such as joining and leaving the group. Consumers also send heartbeats so the coordinator can determine whether members are still active. Membership changes can lead to a rebalance and new partition assignments.

What causes a consumer group rebalance?

A rebalance occurs when the membership or assignment requirements of a group change. Common triggers include a consumer joining or leaving, a consumer being considered dead after missing heartbeats, or changes that require the group to redistribute partitions. The coordinator manages the group transition and communicates the resulting assignment. The exact behavior also depends on the rebalance protocol and assignment strategy.

How do Kafka consumer heartbeats relate to the coordinator?

Heartbeats let a consumer tell the coordinator that it is still an active member of the group. If the coordinator does not receive heartbeats within the configured session timeout, it can remove the member from the group. That membership change can trigger a rebalance. Heartbeats are therefore part of failure detection rather than a mechanism for fetching application records.

Is the consumer group coordinator the same thing as the consumer group leader?

No. The coordinator is a broker responsible for managing the consumer group's state and membership. The group leader is a consumer role used by assignment protocols to help determine partition assignments. A coordinator can therefore coordinate a group even though the consumer responsible for assignment calculations is a separate process. The distinction is important when diagnosing group-management behavior.

What happens when the consumer group coordinator fails?

The group must obtain a new coordinator when the current coordinator is unavailable. Consumers can rediscover the coordinator and retry group-management operations. During the transition, group-management requests may fail or be retried, and the group may need to stabilize before normal coordination resumes. Kafka's broker-side coordination and metadata mechanisms allow the coordinator role to move rather than permanently tying a group to one broker.