Kafka Consumer Group

24 Oct 2026

This write-up is to summarise my understanding of the kafka consumer group’s protocol and its evolution. We will look at the following:

• Recap how a consumer group connect to the kafka cluster
• How partitions are assigned to consumers
• The classic rebalance protocol and its problem
• Improvements to elevate above issues
• The next generation rebalance protocol

Recap: The Consumer Group

The above diagram shows how the protocol dance between the classic consumer and the brokers/coordinator.

1. After initiating TCP connection with the broker (any one of them), the consumer sends a FindCoordinator request and receives the ip+port of the Coordinator it should contact.
2. The consumer then sends a JoinGroup request to the Coordinator. One of the consumers will be selected to be the leader for the group (usually the first one). The Coordinator returns memberId for all consumers, and for the leader it also returns the member lists and subcription (i.e. the topics that the consumers are interested in)
3. Then the consumer group’s leader will do topic-partition to consumer instance assignment and send a SyncGroup request with the computed assignment.
4. The other consumers will also send a SyncGroup request with only the memberId, and they will receive their assignments of topic-partitions to consume.
5. The consumption (i.e. Fetch request) can then be started.

In the background, we may also have 2 threads running.

1. The OffsetCommit thread will run in the background if enable.auto.commit is true. The consumer will send OffsetCommit request periodically to the Coordinator. The default interval is 1s and we can edit it through auto.commit.interval.ms.
2. The HeartBeat thread will also run in the background where the consumer sends periodical HeartBeat requests to indicate to the Coordinator its still alive and well. The interval follows heartbeat.interval.ms and the default should be 3s.

How to assign partitions to consumers?

1. Range Partition
   • Goes through each topic’s partition and assign them to consumers.
   • Example suppose we have 4 consumers (C1, C2, C3, C4):
     • topicA-0 -> C1
     • topicA-1 -> C2
     • topicB-0 -> C1
     • topicB-1 -> C2

     ⚠️ Note that in this case C3 and C4 will have no partitions asssigned

2. Round Robin
   • All topic partitions spread evenly across consumers.
   • Example suppose we have 4 consumers (C1, C2, C3, C4):
     • topicA-0 -> C1
     • topicA-1 -> C2
     • topicB-0 -> C3
     • topicB-1 -> C4
3. Sticky Partition
   • Round Robin + Makes best effort to stick previous assignment during rebalance (don’t revoke partition first, because it may be assigned to same consumer)
   • This tries to solve problem #1 below
4. Cooperative Sticky
   • 1st step: revoke partition
   • 2nd step: partitions that are not revoked can continue consuming
   • This tries to solve problem #1 and #2 below
   • But this needs to use Incremental Cooperative Rebalance Protocol

The Classic Rebalance Protocol

First, what triggers a rebalance? Three things

1. Consumer did not send heartbeat within the interval (maybe its dead)
2. Consumer voluntarily send LeaveGroup(maybe its shutting down)
3. Topic partition increases
4. New topic created that matches the Consumer subscription regex Here, 1 & 2 are the more common scenarios.

• The Coordinator tells the consumers that a rebalance is in progress (i.e. REBALANCE_IN_PROGRESS) using the HeartBeat response or the Fetch Response.
• The consumers then revoke their partitions, cleanup local states relating to it.
• Then proceed to send JoinGroup and SyncGroup request as if its a new consumer.

Problems with the Classic Protocol

1. Rebuilding State Unneccessarily

After the rebalance, some partitions may actually still be assigned to the original consumer holding it. So revoking it doesnt make sense, it has to rebuild local app state all over again.

2. Stop The World

During the rebalance, because we want to ensure each partition is assigned to one and only one consumer, all of the consumer will stop consumption. This pauses the whole fetching and reduces throughput.

Improvement: Static Membership (KIP-345)

Goal: Don’t rebalance if consumer instance can join back fast enough

1. Set a group.instance.id for each consumer instance (e.g. through environment variable)
2. When the group leaves, it won’t send a LeaveGroup.
3. So when it joins back, the Coordinator sees that it is a instance it recognise, the consumption resumes

⚠️ Consumer that joins back before session.timeout.ms

Improvement: Incremental Cooperative (KIP-429)

Goal: Don’t rely on a single global synchronization barrier to ensure each partition assigned to one consumer, use consecutive rebalance to get to a final stable state. Revoke only those partitions needing assignment.

1. Do not revoke partitions initially, send JoinGroup request with all your currently owned partitions.
2. The consumer leader will do the first round of assignment such that if we only revoke some partitions that need re-assginment (i.e. suppose a consumer has p0,p1,p2 and we want to revoke p2, we just assign it p0,p1)
3. For consumers who has previously owned partition that is not in the newly assigned ones, these will trigger next round of rebalance (i.e. consumer will realise p2 is not assigned to it and revoke it)
4. The process continues until no more partitions need to be revoked

⚠️ You need to use the CooperativeStickyAssignor as mentioned above for the benefits. If not we will just do more rebalance round with little to no benefit.

⚠️ Commits still cannot succeed during these whole rebalancing.

Revamp: Next Generation Rebalance Protocol (KIP-848)

I realised this probably needs a write-up by itself. I will just give a short summary of it here and maybe do a post on its own in the future.

Goals:

1. Move logic to the Coordinator to make clients simple.
2. No group wide synchronization barrier. It should be truly incremental and cooperative.
3. It should still allow client’s to specify assignor.
4. Same guarantee of previous protocol of at-least-once in worst case and exactly-once in best case.
5. No downtime during upgrades.

The key point is that the consumer now uses a new ConsumerGroupHeartbeat API that lets the Coordinator to assign/revoke partitions to/from group members.

So when a new consumer joins:

1. The Coordinator receive the ConsumerGroupHeartbeat request from the new member with no memberId and epoch 0.
2. Coordinator computes the topic-partitions to consumer instances assignment.
3. Coordinator revokes partitions that are no longer in the target assignment of the member and send back via the ConsumerGroupHeartbeat response.
4. Coordinator waits for the ack via the next round of ConsumerGroupHeartbeat request and response. This should be done within a rebalance timeout interval.
5. Coordinator can now assign the partitions to new member.

This process continues until the assignment reaches the target assignment computed by the Coordinator.

We can see that in the process,

• Partitions belonging to a consumer that is not revoked can continue to fetch and commit offsets.
• If a consumer do not have partition needing revoke, it will not be affected.
• There is also no group wide synchronization barrier.
• Client does not keep any state, the Coordinator manages all these complexity (including persisting the group metadata, epochs etc.)

References

• Consumer Group Protocol
• KIP-345
• KIP-429
• KIP-848