Kafka 4.x Client Compatibility and KIP-848 Migration

The Kafka 4.3.1 brokers are stable. Old consumers still read records. Then one canary adds group.protocol=consumer, and the migration stops being a broker upgrade. It becomes an assignment and application-behavior change.

Three switches are independent:

1. the broker binary;
2. the cluster's finalized group.version;
3. the consumer's group.protocol.

Client compatibility asks whether a client and broker can select compatible request versions, authenticate, and complete the APIs the application actually uses. KIP-848 migration asks whether one consumer group can move from the classic rebalance protocol, the JoinGroup and SyncGroup path, to the consumer rebalance protocol, Kafka's broker-driven replacement.

A broker upgrade proves that old behavior still works. It does not prove that the new group protocol preserves your assignment contract.

The KRaft cutover belongs to the previous post. Transaction protocol v2 belongs to Kafka Producer Idempotence, Fencing, and Transactions. Keeping those state machines out of this rollout makes failure attribution possible.

Kafka 4.x has a protocol floor, not a universal client certificate

KIP-896 sets Kafka 2.1 as the general protocol baseline for 4.x and removes selected older request versions. A protocol floor is the oldest wire contract the broker retains. It does not certify a language client, framework, connector, authentication path, or wrapper built above that contract.

Compatibility failures have two shapes:

Kafka 3.7 added the broker metric DeprecatedRequestsPerSec, tagged with request, version, client software name, and client software version. Watch it long enough to include batch jobs, connectors, disaster-recovery tools, and administrative clients. A quiet hour only proves the hourly workload is quiet.

The 4.0 line exposed why Produce and Fetch smoke tests are insufficient. KAFKA-19444 records a librdkafka Generic Security Services API (GSSAPI) failure on its Kerberos path: Kafka 4.0.0 had removed JoinGroup v0 and v1, while librdkafka still inspected JoinGroup v0 during its capability check. Apache restored those versions in later patches while librdkafka shipped its fix.

The application did not need an old rebalance schema. Its authentication path did.

Use the latest supported patch in the chosen line. Test every client family through connection, authentication, ApiVersions, metadata, Produce, Fetch, offset commit, group membership, and required Admin APIs. For older Java clients, Connect, and Streams, KIP-1124 defines supported bridge paths that go beyond wire compatibility.

group.version=1 enables the server; it does not convert a group

A feature level is a finalized cluster-wide version flag. In Kafka 4.3.1, group.version=0 is the original coordinator and level 1 enables KIP-848. Level 1 requires metadata.version at least 4.0-IV0.

Inspect and update the feature separately from the application release:

# ① Confirm metadata.version and the supported group.version range.
bin/kafka-features.sh \
  --bootstrap-controller controller-1:9093 \
  describe
 
# ② Validate the narrow server-side update.
bin/kafka-features.sh \
  --bootstrap-server broker-1:9092 \
  upgrade --feature group.version=1 --dry-run
 
# ③ Apply only after every broker supports it.
bin/kafka-features.sh \
  --bootstrap-server broker-1:9092 \
  upgrade --feature group.version=1

① must show the metadata prerequisite and broker support. ② asks the controller to validate the feature change. ③ enables the server path. Existing classic groups stay classic until a client opts in.

Kafka 4.3 deprecates group.coordinator.rebalance.protocols. The 4.3 upgrade guide states that feature versions become the long-term server switch. The per-application switch remains:

group.protocol=consumer
group.remote.assignor=uniform

group.remote.assignor is optional. If the client omits it, the coordinator uses the first entry in group.consumer.assignors; Kafka 4.3.1 orders uniform before range.

KIP-848 moves assignment policy into the coordinator

A group coordinator is the broker component that stores membership and assignment state. KIP-848 replaces the classic group-wide barrier with a desired assignment and per-member reconciliation loop. The coordinator tells each member what to revoke and acquire. A member whose assignment does not change need not pause for another member's move.

Policy ownership changes with that state machine:

The Apache 4.3 consumer-protocol guide maps classic assignors to the built-in server assignors:

The mapping is not equivalence. Two algorithms can both look balanced while moving different state, breaking co-partition assumptions, or changing which member receives a costly partition.

A custom assignor becomes a broker deployment

The consumer protocol does not run a custom classic assignor inside the client. To keep a custom policy, implement Kafka's ConsumerGroupPartitionAssignor interface and configure its class in group.consumer.assignors.

That changes the release owner:

1. build and test the assignor against the target broker patch;
2. place the same JAR on every broker that can lead a __consumer_offsets partition;
3. configure the full class name on every broker and roll the fleet;
4. set group.remote.assignor to the name returned by the assignor;
5. move a group only after coordinator failover proves the class is available everywhere.

The 4.3.1 coordinator source loads built-in names or instantiates configured classes. A JAR on one broker can therefore work until coordinator leadership moves.

Online migration is supported only when the classic assignor does not embed custom metadata. Rack-aware server assignment also remains incomplete. Keep the group classic until the server policy reproduces the behavior the application needs.

Three edge cases decide whether the canary is representative

Regex subscriptions are eventual and use RE2/J

Kafka's SubscriptionPattern stores an expression compatible with RE2/J, the Java port of Google's RE2 regular-expression engine, and delegates validation to the broker. Replacing subscribe(Pattern) is therefore an API and regex-language change.

The immutable 4.3.1 coordinator source sets group.consumer.regex.refresh.interval.ms to 10 minutes and group.consumer.assignment.interval.ms to 1 second. The regex refresh value is an internal source-level setting, not a public tuning promise. Test how long a newly created matching topic takes to enter the assignment. Do not promise instant discovery because the expression matches.

Static membership extends the real failure clock

Static membership gives a consumer a stable identity through group.instance.id. It can preserve assignments across short restarts.

max.poll.interval.ms still limits how long the application can go without poll(). When a static member exceeds that interval, the client stops heartbeats, but Kafka waits for the broker-owned consumer session timeout before reassigning its partitions.

A long session timeout can smooth a deployment and prolong a real stall. Test both:

• restart the same instance ID inside the session timeout;
• leave it absent past the timeout and measure reassignment;
• block processing past max.poll.interval.ms;
• time revoke and assign callbacks separately from broker detection.

Migration policy controls mixed non-empty groups

Kafka 4.3.1 defaults group.consumer.migration.policy to bidirectional. The release-pinned source defines bidirectional, upgrade, downgrade, and disabled. Empty groups can convert in either direction regardless of the policy; the setting controls non-empty groups.

Inspect it before choosing an online rollout. A runbook cannot promise an online downgrade that the coordinator policy rejects.

Choose the cutover from the assignment contract

My judgment: mixed membership is a transition and rollback tool, not a stable operating model. It doubles the protocol paths an operator must explain while assignments are moving.

KIP-1274 is accepted. Kafka 4.3 logs a recommendation when KafkaConsumer uses classic. The accepted plan targets the consumer protocol as the default in 5.0 and removal of classic support from KafkaConsumer in 6.0. Those releases are future targets, not evidence about binaries that do not yet exist.

What could go wrong during conversion?

• An indirect client dependency fails first. KAFKA-19444 shows that authentication code can depend on a group API version the application never calls directly.
• The assignor changes valid business placement. A balanced assignment can still violate co-partitioning, state locality, or warm-cache assumptions.
• A custom assignor disappears on coordinator failover. Broker-local plugins need the same artifact and configuration on every eligible coordinator host.
• A regex topic arrives later than the deployment assumes. Server-side refresh makes matching eventual.
• Static membership hides a dead worker. A long session timeout delays partition reassignment after the application stops polling.
• A slow callback looks like broker instability. Revoke and assign work runs in application code; measure it separately from coordinator progress.

The new protocol removes a global barrier. It does not remove application work from the handoff.

Prove one group through five evidence gates

Kafka 4.3.1 exposes useful broker and client signals. Exporters may rename them, so verify the emitted names before writing alerts.

Do not collapse these into one "rebalance time" chart. Stable broker convergence with slow callbacks points to the application. Fast callbacks with groups stuck in Reconciling point to coordinator state or assignment policy.

Roll out in this order:

1. Inventory. Record client implementation and version, authentication, assignor, subscription API, static membership, poll interval, callback work, and broker migration policy.
2. Upgrade the client while staying classic. Separate dependency, authentication, API, and threading changes from the group-protocol change.
3. Prepare the server contract. Require group.version=1, deploy any custom assignor everywhere, and test regex and timeout behavior.
4. Convert one production group. Choose a group with known duplicate and side-effect behavior. Compare it with its own healthy baseline.
5. Finish or reverse. Move every member to the consumer protocol or return before the group breaches its service objective.

There is no useful fleet-wide rebalance threshold. Group size, callback work, state restoration, and partition cost differ. The gate is a measured result against that group's own assignment and processing contract.

KIP-848 gives the broker more control. A safe migration starts by proving what the application must not lose.

Next, the series leaves protocol migration and asks what a change in corporate ownership should, and should not, change in an engineering decision.

Sources

Apache specifications and operations

• Kafka 4.3 Consumer Rebalance Protocol guide
• Kafka 4.3 consumer configuration
• KIP-848: The Next Generation of the Consumer Rebalance Protocol
• KIP-896: Remove old client protocol versions
• KIP-1124: Kafka client upgrade paths for 4.x
• KIP-1274: Classic protocol transition plan
• KAFKA-19444: librdkafka GSSAPI regression

Kafka 4.3.1 source

• GroupVersion.java
• GroupCoordinatorConfig.java
• SubscriptionPattern.java
• GroupCoordinatorMetrics.java
• ConsumerRebalanceMetricsManager.java
• RebalanceCallbackMetricsManager.java
• HeartbeatMetricsManager.java