Kafka 4.x Without ZooKeeper: Planning the KRaft Cutover

Your application canaries pass against Kafka 4.3.1. The first broker restart does not. Its configuration still contains zookeeper.connect, and Kafka 4.x has no ZooKeeper mode.

The last bridge release, a Kafka version that can operate on both sides of the metadata architecture change, is 3.9. The cluster must move to KRaft, Kafka's Raft-based metadata control plane, while brokers still run that bridge. Only after the migration is finalized can the cluster roll to 4.x.

That creates two separate commitments:

1. Move metadata authority from ZooKeeper to KRaft.
2. Advance 4.x binaries and finalized feature levels, cluster-wide flags that activate metadata and subsystem contracts.

Treating those commitments as one upgrade turns every rollback question into guesswork.

The consumer-group ownership model defines who should own this fleet migration. This post defines the state transitions that owner must prove.

As of August 1, 2026, Apache lists Kafka 4.3.1 as the latest supported release. The same page lists 3.9.2 as archived. It remains the final bridge and includes KIP-1252's ZooKeeper and KRaft AlterConfigPolicy compatibility switch, but it no longer receives the support status of a current branch. Plan the bridge as a controlled transition, not a long-lived destination.

The cutover has two one-way doors

A controller is the Kafka process that orders cluster metadata changes. A controller quorum is the set of KRaft voters that elects the active controller and commits the metadata log, the ordered __cluster_metadata history used to rebuild cluster state.

Apache's 4.3 upgrade guide requires ZooKeeper clusters to migrate before installing 4.x. Apache's KRaft operations guide identifies 3.9 as the last bridge.

The first one-way door is migration finalization. The second is any feature finalization whose metadata cannot be safely downgraded.

KRaft becomes authoritative before the first broker converts

KIP-866 uses dual writes so operators can return to ZooKeeper until the final step. The phases are easy to misread because "all brokers run KRaft" is not the same as "the migration is final."

After the initial load, the KRaft controller takes authority and mirrors each new metadata mutation into ZooKeeper. Broker conversion happens under that authority. The all-KRaft, pre-finalization phase is therefore the last state in which ZooKeeper still holds a current fallback copy.

The bridge preflight has several hard requirements:

• Brokers must run Kafka 3.9.1 or newer with inter.broker.protocol.version=3.9. For a new migration in August 2026, use 3.9.2 and enable KIP-1252's compatibility behavior when a custom AlterConfigPolicy needs it.
• Do not change metadata.version during the migration. The 3.9 limitations warn that doing so can break the cluster.
• Provision dedicated KRaft controllers. KIP-866 does not migrate into combined broker and controller processes.
• Keep controller node.id values outside the existing broker ID namespace. A converted broker preserves its old broker ID as its KRaft node.id.
• Move from kafka.security.authorizer.AclAuthorizer to org.apache.kafka.metadata.authorizer.StandardAuthorizer as the broker migration procedure requires.
• Repair failed log directories before converting a broker. During migration, a ZooKeeper broker with multiple log directories shuts down if any directory fails.

These are compatibility conditions, not tuning suggestions.

Rollback is a metadata repair procedure

Before finalization, Apache documents a return to ZooKeeper. The word "rollback" can make that path sound simpler than it is.

A controller epoch is a generation number used to fence stale controllers. During dual write, KRaft can place its controller epoch in ZooKeeper partition state. A returning ZooKeeper controller can then start with a lower epoch and reject those partition records as newer than itself.

KAFKA-20488 records a rollback that left partitions offline. The 3.9 runbook now tells operators to compare ZooKeeper's /controller_epoch with the KRaft epoch recorded in /migration, then raise the ZooKeeper value when required before restoring controller authority.

Another failure can block the migration before broker conversion. KAFKA-19480 documents the ZooKeeper security migration tool creating a null /migration znode. The KRaft migration loop then repeatedly failed while parsing that state. The documented workaround is to remove the malformed znode before retrying.

A credible rollback plan is falsifiable: another operator can identify the last completed phase, name the authoritative metadata store, and execute the epoch checks without discovering them during the incident.

KRaft removes ZooKeeper but not control-plane failure

KRaft moves the metadata dependency into Kafka. The dependency still needs isolation, durable identity, and direct health checks.

Dedicated controllers separate metadata work from broker pressure

Kafka supports broker, controller, and combined broker,controller process roles. Apache's KRaft deployment guidance says combined mode is suitable for small development use and should be avoided in critical deployments because brokers and controllers cannot roll or scale independently.

A majority of voters must remain available. Three controllers tolerate one controller failure; five tolerate two. That arithmetic says nothing about shared racks, credentials, maintenance automation, or storage. Place voters across the failure domains the runbook claims to survive.

Explicit formatting protects cluster identity

KRaft requires kafka-storage.sh format with the existing cluster ID. Apache explains why: automatic formatting could let a majority of controllers start with empty metadata logs and elect a leader missing committed state.

Treat metadata.log.dir, meta.properties, the controller directory ID, and the cluster ID as recovery assets. An empty replacement directory is evidence of a failed node, not permission to create a new cluster.

Quorum health needs its own command

Use the metadata quorum tool during every migration gate:

bin/kafka-metadata-quorum.sh \
  --bootstrap-server broker-1:9092 \
  describe --status

The output reports the active controller, metadata high watermark, follower lag, current voters, and observers. A healthy produce dashboard cannot prove that the quorum can commit a topic creation, broker registration, or leader change.

New binaries do not activate every new subsystem

A rolling binary upgrade changes what each node can support. Finalized feature levels decide which metadata schemas and subsystem behavior the cluster actually uses.

Kafka 4.3.1's --release-version path updates metadata.version and every non-zero production feature default for that release. Inspect the bundle before applying it:

# ① Capture the current finalized levels.
bin/kafka-features.sh \
  --bootstrap-controller controller-1:9093 \
  describe
 
# ② Print the release-to-feature mapping.
bin/kafka-features.sh \
  version-mapping --release-version 4.3
 
# ③ Validate the complete update.
bin/kafka-features.sh \
  --bootstrap-server broker-1:9092 \
  upgrade --release-version 4.3 --dry-run
 
# ④ Apply only after the binary roll passes its evidence gates.
bin/kafka-features.sh \
  --bootstrap-server broker-1:9092 \
  upgrade --release-version 4.3

① is the rollback baseline. ② shows the full change set. ③ checks whether the registered nodes support it. ④ crosses the feature boundary.

The immutable Kafka 4.3.1 feature definitions map release 4.3 to:

The 4.3 upgrade guide states that metadata downgrade is not supported across 4.3 because 4.3-IV0 changes metadata. The other feature levels also represent separate contracts. This is why a release-level command deserves a reviewed diff rather than a label such as "make the cluster 4.3."

Dynamic controller membership is a later migration

KIP-853 stores the voter set in the metadata log and permits one controller membership change at a time. Kafka 4.1 added the supported static-to-dynamic upgrade path.

Do not combine it with the ZooKeeper cutover. First prove the KRaft migration. Then roll binaries that support kraft.version=1, finalize that feature, and replace controller.quorum.voters with controller.quorum.bootstrap.servers on every broker and controller as the 4.3 procedure requires.

Every joining controller first needs the existing cluster ID and observer format:

bin/kafka-storage.sh format \
  --cluster-id <CLUSTER_ID> \
  --config config/controller.properties \
  --no-initial-controllers

After startup, wait until the metadata quorum tool shows that the observer has caught up. Then choose one membership path:

Kafka 4.3.1's pinned QuorumConfig sets automatic joining to false by default. KIP-853 documents the automatic membership protocol, while KIP-1186 defines automatic controller joining. Do not run the manual add command when automatic joining is enabled. In either branch, verify that the new controller appears in CurrentVoters before removing the old voter. One membership change at a time preserves an overlap between the old and new majorities.

Dynamic membership removes the all-controller shutdown once required for voter replacement. It does not make quorum changes concurrent or free of failure domains.

Replace calendar waits with evidence gates

The pause at each boundary should last long enough to exercise the operations that matter for that cluster. A fixed seven-day wait can still observe nothing useful.

Keep KIP-848 out of the KRaft cutover. The next post changes the consumer group state machine after the metadata control plane is already stable.

Kafka 4.x removes ZooKeeper from steady-state operations. Reaching that simpler architecture requires a more explicit transition and an honest point of no return.

Sources

Apache operations and releases

• Supported Apache Kafka releases
• Kafka 3.9 ZooKeeper to KRaft migration guide
• Kafka 4.3 KRaft operations guide
• Kafka 4.3 upgrade guide
• Kafka 3.9.2 release announcement

Design and failure records

• KIP-866: ZooKeeper to KRaft Migration
• KIP-853: KRaft Controller Membership Changes
• KIP-1186: Update AddRaftVoterRequest to support auto-join
• KAFKA-19480: malformed migration znode
• KAFKA-20488: controller epoch rollback failure

Kafka 4.3.1 source

• FeatureCommand.java
• Feature.java
• MetadataVersion.java
• KRaftVersion.java
• QuorumConfig.java