Kafka Replication: How the Next Leader Proves Durability

Your Kafka producer, the client that appends records, receives a success response. Then the leader—the broker authorized to accept writes for its partition, one ordered log shard—dies. The replacement must decide whether the old leader’s final records belong to committed history or to a divergent tail.

“It existed on multiple disks” doesn’t settle that question. A stale replica can have extra records, while a newer replica can have a shorter but authoritative history.

Here’s my stricter test: a write is durable only when the next eligible leader—a replica Kafka may elect—can prove it. Kafka reached that rule by discovering, one failure at a time, that copy count, a local watermark, and one epoch comparison each left room for ambiguity.

ISR made “caught up” a live claim

A partition’s replication factor is the number of assigned copies. One replica leads; the other followers pull records from it. Assignment alone says nothing about how current a copy is, so Kafka maintains the in-sync replica set (ISR): the replicas currently participating in the partition’s committed history.

ISR isn’t a fixed majority. It can contain all three replicas, two replicas, or only the leader. A follower may remain in ISR while momentarily behind, but the leader removes it if it stops fetching or fails to catch the leader within replica.lag.time.max.ms, which defaults to 30,000 ms.

Kafka originally used message-count lag as part of this decision. KIP-16 removed that threshold because a fixed record count confused traffic bursts with unhealthy replication. Time since the follower last caught up better reflects whether it can still follow the log.

This distinction matters during failure. An assigned replica may hold most of the log yet remain outside ISR. Conversely, shrinking ISR changes which replicas constrain the committed boundary. Kafka therefore needs more than an ISR list; it needs an exact offset up to which those replicas agree.

The high watermark is an offset boundary, not a record number

An offset identifies a record’s position in a partition. Kafka represents important positions as next-offset boundaries:

• The log end offset (LEO) is where the replica would append its next record.
• The high watermark (HW) is the boundary below which Kafka considers records replicated across the required in-sync history.

In the ordinary case, before eligible leader replicas enter the calculation, the leader can advance HW no farther than the smallest LEO it knows among ISR members. The replicated-log design uses this boundary to separate committed records from an uncommitted tail.

Consider three replicas:

The leader calculates HW = 42. Records with offsets below 42—through offset 41—belong to the committed prefix. Record 42 does not, even though two replicas already have it.

Once follower C appends record 42, its LEO becomes 43. The leader can then move HW to 43, making record 42 committed. Thinking in boundaries avoids the common off-by-one mistake: HW 43 commits record 42; HW 42 does not.

A leader’s LEO can run ahead of HW. Those extra records are real bytes, but Kafka may discard them after an election if they conflict with the authoritative history. The leader’s local append is evidence of receipt, not yet evidence of durability.

Transactions add a visibility boundary, not another replication quorum

Transactions introduce another offset that answers a different question. The last stable offset (LSO) is the first offset that a consumer configured with isolation.level=read_committed must not cross.

Conceptually, LSO is the lower of HW and the first offset belonging to an open transaction, a group of records that must commit or abort together. KIP-98 introduced this boundary and the metadata needed to filter aborted transactions.

Suppose HW is 120, but the oldest open transaction begins at 113. LSO remains 113, so a consumer, the client reading records, using read_committed receives only records below 113. Even a non-transactional record at offset 118 waits behind that open transaction because Kafka preserves partition order.

LSO does not make replication more durable. HW still marks the replicated prefix; LSO marks the stable portion currently visible to transactional consumers. A widening HW–LSO gap usually points to transaction progress, not follower replication failure.

Followers acknowledge replication by asking for the next offset

Kafka followers don’t receive pushed records and then send a dedicated acknowledgement request. They use the same Fetch request family that consumers use, with replica-specific semantics. The request includes the next offset the follower needs, while the response carries records and the leader’s current HW. The protocol reference documents both fields.

That design turns the next Fetch into evidence about the previous one. Isolate one follower and assume both replicas remain in ISR:

① By asking for offset 42, the follower says its log already contains every required offset below 42.

② The response supplies record 42 and piggybacks the leader’s current HW. There is no separate “update your watermark” request.

③ The next request for offset 43 tells the leader that follower B appended through offset 42. Once every required replica reports enough progress, the leader can advance HW and complete a pending acks=all produce request.

The follower’s local HW may trail the leader’s until a later Fetch response carries the new value. That propagation delay once interacted dangerously with truncation, but modern recovery does not blindly treat a stale local HW as the truncation oracle.

The visualization below assumes one partition with replication factor three, all three replicas staying in ISR, no transactions, no retries, and no elections. Its S0–S8 markers show protocol sequence, not measured latency. Treat each LEO and HW value as a next-offset boundary. In the acks=0 view, “acknowledgement” means the producer stops waiting; the broker sends no Produce response.

The event to watch isn’t a particular sequence marker. It’s the follower’s second Fetch: the requested offset acts as the replication acknowledgement, while the response carries commit progress in the other direction.

acks=all needs min.insync.replicas to preserve its meaning

The producer’s acknowledgement policy controls what evidence it requires before treating a request as successful:

• acks=0 waits for no broker response.
• acks=1 waits for the leader’s local append.
• acks=all waits until every current ISR member has replicated the record.

min.insync.replicas, usually shortened to minISR, is an admission floor. With acks=all, Kafka rejects a write when the ISR contains fewer than minISR replicas. It may return NOT_ENOUGH_REPLICAS before appending or NOT_ENOUGH_REPLICAS_AFTER_APPEND if the ISR shrinks while the request waits.

MinISR does not tell acks=all to wait for only that many replicas. If ISR contains three members and minISR is two, the producer still waits for all three. MinISR decides whether Kafka may accept the write at all.

The table uses acks=1, minISR one as its latency and durability baseline. It deliberately avoids universal RTT multipliers; batching, follower Fetch cadence, storage, and network topology determine the actual wait.

For a critical three-replica topic, my starting policy is acks=all, min.insync.replicas=2, and unclean.leader.election.enable=false. That is a trade-off, not a universal durability promise: it preserves a two-replica success floor while allowing one replica to be unavailable.

A slow follower still matters. While it remains in ISR, acks=all waits for it. If Kafka removes it, minISR decides whether writes may continue. Workloads that can reconstruct records may rationally choose acks=1 instead of accepting that availability cost.

Historical behavior: a stale HW once made committed data look disposable

The Fetch sequence exposes an awkward interval. The leader can know that all ISR members copied a record before a follower learns the updated HW.

Suppose record 42 produces this state:

1. Leader and follower both reach LEO 43.
2. The follower requests offset 43, proving it appended record 42.
3. The leader advances HW to 43 and acknowledges the producer.
4. The leader fails before its next Fetch response gives the follower HW 43.
5. The follower still has record 42, but its local HW remains 42.

The next step describes historical, pre-fix behavior—not current Kafka recovery. A recovery path that truncated to the replica’s stale local HW could cut the log back to 42, deleting record 42 even though the producer had received success. KIP-101 documents why HW was unsafe as the sole truncation input. KAFKA-7128 was a separate older ISR-expansion bug: a joining replica had to catch up within the current leader epoch before becoming eligible to constrain committed history.

KIP-101 introduced the leader epoch, a monotonically increasing identifier for each period of partition leadership. Record batches carry their leader epoch, and replicas retain an epoch-to-offset history.

During recovery, a follower uses the OffsetForLeaderEpoch API to ask the current leader where a shared epoch ends. It can then preserve the common history and truncate only the conflicting suffix. HW remains the commit and visibility boundary; leader epochs provide the lineage evidence needed for safe truncation.

That split is important. A watermark says, “records below this offset were committed from my point of view.” An epoch history says, “this exact sequence of leaders produced the records up to this boundary.” Recovery needs both meanings, not one overloaded number.

KIP-279 caught divergence during Fetch, not just leader changes

KIP-101 fixed reliance on stale HW, but rapid consecutive elections exposed another gap. KAFKA-6361 showed that replicas could retain different records at the same offsets after fast leader failover.

KIP-279 extended Fetch so a follower reports the epoch associated with its fetch position. If the leader’s epoch history disagrees, the response identifies a diverging epoch and end offset. Recovery can work backward until both logs find a common epoch, then truncate the follower’s conflicting suffix.

The non-obvious lesson is that matching offsets do not prove matching history. Offset 500 can name different records on two divergent logs. Epochs make the offset meaningful.

KIP-966 preserved the best fallback after ISR collapsed

Leader epochs prove where histories diverge, but Kafka still had to decide which history could lead. ISR alone could discard useful election evidence during the “last replica standing” failure.

Assume replication factor three, minISR=2, acks=all, and replicas A, B, and C:

1. C falls behind and leaves ISR. A and B can still acknowledge new writes.
2. B later leaves ISR, so only leader A remains.
3. New acks=all writes now fail because ISR is below minISR.
4. B still contains every record acknowledged before the ISR crossed that floor. C may not.
5. A becomes unavailable.

Before eligible leader replicas, the controller’s ISR metadata named only A as a clean candidate. Kafka could wait for A or perform an unclean leader election, choosing outside the proven set and risking a stale log. The metadata no longer distinguished B, the useful fallback, from C, the older copy.

KIP-966 introduced Eligible Leader Replicas (ELR) for non-ISR replicas that Kafka can still prove contain the committed prefix. Its first phase shipped in Kafka 4.0. ELR is a finalized cluster feature rather than a normal broker toggle: level 1 requires metadata.version at least 4.0-IV1, and it becomes the default feature level for release versions 4.1 and later.

After every broker and controller supports the feature, an upgraded cluster can enable it through the supported feature API:

bin/kafka-features.sh --bootstrap-server broker-1:9092 \
  upgrade --feature eligible.leader.replicas.version=1 --dry-run
 
bin/kafka-features.sh --bootstrap-server broker-1:9092 \
  upgrade --feature eligible.leader.replicas.version=1

The ELR operations guide requires checking existing min.insync.replicas values first because finalization creates a cluster-level value when one is absent and removes broker-level overrides. Confirm FinalizedVersionLevel: 1 before treating ELR as part of the recovery contract.

In the example, B can remain eligible when ISR falls below two because successful acks=all writes stop at that transition. C cannot inherit that status merely because it has some records. If A fails and B is available, the controller can choose B before falling back to the current last-known-leader recovery path.

ELR doesn’t make every old replica safe, repair a lost disk, or merge conflicting tails. It preserves election evidence that the old ISR model threw away. The exact Kafka 4.3 order depends on the election type:

The controller's PartitionChangeBuilder uses lastKnownElr as a one-element recovery slot when ISR and ELR are empty, not as a general pool of old eligible replicas. Electing it enters recovering state and is marked unclean.

The visualization illustrates why preserving B as an eligible successor matters. Its scenario assumes the replicas' logs match the offsets shown; it does not encode every Kafka 4.3 election fallback or claim ELR can reconstruct missing data.

Partition replication is separate from KRaft’s metadata quorum

The controller that records ISR, epochs, and ELR belongs to KRaft’s separate metadata quorum. KIP-595 defines how controller voters replicate cluster metadata.

User partitions do not become Raft groups. Controller voters do not acknowledge producer records, and their majority does not replace a partition’s replication factor, ISR, or minISR. A healthy KRaft quorum can coexist with under-replicated or offline data partitions; it can record the problem without supplying the missing user data.

What could still go wrong

Kafka’s proof has a failure envelope:

• Replica append is not per-record durable media flush. Kafka relies heavily on the filesystem page cache and replication, as its filesystem design explains. Correlated power or storage loss can defeat several replicas at once.
• MinISR trades writes for evidence. acks=all with minISR=2 intentionally rejects writes after the second replica becomes unavailable. Raising minISR without planning for that outage moves the failure from data loss risk to producer errors.
• Unclean election bypasses the strongest proof. Electing a replica outside ISR or ELR may restore availability with a shorter history. Keep the option explicit and treat every unclean election as a possible loss event.
• Long transactions can hide replicated records. An open transaction can hold LSO behind HW, making read_committed consumers appear stuck while replication remains healthy.
• Replica count does not imply failure-domain diversity. Use broker.rack and inspect assignments. Three replicas sharing one failure domain still share that domain’s fate.

None of these points makes acks=all useless. They define what its success response proves—and what remains outside that proof.

Durability signals need a local baseline

Kafka exposes useful replication metrics, but one fleet-wide warning threshold rarely survives different traffic, storage, and maintenance patterns. Compare movement with each topic’s replication policy and the cluster’s normal behavior.

Alert on scope and duration, not a borrowed “non-zero for five minutes” rule. One partition at minISR and hundreds of partitions repeatedly crossing it describe different incidents, even when an aggregate dashboard gives them the same color.

A useful durability review reads the failover path backward: who may lead next, which history that replica can establish, where HW can land, and only then what success the producer receives. Starting with acks alone reverses cause and effect.

Replica count says where bytes may exist. HW identifies the committed prefix. LSO identifies its transactionally visible portion. Leader epochs prove lineage, and ELR preserves the right successor when ISR disappears.

Within Kafka’s stated storage and failure assumptions, the hard-won rule is this: a write is durable only when the next eligible leader can prove it.

Next, compression asks how to reduce those replicated bytes without choosing a codec by folklore.

Sources

Apache Kafka documentation

• Replication and committed-log design
• Kafka wire protocol
• acks producer configuration
• min.insync.replicas topic configuration
• Broker and replication monitoring

Protocol changes

• KIP-16: Automated Replica Lag Tuning
• KIP-98: Exactly Once Delivery and Transactional Messaging
• KIP-101: Leader-Epoch-Based Truncation
• KIP-279: Fast-Failover Divergence Recovery
• KIP-966: Eligible Leader Replicas
• KIP-595: KRaft Metadata Quorum

Historical failure reports

• KAFKA-7128: Lagging high watermark could lead to committed data loss
• KAFKA-6361: Log divergence after fast leader failover