Redpanda Raft: From Acknowledgement to Durable Media

A producer, a Kafka client that writes a record batch, an encoded group of records appended together, receives success. The partition's leader—the replica, one broker-held copy of that partition log, currently accepting writes—then loses power, and another replica takes over. What, exactly, did that success prove?

The answer depends on Raft, the consensus protocol that orders a replicated log, but “replicated” is not precise enough. That success may prove only that the leader accepted the append, that a majority of replicas received it, or that required copies reached durable media, storage expected to retain bytes across power loss. Raft may also have advanced the commit index, the highest log position it may treat as committed, while a state machine, deterministic logic that turns committed entries into visible state, may or may not have applied the batch. Those boundaries can coincide, but Redpanda's source keeps them separate.

An acknowledgement policy tells the producer which broker evidence it requires before treating a write as successful. In Redpanda v26.1.13, Kafka acks selects a Raft consistency level, while write caching, a policy that can acknowledge quorum-replicated data before its background flush, decides whether acks=all waits for a majority of durable flushes or only a majority of replicated copies.

Implementation links in this post use Redpanda v26.1.13. The general Raft safety model comes from the Raft paper; Redpanda-specific behavior comes from the tagged source.

Five offsets answer five different questions

An offset is a position in a Raft log. Redpanda tracks several boundaries because no single offset can answer every durability and visibility question. An fsync asks the operating system and storage stack to persist prior file writes before reporting completion. A quorum is the voting majority a Raft group needs to make progress.

These are not interchangeable labels for one watermark. The tagged consensus interface exposes flushed, committed, majority-replicated, visibility-upper-bound, and last-visible indexes separately. In particular, last_visible_index() is the minimum of the majority-replicated index and the visibility upper bound.

The distinction starts in storage. Redpanda's segment_appender states that an append() completion may still be unflushed and not fsynced; the future returned by flush() is the durable local boundary.

Raft adds quorum state. The replication_monitor has separate waits for “majority replicated” and “committed.” The latter also detects a later truncation that replaces the appended entry with committed history from another term.

That is the mental model to keep:

append locally
  → replicate to followers
  → flush where policy requires it
  → advance quorum boundary
  → commit
  → apply or expose

The arrows describe dependencies, not fixed latency stages. Redpanda overlaps local append, follower RPC, and flush work where the consistency policy allows it.

The lab below lets you stop one write at each boundary, then remove the leader. Keep the replica count fixed while you compare client success with quorum and flush state.

The consequence is visible when write caching is on: a majority can acknowledge the batch before a majority crosses the modeled durable-media boundary.

Kafka acks maps directly to three Raft modes

The mapping is explicit in produce.cc:

switch (acks) {
case -1:
    return {raft::consistency_level::quorum_ack, timeout}; // ①
case 0:
    return {raft::consistency_level::no_ack, timeout};     // ②
case 1:
    return {raft::consistency_level::leader_ack, timeout}; // ③
}

① acks=all becomes quorum_ack.

② acks=0 gives the client no Produce response, although the broker still processes the request.

③ acks=1 completes at the leader stage and lets follower replication continue in the background.

The three values live in raft/replicate.h. The names are internal; Kafka clients still send ordinary Kafka Produce requests.

acks=1 is not a durable-media promise

leader_ack completes from the leader result in replicate_batcher.cc. The storage appender's contract tells us why wording matters: local append can finish before fsync.

An acks=1 success therefore proves that the elected leader accepted the batch under its current term. It does not prove a follower copied it or that the leader's drive made it persistent.

acks=all changes meaning when write caching changes

By default, Redpanda v26.1.13 sets write_caching_default to false. When a batch contains a quorum_ack request and write caching is disabled, replicate_batcher marks the append as requiring a flush.

The next distinction is source-visible:

• when flush is required, replicate_entries_stm::wait_for_majority() waits for the committed boundary;
• without a required flush, it waits for the majority-replicated boundary.

The leader commit calculation in consensus.cc uses flushed progress and caps the commit index at the leader's flushed offset. Under the default write-caching policy, acks=all therefore waits for quorum replication with the required flush path.

The quorum calculation is concrete. For the leader it uses _flushed_offset; for each follower it uses min(last_flushed_log_index, match_index). The commit index can cross the appended offset only when a voting majority's flush-aware match indexes, including the leader's, have crossed it in the current term.

If an operator enables write_caching_default=true or the topic-level write.caching=true, acks=all can complete after a majority acknowledges the write without waiting for disk flush. The source documents the background limits:

• raft_replica_max_pending_flush_bytes defaults to 256 KiB;
• raft_replica_max_flush_delay_ms defaults to 100 ms;
• the first threshold reached triggers a flush.

Those defaults come from configuration.cc. They are runtime defaults, not a guarantee that every deployed cluster kept them.

The historical raft: Lazy flush mode for slow drives issue #1836 states the intended trade directly: acknowledge after a quorum has the write in memory to avoid per-message flush pressure, while accepting loss if the quorum fails before those copies reach durable media. The issue motivates the mode; it is not evidence that a particular production cluster lost data.

Assumptions behind the durability table

The table below assumes a three-broker topic with replication factor three, meaning three assigned copies, one replica per independent host, a crash-fault model rather than malicious or arbitrary corruption, and storage that honors a completed fsync. “Survives a minority failure” means one broker or storage device fails after the required boundary. It does not cover a shared power domain, a drive that lies about flush completion, simultaneous loss of a majority, or correlated filesystem corruption.

The last row is not “unsafe Raft.” It is a different durability contract. Write caching trades stable-media latency for a window in which the quorum's copies remain volatile.

Kafka durability properties do not carry Kafka semantics here

Redpanda accepts several Kafka topic properties so tooling does not fail, but marks them as irrelevant no-ops. They are accepted by compatible create or alter requests and then omitted from the effective Redpanda topic configuration; they are not retained as inert metadata. The pinned allowlist_topic_noop_confs includes:

• min.insync.replicas;
• unclean.leader.election.enable;
• flush.messages;
• Kafka replication-throttling properties.

This is a migration trap. In Apache Kafka, min.insync.replicas is an admission floor for acks=all. In Redpanda v26.1.13, the property disappears from effective configuration and does not change Raft quorum size or write availability.

Redpanda's own write-caching controls are write.caching, flush.ms, and flush.bytes, plus the cluster defaults described above. Copying a Kafka benchmark topic configuration with min.insync.replicas=2 and flush.messages=1 does not make those settings active on Redpanda.

This difference belongs in benchmark methodology: configuration files can look equivalent while the brokers enforce different contracts.

A partition is a Raft group; the controller is raft0

Every topic partition forms an independent Raft group, as the pinned architecture guide states. One leader handles appends; followers receive log entries; a majority defines quorum.

Cluster metadata uses another group. The tagged create_raft0() implementation creates group ID zero for the controller partition.

The topology is therefore:

raft0
  cluster membership, topics, assignments, features, configuration
 
partition Raft group N
  user records and partition-local state machines

There is no “one Raft group per core.” A core can host replicas from many Raft groups, and a Raft group spans the brokers assigned to that partition. Shard placement decides where a local replica runs; replication factor decides which brokers participate.

Batching reduces overhead without changing the quorum rule

Redpanda's replicate_batcher collects pending record batches behind a bounded semaphore, preserves each caller's consistency level, and dispatches them through one replicate_entries_stm. The batcher can share append and RPC work among requests, but it completes leader_ack and quorum_ack callers at different stages.

This is an important limit on the performance story:

Batching amortizes work. It does not weaken the quorum condition selected by the request.

Follower state enables multiple replication operations and recovery work to progress without serializing the entire broker. The exact throughput depends on batch shape, disk, network, recovery traffic, and shard load. The source does not justify a universal “100× faster Raft” claim.

Membership changes pass through learners and joint state

A learner is a non-voting replica that can catch up before participating in elections. Redpanda's group_configuration stores voters and learners and models simple, transitional, and joint configuration states.

The transition exists because changing a quorum in one step can create two disjoint majorities. Redpanda's configuration state machine adds replicas, waits for catch-up, and removes old voters through replicated configuration entries.

This mechanism underpins broker decommissioning and partition movement, but the Raft-level point is narrower: membership is log state, not an out-of-band edit.

Four ways the durability contract can fail

Under the assumptions above, four failure modes define the durability boundary.

A Kafka runbook relies on a no-op property

min.insync.replicas=2 can be accepted and then disappear from the effective Redpanda configuration. Verify write.caching, flush.ms, and flush.bytes instead, then test a broker loss. Do not infer durability from a familiar config name.

Write caching widens the power-loss window

Quorum replication protects against a minority of broker failures. It does not make volatile caches survive simultaneous power loss. If stable media is part of the requirement, leave write caching disabled or prove the storage and power failure model you are accepting.

A learner cannot catch up

Membership changes stall when the new replica lacks disk, network, or recovery bandwidth. Removing an old replica before the learner is ready would reduce fault tolerance, so the safe transition waits.

A majority is unavailable

Raft chooses safety over write availability when no quorum can form. Changing timeouts does not create another copy of the log. Restore a voting majority or recover through an explicit, documented disaster procedure.

Trace the acknowledgement path in source

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "acks_to_replicate_options" -- \
  src/v/kafka/server/handlers/produce.cc
git grep -n "enum class consistency_level" -- src/v/raft/replicate.h
git grep -n "has_quorum_ack_requests" -- src/v/raft/replicate_batcher.cc
git grep -n "wait_until_majority_replicated" -- \
  src/v/raft/replicate_entries_stm.cc
git grep -n "allowlist_topic_noop_confs" -- \
  src/v/kafka/server/handlers/topics/types.h

These commands establish the code path. A durability test still needs real hardware, explicit write-caching settings, and controlled broker and power failure scenarios.

Replication is not one binary state

• acks=0, acks=1, and acks=all map to distinct Raft completion stages.
• Local append, fsync, majority replication, commit, and visibility are separate boundaries.
• acks=all waits for the flush-aware commit path by default, but write caching changes that contract.
• Kafka's min.insync.replicas and flush.messages are accepted no-ops in Redpanda v26.1.13.
• Learners and joint configuration protect membership changes; they do not make recovery free.

Previous: Redpanda Architecture: Kafka Outside, Shards Inside ←

Sources

• Raft paper
• Pinned producer acknowledgement docs
• produce.cc at v26.1.13
• replicate_batcher.cc at v26.1.13
• replication_monitor.cc at v26.1.13
• consensus.h visibility boundaries at v26.1.13
• Follower flush-aware match index at v26.1.13
• Write-caching defaults at v26.1.13
• Raft group configuration at v26.1.13
• Live Redpanda documentation — secondary, moving reference