Redpanda Cluster Control: raft0, Moves, and Limits

You decommission a broker, the server process that hosts replicas and participates in cluster membership, permanently removing its identity. The command returns, but data still has to move, and new learners, non-voting replicas, must catch up before promotion. Controller state, the replicated cluster metadata describing membership and partition assignments, must record each completed reconfiguration, a transition from one replica assignment to another, while the cluster retains a quorum, a voting majority, throughout the drain.

The button is not the operation.

Redpanda's controller turns cluster changes into ordered metadata commands, but the controller cannot make membership, recovery bandwidth, disk capacity, or product licensing disappear. Cluster management is a state machine plus a resource plan.

This post pins implementation details to Redpanda v26.1.13 and uses the July 2026 self-managed documentation. The partition-level Raft details are out of scope here; the focus is cluster-wide decisions.

raft0 is the control plane

Redpanda stores cluster metadata in the controller partition, a system partition replicated by Raft group zero, commonly called raft0.

The tagged create_raft0() path creates the controller log around model::controller_ntp and group ID zero. The pinned architecture documentation says every cluster broker participates in this Raft group.

Controller commands cover state such as:

• broker membership;
• topic creation and deletion;
• partition assignments and movements;
• cluster configuration;
• security metadata;
• feature activation;
• enterprise subsystem metadata.

The source types live in cluster/commands.h, and controller_stm.cc applies the replicated batches to controller state.

Admin or Kafka metadata operation
  → controller leader validates request
  → raft0 replicates command
  → controller state machines apply command
  → broker-local reconciliation performs the work

The last arrow belongs to a reconciler, broker-local code that makes actual partition state match controller intent. A topic-creation command can commit quickly while partition creation continues across brokers. A partition-movement command can commit before the learner finishes copying data.

Snapshots bound replay; they do not replace the log

The controller is a replicated state machine. Replaying its entire history on every restart would make startup cost grow with cluster age, so Redpanda writes controller snapshots.

The pinned docs state that controller snapshots are stored on each broker and hydrated on join or restart. The snapshot captures current metadata state; the log remains the ordered source of commands after that snapshot.

Three objects have different jobs:

A healthy data plane, the partition leaders and request paths that serve client traffic, can briefly coexist with an unhealthy control plane. Existing partition leaders may serve traffic while topic creation, decommissioning, or configuration changes fail.

Topic creation is placement plus reconciliation

Creating a topic requires more than inserting a name. The controller chooses replica assignments, records them in raft0, and brokers reconcile those assignments into local storage and partition Raft groups.

Redpanda's partition_allocator works with per-broker and per-shard allocation state. The partition_manager creates the local log, offset translator, Raft group, and partition wrapper on the assigned shard.

Placement attempts to distribute replicas. It cannot predict application hot keys, uneven message sizes, or a consumer pattern that concentrates work on a few leaders.

This is why “balanced partition count” and “balanced load” are different claims.

Leadership balance and replica balance move different things

Redpanda exposes several balancers:

The distinctions come from the pinned cluster-balancing guide.

Leadership transfer chooses another existing replica to lead. Replica movement copies partition data to another broker, catches the new learner up, and changes the Raft membership. Intra-broker balancing does not copy segment data to another disk or broker. The tagged transfer_partition() path stops the partition on the source shard, copies per-shard persistent state, updates placement, and lets reconciliation reopen the same on-disk log from the destination shard.

Each operation consumes a different budget:

• leadership changes consume election and request-routing capacity;
• cross-broker moves consume disk, network, and recovery bandwidth;
• cross-core moves consume shutdown/reopen time, per-shard state transfer, and shard-management capacity rather than a second copy of segment data.

Turning on more balancing does not guarantee lower latency. During a busy recovery, foreground traffic and movement share the same hardware.

Decommissioning is a permanent membership change

The pinned decommission guide defines the sequence:

1. mark the broker as decommissioning so new partitions avoid it;
2. let the controller create a reallocation plan;
3. move replicas in bounded batches;
4. wait for every partition-level reconfiguration;
5. remove the broker from cluster membership.

The exact control is partition_autobalancing_concurrent_moves, which defaults to 50 simultaneous reassignments. That is a concurrency default, not a duration estimate.

The planner below turns that concurrency into waves under one explicit size assumption. Select the target broker to see how remainder placement changes the drain.

More concurrency shortens the wave count, but it does not bound source-broker failure exposure. Until each move completes, every replica still assigned to the target broker can become one replica short; the planner reports that full upper bound.

Once decommission finishes, the broker cannot rejoin with the same identity. If the process is still decommissioning, the recommission command can stop the active decommission. It cannot restore a broker that has already been removed.

That point should appear in automation. Reusing a removed node ID is not a rollback.

Recovery bandwidth decides how long “self-healing” takes

A replacement replica joins as a learner and copies missing log history. Redpanda can redistribute recovery bandwidth across shards, but the cluster still has finite disk and network capacity.

The pinned Continuous Data Balancing guide lists cases that stall movement:

• too few healthy nodes;
• insufficient free disk;
• loss of partition quorum;
• maintenance mode on a required node.

It also separates two failure policies:

• after the availability timeout, Redpanda can rebuild replicas elsewhere while keeping the unavailable node in membership;
• an optional auto-decommission timeout permanently removes the node.

The first is reversible when the node returns. The second is not.

My default operational stance is conservative: automate replica replacement before automating permanent identity removal. A transient network failure is not evidence that the node should never rejoin.

Feature gates make mixed-version clusters explicit

Rolling upgrades create a period in which brokers run different binaries. Redpanda's feature system prevents a new node from immediately emitting every new controller format.

The tagged feature_table maps features to a required cluster version and an activation policy. Some gates apply automatically when the cluster version is ready; others preserve a cluster-history boundary.

In v26.1.13, group_based_authorization, user_based_client_quota, ordered_leaders_pinning, and cloud_topics all require the 26.1.1 cluster version and use the always availability policy. By contrast, validated_batch_timestamps uses new_clusters_only, so a cluster upgraded from an older feature line can retain the previous timestamp behavior until an explicit migration. “Feature gated” therefore does not mean “every feature has a manual enable button.”

This separates two events:

1. installing a binary that understands a format;
2. activating the feature that writes that format.

The pinned rolling-upgrade guide also requires intermediate feature releases when a cluster is more than one feature release behind. Upgrade plans should preserve that supported path rather than jumping from an old binary to the latest patch.

Limits are guards, recommendations, and tested envelopes

Redpanda publishes several partition numbers that answer different questions. Treating them as interchangeable produces bad capacity plans.

The per-shard property appears in the pinned cluster-property reference. The 1,000-per-core and 50,000-per-cluster guidance appears in the pinned decommission capacity checklist. The pinned sizing guide recommends 2 MB per partition replica.

The 5,000 guard is not a supported-performance target. A cluster can pass topic creation and still miss its latency objective because throughput, transactions, consumer groups, compaction, and recovery all add per-partition work.

Product edition changes who owns the balancing loop

Without an Enterprise license, partition_autobalancing_mode=node_add is the default. With a valid Enterprise license, continuous balancing can monitor broker availability, rack placement, and disk thresholds.

Redpanda Cloud moves more responsibility to the vendor, but the application still owns:

• partition count and key distribution;
• replication and retention requirements;
• client retry and timeout behavior;
• business recovery after duplicate or delayed processing.

Managed does not mean unlimited. Use the Cloud service limits and support contract for the exact tier rather than self-managed defaults.

The controller cannot schedule around these constraints

The first failure below has a pinned fix; the rest follow from source and documentation.

Decommission races with membership change

Redpanda v26.1.13 fixed a cluster membership lock that could occur when a node was decommissioned while being added as a raft0 learner. That release note is a concrete reason to pin patch versions and avoid overlapping node-lifecycle operations casually.

The controller is healthy but a partition move cannot finish

raft0 can order the request while the target broker lacks disk or the source partition lacks quorum. Monitor controller health and movement progress separately.

Continuous balancing creates foreground pressure

Self-healing copies data. Cap movement concurrency and recovery bandwidth against the degraded-state traffic objective, not against an idle cluster.

A hard guard is mistaken for a safe operating point

5,000 replicas per shard is a creation guard. Stay within the documented operations envelope and validate latency, memory, startup, and failure recovery at the planned partition count.

Feature activation closes a rollback path

New binaries may be reversible before a new controller format is activated. Record the cluster feature state before and after each upgrade phase.

A controller runbook should expose state, not hide it

Before a node or version change, capture:

rpk redpanda admin brokers list
rpk cluster health
rpk cluster partitions balancer-status
rpk cluster config export --filename cluster-config.yaml

During decommission:

: "${BROKER_ID:?Set BROKER_ID after verifying the broker list}"
 
rpk redpanda admin brokers decommission "$BROKER_ID"
rpk redpanda admin brokers decommission-status "$BROKER_ID"
rpk cluster partitions move-status

If the decision changes before completion:

: "${BROKER_ID:?Set BROKER_ID after verifying the broker list}"
rpk redpanda admin brokers recommission "$BROKER_ID"

The : guard stops the shell before either lifecycle command unless BROKER_ID is set explicitly.

Command availability and flags can change, so use the rpk binary shipped with the target Redpanda release and save rpk version in the change record.

Trace the controller state machines

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "create_raft0" -- src/v/cluster/raft0_utils.h
git grep -n "feature_schema" -- src/v/features/feature_table.h
git grep -n "partition_autobalancing" -- src/v/cluster
git grep -n "recommission" -- src/v/cluster

The source trace identifies the state machines. A safe change record adds broker IDs, racks, partition counts, free disk, movement bandwidth, controller leader, feature version, and rollback boundary.

Cluster management is serialized intent plus asynchronous work

• raft0 orders cluster metadata; broker reconcilers perform the resulting local work.
• Leadership balancing and replica balancing consume different resources.
• Recommissioning can stop an active decommission, but completed decommission is permanent.
• Continuous balancing and continuous intra-broker balancing are Enterprise features.
• Partition limits must be labelled as guards, recommendations, or tested envelopes.
• Mixed-version upgrades rely on cluster feature gates, not only binary compatibility.

Previous: Redpanda Internal RPC: Versioned Bytes Between Brokers ←

Sources

• Pinned architecture and controller snapshots
• Pinned broker decommissioning guide
• Pinned cluster-balancing guide
• Pinned Continuous Data Balancing guide
• Pinned rolling-upgrade guide
• Pinned cluster-property reference
• controller_stm.cc at v26.1.13
• controller_backend.cc shard transfer at v26.1.13
• feature_table.h at v26.1.13
• Live Redpanda documentation — secondary, moving reference