Redpanda Internal RPC: Versioned Bytes Between Brokers

Halfway through a rolling upgrade, replacing one broker, the server process that hosts partition replicas, at a time, half the cluster runs one patch and half runs the next. Kafka clients still speak the public Kafka protocol, the versioned request-and-response contract between clients and brokers. Internally, four kinds of work now cross a version boundary: partition recovery, copying or reconciling a replica's log after failure; a controller command, a replicated instruction that changes cluster metadata; a health check, a probe of node state; and a Raft RPC, an internal request or response used for consensus replication or elections.

That boundary has nothing to do with Avro, Protobuf, or the serialization format inside an application record.

Redpanda has two wire contracts: Kafka outside the cluster and its own Serde-plus-RPC protocol between internal services. The internal design favors bounded parsing, explicit compatibility versions, checksums, ordered dispatch, and fragmented buffers—but it does not make every operation zero-copy or every schema change safe.

This post pins source claims to Redpanda v26.1.13. It does not compare Serde with Protobuf or claim a universal microsecond budget; the tagged source does not establish either result.

Kafka records and internal RPC messages are different byte contracts

A Kafka client sends a versioned Kafka request. Redpanda parses it through the Kafka protocol definitions and handlers. An RPC (remote procedure call) is a request/response exchange between services. Redpanda's internal RPC transport uses Serde, its type-aware serialization framework, when brokers communicate with each other.

Changing an application's record schema, the field layout and encoding rules for its payload, does not change the Raft RPC header. Changing a Serde envelope does not change the Kafka Produce schema. Keeping those layers separate makes failures easier to locate.

A Serde envelope carries version, compatibility floor, and size

An envelope is a header around one internal serialized type. Redpanda's serde::envelope declares:

• the type's current version;
• the oldest compatible version;
• a payload-size boundary.

The regular envelope adds 6 bytes: one byte for current version, one byte for compatibility version, and four bytes for size. A checksum envelope adds a four-byte CRC-32C field.

The write path in serde/rw/envelope.h writes the two versions, reserves the size, serializes fields, then fills in the final payload length.

The read path in read_header.h rejects four important cases:

1. the claimed payload is larger than the remaining buffer;
2. the envelope would cross its parent's byte boundary;
3. the writer's compatibility floor is newer than the reader;
4. the writer's version is older than the reader's compatibility floor.

After known fields are read, the envelope reader can skip remaining bytes inside the declared size. Missing later fields retain their type defaults. That supports a common evolution pattern: append optional or defaultable fields while keeping the compatibility range honest.

It does not support arbitrary change. Reordering fields, changing a field's meaning, or raising the compatibility floor still requires coordinated versioning.

The compatibility panel below gives the parser a writer version, reader version, declared size, and optional trailing fields. Change one control at a time so a version rejection stays distinct from a byte-boundary rejection.

Unknown fields help only when the envelope's compatibility range and declared size remain valid. A successful skip cannot repair incompatible semantics.

struct node_status
  : serde::envelope<
      node_status,
      serde::version<1>,
      serde::compat_version<0>> {                  // ①
    model::node_id id;
    bool alive;
    std::optional<uint64_t> disk_free_bytes;       // ②
 
    auto serde_fields() {
        return std::tie(id, alive, disk_free_bytes); // ③
    }
};

① Version one declares the writer's shape while compatibility version zero permits a version-zero reader.

② The appended optional field needs a safe default when an older writer omits it.

③ Field order is the wire order; changing it is a compatibility change.

The example shows the shape of the API, not a type copied from Redpanda. A version-zero reader can ignore the trailing field only if the declared compatibility and field semantics make that safe.

Internal RPC v2 uses Serde and a 26-byte header

Redpanda's transport_version documents an important historical boundary: RPC v0 and v1 used the older ADL serializer and are no longer spoken as of Redpanda 23.2. The current minimum and maximum are both v2, which uses Serde.

Every internal RPC payload has a fixed 26-byte header:

The header is packed explicitly; readers should not infer its wire size from a C++ struct layout.

The correlation ID connects a response to a pending handler. The method ID selects the service function. The payload's Serde envelope then selects the type-level version rules.

types.cc computes the header CRC-32C, while netbuf.cc computes the 64-bit xxHash payload value. xxHash is a non-cryptographic hash; the upstream project describes the family that way. Neither field authenticates a peer or resists deliberate tampering. TLS and authentication own that security boundary.

The CRC coverage is narrower than “the whole 26-byte header.” The tagged checksum_header_only() function hashes compression, payload size, metadata, correlation ID, and payload checksum. It excludes the version byte and the CRC field itself. Protocol-version validation and peer authentication remain separate checks.

Compression is conditional, not a default win

Internal RPC supports none and zstd. The client option defaults to a 1024-byte compression threshold. In netbuf.cc, Redpanda compresses only when:

1. the caller requested Zstandard;
2. the uncompressed payload meets the threshold.

Otherwise, the header records none.

Compression changes the representation, so it is not a zero-copy operation. It can save network bytes while spending CPU and allocating output buffers. Whether that trade helps depends on payload size, entropy, network pressure, and the shard's CPU budget.

The broker computes the payload checksum after the optional compression step, then prepends the RPC header. A corrupted compressed payload fails before Serde can make sense of the body.

iobuf shares fragments selectively and has no copy-on-write

An iobuf is Redpanda's fragmented byte buffer. It stores a chain of Seastar temporary-buffer fragments and supports prepend, append, parsing, and range sharing.

Its class contract is more careful than “zero-copy means exactly that”:

• share() shares fragment storage without copying payload bytes;
• copy() creates independent fragments;
• appending raw pointers or strings always copies;
• appending another buffer may copy or link fragments depending on sizes;
• append_fragments() transfers fragments without copying;
• shared buffers do not use copy-on-write.

That last rule is the failure boundary. Mutating a shared fragment changes what every sharing iobuf sees.

The right claim is not “Redpanda is zero-copy.” It is “the buffer type exposes copy-avoiding operations where ownership permits them.”

The wider design rule is simple: remove a copy only after naming the resulting ownership and lifetime contract.

Ordered dispatch and correlation solve different problems

The internal rpc::transport keeps:

• an ordered request queue keyed by a local sequence;
• a correlation map from wire IDs to response handlers;
• a memory semaphore;
• timeout and dispatch timestamps.

The request queue preserves send order. Correlation IDs identify the response handler. Those are separate responsibilities: a correlation ID does not give permission to reorder stateful method calls.

The transport also distinguishes:

• request enqueue time;
• memory-reservation time;
• dispatch time;
• buffered-stream write completion;
• whether that buffered write flushed.

Those timestamps make timeout logs more useful. “RPC timed out” can mean the request waited for memory before it ever reached the socket.

Backpressure starts before serialization consumes the broker

The RPC transport uses a semaphore, a bounded permit counter, to limit memory held by in-flight requests. Callers can also attach resource units that remain held until the send buffer is released.

This prevents one producer of internal RPCs from creating an unbounded queue of serialized messages. It does not guarantee fairness by itself: a large request can wait behind other reservations, and a slow peer can keep response state alive until timeout.

Backpressure therefore needs three signals:

Increasing a timeout treats the symptom when the real problem is a saturated memory budget or unavailable peer.

Rolling upgrades depend on envelope discipline

During a rolling upgrade, old and new brokers can exchange current internal messages only when their transport and envelope compatibility ranges overlap.

Serde's size boundary helps an older reader skip appended fields. The compatibility version prevents a newer writer from claiming safety when an older reader cannot interpret the message.

Feature activation belongs to the same story. Redpanda can roll new binaries before activating a feature that depends on a new internal message format. Cluster feature activation supplies that gate.

Five byte-path failures to test

The wire contracts above leave five failures to test.

A developer appends a field but forgets its semantic default

Wire parsing succeeds, yet the older reader behaves incorrectly because the missing field's default means something different. Compatibility is semantic, not only syntactic.

Shared fragments are mutated

iobuf::share() has no copy-on-write protection. Freeze the ownership rule or make a real copy before mutation.

Compression moves the bottleneck to CPU

Zstandard can reduce bytes while increasing shard CPU and tail latency. Keep an uncompressed control run and measure payload-size distributions.

A timeout begins while the request is still local

The request can wait for memory or its turn in the ordered queue before the socket sees it. Use transport timing fields and memory-pressure metrics before blaming the network.

An incompatible message reaches a mixed-version cluster

Serde rejects versions outside the declared compatibility range. Roll all required binaries before activating the feature that emits the new envelope.

Trace the internal wire contract

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "envelope_header_size" -- src/v/serde/envelope.h
git grep -n "compat_version >" -- src/v/serde/read_header.h
git grep -n "size_of_rpc_header" -- src/v/rpc/types.h
git grep -n "min_compression_bytes" -- src/v/rpc
git grep -n "copy-on-write" -- src/v/bytes/iobuf.h

To measure rather than inspect, Redpanda includes rpc_bench.cc. Record the exact Bazel target, release build flags, payload type, size, compression setting, concurrency, core count, and network topology. A serializer-only result does not include queueing or transport cost.

The byte path is an internal compatibility contract

• Kafka wire compatibility and Redpanda internal RPC are separate protocols.
• Serde envelopes carry current version, compatibility floor, and payload length.
• RPC v2 uses Serde behind a fixed 26-byte header.
• Zstandard is optional and thresholded.
• iobuf can share fragments, but some appends copy and shared bytes are mutable.
• Memory reservations, ordered dispatch, correlation, and timeouts each own a different failure boundary.

Previous: Redpanda Data Transforms: Boundaries of In-Broker Wasm ←

Sources

• serde::envelope at v26.1.13
• Serde envelope read path
• Serde compatibility checks
• RPC types and header
• RPC header CRC-32C implementation
• RPC payload xxHash implementation
• RPC transport
• iobuf ownership contract
• xxHash upstream project