Kafka Wire Protocol: How Compatibility Survives Upgrades

Your rolling upgrade—replacing brokers one at a time—looks clean until an older Kafka client reconnects. TCP, the ordered byte-stream transport beneath Kafka, succeeds; the client's first request doesn't. The network is up, yet the two processes disagree about what the next bytes mean.

A broker is the Kafka server process accepting that request. The wire protocol is the binary contract that tells the client and broker how to frame, identify, and decode it. When that contract works, mixed-version deployments look uneventful. When it doesn't, the failure appears before your record schema or application handler gets a vote.

ApiVersions is Kafka's request for discovering the API-version ranges a broker supports. An API key is the numeric operation identifier, while an API version selects one schema revision for that operation. Protocol compatibility is an operational feature, not implementation trivia. Kafka maintains it across three separate boundaries:

Conflating these boundaries produces bad diagnoses. ApiVersions can resolve an RPC schema difference. It cannot repair a malformed frame or an invalid record-format configuration.

A Kafka message starts with a boundary, not an API key

TCP provides an ordered byte stream. It does not preserve the boundaries between individual send calls, as RFC 9293's description of TCP's byte-stream service makes clear. One Kafka request may span several TCP segments, while one segment may contain bytes from several requests.

Kafka therefore wraps every request and response in a frame, one complete protocol message. The Apache protocol reference defines its first field as a big-endian INT32 containing the number of bytes that follow.

① The receiver reads exactly 4 bytes, interprets a signed INT32, and rejects an invalid negative or oversized length. Kafka's release-pinned NetworkReceive implementation performs that check before reading the payload.

② The header supplies the context needed to choose a parser.

③ The selected API schema determines the body. A length alone says nothing about whether those bytes represent Produce, Fetch, or ApiVersions.

This distinction matters in packet captures. TCP packet boundaries are transport artifacts; the length prefix defines the Kafka boundary. Start with those 4 bytes, reconstruct exactly N more, and only then decode the header.

The header makes compatibility local to each API

A request header carries parsing context from client to broker. A response header carries enough context for the client to match the reply to its pending request. Kafka versions those headers separately from request bodies.

The release-pinned RequestHeader.json and ResponseHeader.json define four current header layouts:

Kafka 4.x no longer supports the historical request header v0. An API key is a signed 16-bit operation identifier: Produce is 0, Fetch is 1, and ApiVersions is 18. The adjacent api_version is another INT16. Together they select one request-body schema. Kafka's ApiKeys enum maps that body version to the correct request and response header versions.

The correlation ID is a client-chosen INT32 that the broker copies into the response header. Responses don't repeat the API key or API version; the client uses the correlation ID to recover that context from its in-flight request.

A client ID is the caller-supplied label in request headers v1 and v2. It uses the regular NULLABLE_STRING encoding even in header v2: a 2-byte signed length, UTF-8 bytes, or -1 for null. Don't confuse it with client_software_name and client_software_version, fields that KIP-511 added to the ApiVersions v3 body.

The body schemas come from Apache's generated message definitions. For example, ApiVersionsRequest.json marks v0 through v2 as empty request bodies and adds the software fields in v3. Its paired ApiVersionsResponse.json describes which response fields exist at each version.

Kafka doesn't have one global wire-protocol version. It versions each API independently. One connection can use different versions for Produce, Fetch, and Metadata. A broker release number such as 4.3.1 is therefore not a body schema identifier.

VARINT and UNSIGNED_VARINT do not encode the same number

Kafka's primitive-type specification starts with fixed-width big-endian values such as INT16, INT32, and INT64. Traditional STRING values use an INT16 byte length, while BYTES and arrays use INT32 lengths. These types are simple to parse, but a small value still pays the full fixed width.

A variable-length integer stores 7 payload bits in each byte and uses the high bit to indicate whether another byte follows. Kafka uses two variants that are easy to mistake for each other:

• UNSIGNED_VARINT encodes a non-negative value directly with the base-128 byte sequence.
• Signed VARINT first applies zig-zag encoding, which maps signed integers to unsigned integers, then writes the same base-128 sequence.

For a signed 32-bit integer n, the zig-zag mapping is (n << 1) ^ (n >> 31). Apache's pinned ByteUtils implementation contains both operations.

The interactive encoder below demonstrates the unsigned base-128 stage. Treat its input as an UNSIGNED_VARINT, or as a value that has already passed through zig-zag encoding. It does not perform the signed transformation itself.

Enter 300 and the component returns AC 02. That is the correct UNSIGNED_VARINT representation of 300; Kafka's signed VARINT representation is D8 04. This distinction matters when reading record lengths and deltas by hand.

Variable width doesn't guarantee a smaller value. A full-width 32-bit unsigned value may consume 5 bytes, one more than INT32. The encoding earns its place when the distribution favors small values.

Compact types build on UNSIGNED_VARINT. A COMPACT_STRING encodes its UTF-8 byte length plus one, reserving zero for null in nullable forms. The string "cat" is 00 03 63 61 74 as a regular STRING, but 04 63 61 74 as a COMPACT_STRING. A compact null is 00; an empty compact string is 01.

Tagged fields let compatible parsers ignore new information

A flexible version is an API schema that uses compact types and ends each structure with a tagged-field buffer. KIP-482 introduced flexible versions so Kafka could add optional data without assigning a new API version for every field.

A tagged field has a numeric tag, a byte length, and an opaque payload. Because the parser knows the payload length before interpreting it, it can skip an unknown tag and resume at the next field.

① Zero tagged fields encode as 00, so an empty buffer costs 1 byte.

② Tag IDs must appear in increasing order, without duplicates. The size tells an older flexible-version parser how far to advance.

③ A parser that recognizes the tag decodes the payload. A parser that doesn't recognize it skips exactly that many bytes.

The compatibility claim has a boundary: a pre-flexible client cannot parse compact lengths or tagged buffers. It must negotiate an older API version and receive the older schema. Tagged fields protect clients that already understand the enclosing flexible version.

They also don't make every schema change safe. Changing the meaning of a required field, moving a regular field, or changing its type still needs a new API version. Tagged fields solve optional extension, not arbitrary mutation.

ApiVersions advertises ranges; the client chooses locally

ApiVersions negotiation begins with API key 18. KIP-35 defined the request so clients could discover a broker's supported minimum and maximum version for each API instead of inferring capabilities from a release string.

The exchange is less symmetrical than the word “negotiation” suggests. The client doesn't send its complete capability list. It sends an ApiVersions request, the broker advertises its ranges, and the client intersects those ranges with the versions it implements and permits.

Suppose a client implements versions 3 through 11 of one API, the connected broker advertises 0 through 9, and client policy caps that API at 8. The usable intersection is 3 through 8; the client can select 8. If the ranges don't overlap for an API the client requires, failing is safer than guessing at a body layout.

Clients track this information per broker connection. Apache's NodeApiVersions class stores the usable ranges for a node. During a rolling upgrade, two brokers may therefore lead the same client to choose different body versions. That is expected, not evidence of inconsistent client behavior.

ApiVersions must also bootstrap its own version. KIP-511 added a flexible v3 request while retaining response header v0 for this API, allowing a client to decode an UNSUPPORTED_VERSION response and retry with an older request version. This is also why code that derives the response header solely from “flexible body or not” will misparse ApiVersions.

The handshake has limits. It advertises RPC schemas, not whether a cluster setting enables a feature, whether every broker has finished an upgrade, or which record magic values remain acceptable. For an upgrade canary, compare the advertised ranges from both old and new brokers against the oldest client you still support. That tests the contract ApiVersions can actually guarantee.

Record-format evolution is a separate compatibility axis

Produce and Fetch bodies contain records with their own message format. Its magic byte is independent of the API version in the request header.

KIP-98 introduced the v2 record-batch format. A record batch groups records under shared metadata, including producer ID, producer epoch, and base sequence. Its CRC-32C covers the bytes from attributes through the end of the batch rather than the leading offset and length fields.

Records inside that batch use signed VARINT and VARLONG encodings for lengths and deltas. Signed lengths matter because -1 represents a null key or value. Apache's DefaultRecord source shows the exact read and write order.

KIP-724 removed broker support for message formats v0 and v1 in Kafka 4.0. That change didn't renumber Produce or Fetch around the magic byte. RPC API versions and record formats remain separate inventories, even when their numbers happen to match.

What could go wrong: negotiation cannot start a broken broker

KAFKA-6238 documents a failed rolling upgrade from Kafka 0.10.0.1 to 1.0.0. The published procedure instructed operators to keep inter.broker.protocol.version on the old level while replacing binaries, but it didn't also tell them to pin log.message.format.version.

The new binary selected its newer message-format default while the inter-broker protocol remained old. Kafka's configuration validation rejected that combination, and the broker failed before joining the cluster. The release-pinned Kafka 1.0.0 configuration source contains the relationship the upgrade guide had missed.

ApiVersions could not help. No client handshake occurs when the broker cannot start.

Those exact historical settings aren't a template for a current upgrade. The operational lesson survives: inventory every independently versioned boundary for the precise source-to-target path, test the intermediate mixed state, and establish the rollback point before advancing any compatibility floor. Testing only the final release misses the state in which rolling upgrades spend most of their time.

A 23-byte request makes the contract concrete

You can inspect the contract without a Kafka client library. The following exercise requires a disposable local broker listening on 127.0.0.1:9092 with a PLAINTEXT listener. It sends one ApiVersions v0 request and parses the corresponding v0 response.

ApiVersions v0 is useful here because its body is empty. The request still demonstrates the frame length, request header v1, API key, API version, correlation ID, and client_id. The response demonstrates response header v0 and the broker's advertised ranges.

Save this as api_versions_probe.py.

import socket
import struct
 
 
def read_exact(sock: socket.socket, size: int) -> bytes:
    chunks = []
 
    while size:
        chunk = sock.recv(size)
        if not chunk:
            raise EOFError("connection closed before the frame completed")
        chunks.append(chunk)
        size -= len(chunk)
 
    return b"".join(chunks)
 
 
api_key = 18
api_version = 0
correlation_id = 0x01020304
client_id = b"wire-read"
 
header = struct.pack(">hhi", api_key, api_version, correlation_id)
header += struct.pack(">h", len(client_id)) + client_id
request = struct.pack(">i", len(header)) + header  # ①
 
print("request:", request.hex(" "))
 
with socket.create_connection(("127.0.0.1", 9092), timeout=5) as sock:
    sock.sendall(request)
 
    response_size = struct.unpack(">i", read_exact(sock, 4))[0]
    response = read_exact(sock, response_size)  # ②
 
response_correlation, error_code, api_count = struct.unpack_from(
    ">ihi", response, 0
)  # ③
 
if response_correlation != correlation_id:
    raise ValueError("response correlation ID does not match the request")
 
print("error_code:", error_code)
print("advertised APIs:", api_count)
 
offset = 10
for _ in range(api_count):
    key, minimum, maximum = struct.unpack_from(">hhh", response, offset)
    print(f"key={key:>3} versions={minimum}..{maximum}")
    offset += 6

① header is 19 bytes, so the prefix is 00 00 00 13. The total request occupies 23 bytes because the prefix doesn't count itself.

② recv() may return only part of a TCP frame. read_exact keeps reading until it has the byte count declared by the broker.

③ An ApiVersions v0 response begins with the response-header correlation ID, followed by the body's INT16 error code and INT32 array length. Each advertised entry contains three INT16 values: API key, minimum version, and maximum version.

The request line should start with 00 00 00 13 00 12 00 00 01 02 03 04 00 09. Here, 00 12 is API key 18, 00 00 is version v0, and 00 09 is the byte length of "wire-read".

To capture the same exchange, start tcpdump in one terminal, run the script in another, then ask tshark to decode the reassembled stream:

# macOS; use "lo" instead of "lo0" on Linux.
sudo tcpdump -i lo0 -s 0 -w kafka-api-versions.pcap 'tcp port 9092'
 
# Run in a second terminal, then stop tcpdump with Ctrl-C.
python3 api_versions_probe.py
 
tshark -r kafka-api-versions.pcap \
  -d tcp.port==9092,kafka \
  -Y kafka \
  -V

Read the capture beside four pinned Apache sources: NetworkReceive.java for framing, RequestHeader.json for the request header, and the ApiVersions request and response definitions for the bodies. Every byte in the script should map to one of those contracts.

Compatibility deserves the same owner as schemas

My opinion: teams should give protocol compatibility the same operational ownership they give event schemas. Reviewing Avro or Protobuf evolution while allowing client-version floors, broker API ranges, and upgrade sequencing to drift means owning only half the contract. Record the supported client matrix, test the intermediate broker state, and attach rollback criteria to the upgrade plan.

A schema registry cannot protect a payload that neither side reaches because the request envelope was unreadable.

Kafka survives upgrades not by freezing its bytes, but by making change explicit. Length prefixes preserve boundaries. API keys and versions select parsers. Tagged fields make optional additions skippable. ApiVersions finds an overlap. Record magic keeps the contained data format separate.

When an upgrade fails, identify which boundary moved before changing anything else.

Next, fetch sessions show how retaining protocol state can trade repeated wire metadata for a new operational lifecycle.

Sources

Apache protocol and source

• Kafka protocol reference
• Kafka 4.3.1 NetworkReceive.java
• Kafka 4.3.1 RequestHeader.json
• Kafka 4.3.1 ResponseHeader.json
• Kafka 4.3.1 ApiVersionsRequest.json
• Kafka 4.3.1 ApiVersionsResponse.json
• Kafka 4.3.1 ApiKeys.java
• Kafka 4.3.1 ByteUtils.java
• Kafka 4.3.1 DefaultRecord.java

Compatibility changes and failure record

• KIP-35: Retrieving protocol version
• KIP-98: Exactly Once Delivery and Transactional Messaging
• KIP-482: Optional Tagged Fields
• KIP-511: Collect and Expose Client Name and Version
• KIP-724: Drop message formats v0 and v1
• KAFKA-6238: Kafka 1.0.0 rolling-upgrade documentation failure