Debugging Kafka Batch Offsets Across Appends

The client connects. The server writes bytes. The consumer still does not return the records I expect. That was the useful part of this debugging session: transport success did not establish protocol correctness.

The short answer: a Kafka-compatible append path must agree on the broker-assigned batch base offset, the records' offset deltas, and the next append position. Counting one offset per append call and storing every producer header unchanged breaks that agreement across batches.

This is a personal account from Streamline, an experimental Kafka-compatible server—not an Apache Kafka production incident or a claim of production-ready compatibility. The console excerpts are abridged. They do not preserve enough client settings and wire evidence to reproduce the exact 11-to-one observation on their own. The examples below establish the offset invariants; they do not prove that an empty second fetch erased records already read from the first.

Why implement a protocol yourself?

Before diving into the bug, let me address the obvious question: why build a Kafka-compatible server when Kafka already exists?

Three reasons:

1. You don't truly understand a protocol until you implement it. I've used Kafka for years—configuring producers, tuning consumers, debugging lag issues. But I never really understood why it worked the way it did until I had to make something speak its language.

2. Edge cases become visceral. Reading that "the broker must update baseOffset" is one thing. Watching a test report 10 missing messages after you forgot to do it? That sticks with you.

3. It changes how you use the real thing. After this experience, I now know exactly what's happening when a Kafka consumer "loses" its place, or why certain configurations matter. The mental model is no longer abstract.

If you want to truly understand a distributed system, try implementing its protocol. You'll learn more from your first bug than from a dozen architecture diagrams.

Now, let's see what happens when you miss a critical detail.

I was working on Streamline, a Kafka-compatible streaming server I've been building. The goal is straightforward: existing Kafka clients should work unchanged against Streamline. We're not trying to be a full Kafka replacement—think of it as "Redis for streaming"—but we do need to speak the Kafka wire protocol correctly.

Everything seemed to be working. Unit tests passed. The server started. Clients connected. Life was good.

Then I ran a simple end-to-end test:

# Producer sends 11 messages
$ echo -e "msg1\nmsg2\nmsg3\nmsg4\nmsg5\nmsg6\nmsg7\nmsg8\nmsg9\nmsg10\nmsg11" | \
    kafka-console-producer --bootstrap-server localhost:9092 --topic test-topic
>>>>>>>>>>>
 
# Consumer reads... 1 message?!
$ kafka-console-consumer --bootstrap-server localhost:9092 \
    --topic test-topic --from-beginning
msg1
Processed a total of 1 messages

The reported observation was 11 inputs and one visible output. Do not use the console's prompt characters as acknowledgement evidence: a reproduction needs the producer's result, errors, client configuration, and actual batch boundaries. The reference console producer's send path handles synchronous results or an error callback separately from reading input.

Why this bug was particularly evil

Here's what made this frustrating:

The immediate symptom was missing consumer-visible records despite stored bytes. That is not yet proof that bytes were deleted. Distinguish physical loss from incorrect offset assignment, visibility, filtering, and termination before deciding which layer failed.

Kafka's record batch format: the background you need

Before I explain what went wrong, you need to understand how Kafka structures messages on the wire. This isn't academic—it's the key to everything.

Kafka doesn't send individual messages

Kafka writes records in record batches, containers with their own headers. A batch may contain one record or several. Eleven lines of console input do not guarantee one batch: partitioning, buffering, batch.size, and linger.ms affect batching, as the producer configuration documents.

The examples below deliberately assume one ordinary, contiguous 11-record batch, followed by a three-record batch. That makes the arithmetic inspectable without pretending it captures every detail of the reported client run.

The byte layout (this is where it gets good)

The magic-v2 header layout is documented in the Kafka 4.0 message format and its pinned DefaultRecordBatch implementation. This is a reference for the field arithmetic, not a record of the client version used in the original session.

The header consumes 61 bytes before record data begins. The field list and offsets come from the Kafka protocol specification; the visualization keeps that layout available without forcing the reader to parse a monospaced table.

The critical field: baseOffset

The broker assigns offsets in the partition's log. The stored batch header must reflect that assignment. The reference implementation's setLastOffset writes the base-offset field as assignedLastOffset - lastOffsetDelta. For the ordinary producer batches in this example, retaining an incoming base of zero is correct only when the assigned base is actually zero.

How consumers calculate record offsets

Each record within a batch has an offsetDelta field (a varint, not fixed-size). The actual offset of each record is:

record_offset = baseOffset + offsetDelta

So for a batch with 11 records:

• Record 0: baseOffset + 0
• Record 1: baseOffset + 1
• ...
• Record 10: baseOffset + 10

The batch header also contains lastOffsetDelta (at byte offset 23), which for 11 records would be 10. This lets consumers calculate the next expected offset:

next_expected_offset = baseOffset + lastOffsetDelta + 1

Do not substitute recordsCount - 1 for lastOffsetDelta universally. Compaction preserves the original first and last offset/sequence numbers even when records between them are removed. The calculator below models contiguous batches; it is not a parser for compacted logs.

Tracing the bug

Armed with this knowledge, let's trace what was happening.

Step 1: Verify data is arriving

First, I needed to confirm the producer was actually sending 11 messages:

debug!(
    topic = %topic_name,
    partition = partition_index,
    records_len = batch_bytes.len(),
    "Processing produce request"
);

This log records byte length, not record count or batch boundaries. A decoder must walk the record set and validate each batch before making that claim.

Step 2: Check storage

Looking at our storage code:

// Original produce handler
let value = Bytes::copy_from_slice(records);  // Raw batch bytes
match self.topic_manager.append(&topic_name, partition_index, None, value) {
    Ok(offset) => {
        (offset, NONE)  // Returns offset 0
    }
    // ...
}

And in Partition::append():

pub fn append(&mut self, key: Option<Bytes>, value: Bytes) -> Result<i64> {
    // Get next offset - always increments by 1
    let offset = self.next_offset.fetch_add(1, Ordering::SeqCst);
    // ... store record at 'offset' ...
    Ok(offset)
}

First problem found: We increment offset by 1 regardless of how many records are in the batch.

Step 3: Understand the consumer's perspective

Under the deliberately simplified first-batch assumptions, the sequence is:

1. Consumer connects, requests metadata for test-topic
2. Consumer sends FetchRequest with fetch_offset = 0
3. Server returns batch containing records 0-10
4. Consumer reads batch, sees baseOffset = 0, lastOffsetDelta = 10
5. Consumer calculates: "I received offsets 0-10, next I need offset 11"
6. Consumer sends FetchRequest with fetch_offset = 11
7. Server looks for data at offset >= 11
8. Server's internal offset counter is at 1 (we only had one append() call)
9. Server finds nothing at offset 11
10. Server returns empty FetchResponse
11. An empty response provides no new records; it does not retract the first batch

This sequence alone does not explain ten missing records. If the consumer decoded records 0–10 in step four, it already received them. Nor does an empty fetch by itself mean a normal polling consumer exits. The useful finding is the mismatch between the broker's next append position of one and the consumer's next expected offset of 11. Inspect the next append, the fetch response and visibility bounds, and the client's stop conditions to establish the actual failure.

Those are three different incident questions. The transactions and visibility lab shows legitimate control/aborted offset gaps; the consumer progress lab shows why a known source gap must not become an invented unfinished handler. Neither model claims to reproduce the historical console symptom described here.

Step 4: But wait, there's more

Even if we fixed the offset counting, there's a second problem. Remember: the broker must update baseOffset.

Our code was storing the raw batch bytes without modification. The producer sent baseOffset = 0 in the batch header, and we stored those exact bytes. For the first batch, this happened to work because we also assigned offset 0. But for any subsequent batch? Disaster.

Correct behavior:

Batch 1 arrives: producer's baseOffset = 0, lastOffsetDelta = 10
Broker assigns:  baseOffset = 0, stores at offset 0
                 Updates header bytes [0..8] to contain 0 (big-endian)
                 Increments next_offset by 11

Batch 2 arrives: producer's baseOffset = 0, lastOffsetDelta = 2
Broker assigns:  baseOffset = 11, stores at offset 11
                 Updates header bytes [0..8] to contain 11 (big-endian)
                 Increments next_offset by 3

Our broken behavior:

Batch 1 arrives: producer's baseOffset = 0
Broker stores:   at internal offset 0, bytes unchanged
                 Increments next_offset by 1 (wrong!)

Batch 2 arrives: producer's baseOffset = 0
Broker stores:   at internal offset 1, bytes unchanged (baseOffset still 0!)
                 Increments next_offset by 1

The second batch now occupies the wrong internal range and carries stale offsets. A consumer asking for offset 11 cannot discover that batch where the broker should have placed it. If the broker returns it anyway, the embedded offsets also disagree with the client's requested position.

The fix

Once I understood the problem, the fix became clear. According to the Kafka Protocol Guide, the Produce API response includes a base_offset field—"The base offset" assigned to the batch by the broker. This tells us the broker is responsible for assigning and returning this offset.

The fix needs an append boundary that understands batches. Here is the algorithm for a validated new producer batch with contiguous deltas. It is pseudocode, not a complete broker implementation:

validate framing, magic, lengths, CRC, record count, and record deltas
under the partition's append serialization:
    base = current next offset
    span = checked_add(lastOffsetDelta, 1)
    next = checked_add(base, span)
    rewrite the batch baseOffset with base
    append the batch and maintain the corresponding index
    publish next only after the append has succeeded under the storage contract
return the assigned base for the Produce response

The key operations:

1. Consistent allocation: reserve the offset span for the validated batch, not one offset per append call. An atomic counter alone does not serialize the log write, update an index, or make an append durable.

2. Header modification: Bytes 0-7 get overwritten with the broker-assigned base_offset in big-endian format. This is the critical step we were missing.

3. Return the base_offset so the produce response can include it.

Updated produce handler

Decode every batch in the Produce record set. Reject malformed input rather than silently replacing an invalid count with one. After append, update the log end offset (LEO), the position immediately after the log's last offset, consistently with the stored batch and index. Acknowledgement, replication, visibility, and crash recovery still need their own contracts.

Updating only the base-offset bytes does not change the magic-v2 CRC: the CRC-covered region starts at attributes. That does not excuse skipping CRC validation or recomputation when changing fields inside the covered region.

Validation

The original account reported the following result after the offset fixes. Treat this as an abridged transcript, not a self-contained test script:

# Send 11 messages
$ echo -e "msg1\nmsg2\nmsg3\nmsg4\nmsg5\nmsg6\nmsg7\nmsg8\nmsg9\nmsg10\nmsg11" | \
    kafka-console-producer --bootstrap-server localhost:9092 --topic test-topic
 
# All 11 received!
$ kafka-console-consumer --bootstrap-server localhost:9092 \
    --topic test-topic --from-beginning
msg1
msg2
msg3
msg4
msg5
msg6
msg7
msg8
msg9
msg10
msg11
Processed a total of 11 messages

And with multiple batches:

# Send batch 1 (3 messages)
$ echo -e "batch1_msg1\nbatch1_msg2\nbatch1_msg3" | kafka-console-producer ...
 
# Send batch 2 (3 messages)
$ echo -e "batch2_msg1\nbatch2_msg2\nbatch2_msg3" | kafka-console-producer ...
 
# All 6 messages, in order
$ kafka-console-consumer --from-beginning ...
batch1_msg1
batch1_msg2
batch1_msg3
batch2_msg1
batch2_msg2
batch2_msg3
Processed a total of 6 messages

A consistent illustrative trace for the earlier 11-record batch followed by a three-record batch would be:

DEBUG Kafka batch appended topic=test-topic partition=0 base_offset=0 record_count=11
DEBUG Kafka batch appended topic=test-topic partition=0 base_offset=11 record_count=3

To exercise the field arithmetic independently, run this site's small offset example:

pnpm exec tsx scripts/examples/kafka-batch-offsets.ts

It asserts next offset 11 for the first batch, next offset 14 for the following three-record batch, and next offset 11 for a compacted header retaining one record from the original offset span. It constructs header fields only: there is no Kafka broker, valid serialized payload, CRC validation, or reproduction of the historical console result in that example.

For a real end-to-end test, retain the client versions and settings, use a fresh disposable topic, bound the consumer with an explicit message limit or timeout, and record decoded batch ranges, producer results, and visibility bounds. Those observations separate offset corruption from a misleading console transcript.

Lessons learned

1. Protocol specs are contracts, not suggestions

The stored batch offsets and the offsets assigned by the broker must agree. Clients depend on this behavior. Retaining a producer's placeholder base on a later append breaks that contract even when the byte write itself succeeds.

For your protocol implementations, this means: Read specs like legal documents. Every "must," "will," and "shall" is a requirement you need to implement.

2. Partial success masks total failure

The first batch worked because:

• Producer sent baseOffset = 0
• We happened to assign offset 0
• We happened to increment to 1 (wrong, but irrelevant for the first batch)

This made it look like things were working. The bug only manifested on the second batch, or when a consumer tried to resume from a previous position.

For your testing strategy, this means: Test multi-batch scenarios. Test resume. Test the second request, not just the first.

3. Your abstractions must match the protocol's model

Our storage layer had a simple model: each append() is one record, one offset. Kafka's model is different: each append is a batch, multiple offsets. This mismatch was the root cause.

We had two choices:

1. Change our storage abstraction to understand batches
2. Translate at the protocol layer

We chose option 1 (append_batch) because it's semantically correct—we really are storing a batch, not a single record.

For your architecture decisions, this means: When implementing protocol compatibility, your abstractions should either match the protocol's model or have an explicit, well-tested translation layer.

4. Trace data end-to-end

The bug wasn't in the network code. It wasn't in the storage code. It wasn't in the fetch code. It was in how they interacted:

Produce:  Store batch at offset 0 (should reserve 0-10)
Storage:  Increment offset by 1 (should be 11)
Fetch:    Return batch with baseOffset=0 (correct)
Consumer: Request offset 11 (correct per protocol)
Storage:  Next append starts at 1 instead of 11 (BUG!)

The empty fetch is not itself evidence that the first batch lost records. The next append makes the address-space mismatch observable.

For your debugging workflow, this means: When debugging protocol issues, trace a single request through every layer. Draw the sequence diagram. The bug is often at a layer boundary.

5. Byte layouts matter

The fix was literally changing 8 bytes at the start of the batch:

batch_bytes[0..8].copy_from_slice(&base_offset.to_be_bytes());

Eight bytes. Big-endian. At byte offset zero, after validating the batch. The allocation and append contracts still have to agree with those bytes.

But to know those 8 bytes needed to change, I had to understand:

• The Record Batch binary format
• That the broker is responsible for writing the baseOffset
• How consumers use baseOffset + offsetDelta
• The relationship between lastOffsetDelta and batch size

For your protocol work, this means: Protocol implementation requires byte-level understanding. Hexdump is your friend. The spec is your bible.

Debugging checklist for protocol implementations

For future me (and you), here's what I'll check next time:

Data flow analysis

• Trace a single request from client to storage and back
• Inspect synthetic or appropriately redacted bytes at each layer boundary
• Verify data isn't being transformed unexpectedly

Protocol compliance

• Re-read the protocol specification for the failing operation
• Check if you're handling all required fields
• Verify you're updating mutable fields the protocol requires

Offset/sequence handling

• Verify offset calculations match protocol expectations
• Check for off-by-one errors
• Ensure atomic operations where required

Multi-batch testing

• Test with multiple sequential batches
• Test consumer resume from various offsets
• Test producer sending, stopping, sending again

What could go wrong with an eight-byte fix?

• Malformed input: checking only that eight bytes exist does not validate a 61-byte header, record lengths, or the CRC.
• Compaction: retained record count can differ from the original offset span. Preserve the message-format contract rather than renumbering survivors.
• Append failure: advancing a counter and writing a log are separate operations. Recovery must reconcile the log, index, and published position.
• Visibility: an assigned offset is not the same thing as a committed or transactionally visible record.

These are reasons to validate the whole append/fetch boundary, not reasons to treat every incorrect counter as proof of deleted bytes.

The real takeaway: implementation as education

This bug took hours to find but minutes to fix. The ratio of investigation time to code changed is probably 80:1. That's normal for protocol implementation bugs.

The small header edit still required understanding:

• Kafka's 61-byte Record Batch header layout
• The broker's contract to update baseOffset
• How consumers use baseOffset + offsetDelta for offset calculation
• Atomic offset reservation for multi-record batches

Here's the thing: I've used Kafka in production for years. I've configured producers, tuned consumer groups, debugged rebalancing storms, and stared at lag metrics at 2 AM. But I never truly understood Kafka until I tried to implement its protocol.

Now when I see a consumer "lose its place" in production, I don't just restart it and hope for the best—I understand exactly what offset calculations are happening and where they might go wrong. When I configure fetch.max.bytes, I know how that interacts with batch sizes at the byte level.

If you want to level up your understanding of any distributed system:

1. Pick a protocol you use regularly (Kafka, Redis, PostgreSQL wire protocol)
2. Try implementing basic compatibility—even just enough to handle a single request type
3. Watch your mental model transform from "magic box" to "bytes on wire"

The bugs you hit will teach you more than any documentation ever could.

If you're implementing Kafka compatibility—or any complex protocol—I hope this helps you avoid the same pitfall. If a test reports partial success or missing records, start at the byte layout and the layer boundaries. Then collect enough wire and client evidence to distinguish an offset mismatch from actual data loss.

For the application-side version of this boundary problem, read Kafka Consumer Groups Are Platform Infrastructure. The Replay Readiness Checklist covers the questions to answer before attempting operational recovery; it is not authorization to replay.

Sources and References

Kafka Protocol Documentation

• Record Batch Format: Kafka 4.0 Message Format - Fields, CRC coverage, and compaction boundaries
• Wire Protocol: Kafka Protocol Specification - Complete wire protocol documentation including Produce/Fetch API definitions
• Message Format Overview: Kafka Documentation - High-level overview of message and batch structure
• KIP-98: Exactly Once Delivery and Transactional Messaging - The KIP that introduced the current record batch format

Reference Implementation

• Batch Header Layout: Kafka 4.0.0 DefaultRecordBatch.java - Pinned field offsets and offset-assignment methods
• Batching Settings: Kafka 4.0 producer configuration - Why console input lines do not determine batch boundaries

API Details

• Produce API (Key: 0): Returns base_offset field in response—the offset assigned by the broker
• Fetch API (Key: 1): Uses fetch_offset to request data from a specific position