Redpanda Storage: Logs, Offsets, Recovery, Compaction

A broker, the server process that hosts a local partition replica, restarts. A consumer, a Kafka client that reads records, asks it for Kafka offset 8142, a client-visible record position. The local file does not contain only Kafka records: it can also contain a Raft configuration batch, an internal batch that records a replica-membership change, and other internal state. Compaction, background rewriting that removes older keyed versions, may have rewritten closed files, and recovery, startup validation and reconstruction after an interrupted write, may have truncated an incomplete tail.

The storage engine has to answer a deceptively small question: which bytes still represent offset 8142?

Redpanda's local log is more than an append-only file. It is built from segments, bounded portions of one partition log, with sparse indexes, Kafka offset translation, background compaction, and a crash-recovery path that trusts only complete batches with valid checksums.

This post stays below Raft consensus and above Tiered Storage. Replication decides which history is authoritative. This layer decides how one local replica stores, finds, rewrites, and recovers that history. Source links are pinned to Redpanda v26.1.13.

The log API keeps mutation narrow

Redpanda's abstract storage::log exposes a small set of operations:

• create an appender;
• create an offset-bounded reader;
• flush;
• truncate a suffix;
• truncate a prefix;
• apply retention and compaction housekeeping;
• translate between internal and Kafka-visible offsets;
• query by timestamp.

That API matters because storage has several writers in the broader system: normal Raft replication, configuration changes, snapshots, retention, and compaction. They do not mutate arbitrary bytes in place. New batches append to the active segment; maintenance replaces or removes closed ranges under storage-layer coordination.

Segment boundaries make retention and compaction practical: Redpanda can operate on closed files without stopping appends to the active segment.

The pinned segment_index stores a sparse index, offset and timestamp entries recorded at intervals rather than for every batch. Its default indexing step is 32 KiB. A lookup therefore finds a nearby file position, then the reader parses forward to the requested batch. The index narrows the scan; it does not promise one disk read or constant latency.

Kafka offset
  → translate to internal log offset
  → choose segment
  → find nearest sparse index entry
  → parse complete batches forward
  → return matching Kafka records

The first arrow removes internal control batches from the client's offset space. Segment selection narrows the candidate file by base offset. The sparse index finds a nearby byte position rather than an exact record, so the parser must still validate complete batches until it reaches the requested offset. The block is a lookup path, not a claim that one syscall or one disk read serves the fetch.

Append and flush are separate contracts

Raft replication and durable media are separate boundaries. The storage source makes the local boundary explicit.

Redpanda's segment_appender says an append can be logically complete while its bytes remain partly unflushed and not fsynced. A logical append updates the appender's in-memory position and queues direct I/O, file operations that bypass the page cache. A completed dma_write means the direct write finished at the file API boundary; it is still not an fsync guarantee. A later flush() future establishes that prior appends have been flushed and fsynced.

That contract prevents a common wording error:

“Written to the log” is not automatically “durable on the device.”

The appender uses Seastar's DMA-oriented file APIs, and the pinned sizing documentation states that Redpanda bypasses the Linux page cache and manages its own memory and disk I/O. That gives the broker control over queue depth and memory use, but it also makes Redpanda's I/O configuration part of the result.

The useful boundaries are:

Raft policy connects these rows; storage defines why they remain different rows.

Kafka offsets hide internal Raft batches

A Kafka offset counts records visible through the Kafka API. An internal log offset also advances for some Redpanda control batches. If clients saw both spaces directly, an internal metadata write could create a surprising gap or change the next fetch position.

Redpanda's offset_translator tracks the difference. It checkpoints translation state to the local key-value store and truncates that state with the log.

The list of filtered batches is source-defined in record_batch_types.h. In v26.1.13, it includes Raft configuration, archival metadata, version fences, prefix truncation, partition-property updates, and several newer internal state-machine records.

Those batches remain in the internal log, but they do not increment the offset returned for Kafka fetches.

Consider this simplified sequence:

Internal log offset 100: Kafka data batch, two records
Internal log offset 102: Raft configuration batch
Internal log offset 103: Kafka data batch, one record
 
Kafka-visible offsets: 100, 101, 102

The example is illustrative; actual batch offsets and record counts depend on the written batches. The invariant is source-backed: filtered internal batch types do not consume Kafka-visible positions.

This translation is not optional glue. It is what lets Redpanda put Raft and application state in one log without changing the client's offset contract.

Move the cursor below in either offset space. Then add control batches and compaction separately: one changes the translation delta, while the other changes which assigned offsets still return records.

That separation matters during incident review. A filtered internal batch explains an offset-space delta; compaction explains a fetch hole without renumbering the log.

Recovery rebuilds trust from complete, checksummed batches

After an unclean shutdown, the final file may end in a partial batch. An index may also be absent or stale relative to the segment.

The pinned log_replayer does not trust the index during recovery. It reads the segment from the beginning, reconstructs index entries, extends a CRC-32C checksum over each batch, and records the file position after the last valid batch.

If parsing or checksum validation fails, recovery keeps the last valid checkpoint. The surrounding storage recovery code can truncate the invalid tail rather than inventing bytes or exposing a partial record batch.

That gives recovery a clear rule:

valid header + complete payload + valid CRC
  → candidate batch
 
partial bytes or CRC mismatch
  → stop at the previous valid checkpoint

CRC validation detects accidental corruption in the covered bytes. It does not repair a corrupt batch, prove the storage device is healthy, or replace replication. Raft recovery must fetch authoritative data from another replica when the local copy is missing or divergent.

Compaction is best effort, not event deduplication

Log compaction retains a later record for a key while making older versions eligible for removal. Redpanda runs it against closed segments in the background.

The pinned compaction guide states two limits that application designs often miss:

1. compaction does not guarantee perfect deduplication;
2. a key can still appear more than once across the topic while subsets of a partition await or undergo compaction.

The cleanup policy chooses the maintenance work:

For a local-storage compacted topic, a tombstone is a keyed record with a null value. It asks compaction to remove older values for that key after the configured tombstone-retention boundary. Consumers that need a complete reconstructed map must finish before tombstones become eligible for removal. Tiered Storage adds a different tombstone boundary, covered in the next episode.

Compaction is therefore a storage policy, not an idempotency mechanism. A payment event written twice under the same key can still be read twice before compaction, and compaction timing is not part of a processing transaction.

Retention deletes ranges; prefix truncation changes the readable start

Retention and compaction solve different problems. Retention chooses how much history remains. Compaction chooses which keyed versions remain within that history.

Redpanda applies retention at segment granularity where possible. The log::truncate_prefix contract also allows the reported start offset to land inside a batch while keeping the complete batch on disk. That preserves batch integrity even when the logical read boundary advances.

For local-only topics, removing the last local copy removes the data from that replica's readable history. For Tiered Storage topics, local deletion is gated by remote-upload progress, which is a separate storage boundary.

Time queries use an index and still verify the log

Kafka's ListOffsets API can ask for the first record at or after a timestamp. Redpanda's log API exposes timequery(), and the segment index tracks timestamp entries beside offsets.

The index is a starting hint, not the returned truth. Batch timestamps can be non-monotonic, and older segments may lack newer broker-time metadata. The segment_index contains explicit fallback handling for those cases.

Redpanda 26.1.13 also has a cluster-history boundary. The tagged validated_batch_timestamps gate is automatic for new clusters from the 25.3 feature line, while the pinned retention guide explains that older upgraded clusters can continue using broker timestamps until the feature is activated. Before using event time as a retention or archival boundary, record the feature state and effective message.timestamp.before.max.ms and message.timestamp.after.max.ms limits.

Failure modes the log cannot repair

The storage contracts above leave five failure modes the log cannot repair.

The active segment ends with a partial write

Recovery stops at the last complete batch with a valid checksum. Expect startup work and replica recovery rather than assuming the final file is immediately readable.

Compaction competes with foreground I/O

Compaction reads old segments and writes replacements. Redpanda's compaction controller schedules that work, but the disk remains shared. The tagged segment_utils.cc also treats a missing, incomplete, or invalid compaction index after a crash as state that needs rebuilding. Watch compaction backlog, rebuilds, foreground latency, and I/O saturation together before increasing compaction concurrency.

A timestamp moves retention unexpectedly

Producer-controlled timestamps can be far in the future or past. Use the documented timestamp validation properties and inspect effective topic configuration before relying on time-based retention.

An offset gap is internal, not lost user data

Raft configuration and other filtered batches advance internal offsets without advancing Kafka offsets. Diagnose in the correct offset space before calling a gap corruption.

The disk fills before housekeeping can reclaim space

Retention works on eligible closed ranges. A hot active segment, slow compaction, a stalled Tiered Storage upload, or a recovery backlog can delay reclamation. The pinned disk-utilization guide states that once a broker crosses storage_min_free_bytes, external writes are rejected cluster-wide while internal replication and rebalancing can continue.

Trace recovery and offset translation in source

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "virtual ss::future<> flush" -- src/v/storage/log.h
git grep -n "hasn't been fsynced" -- src/v/storage/segment_appender.h
git grep -n "default_data_buffer_step" -- src/v/storage/segment_index.h
git grep -n "offset_translator_batch_types" -- \
  src/v/model/record_batch_types.h
git grep -n "checksumming_consumer" -- src/v/storage/log_replayer.cc

For an operational trace, pair those source results with rpk topic describe <topic> -p, disk latency, compaction metrics, and the broker logs from one controlled restart. Source explains the mechanism; the trace shows which boundary moved on your hardware.

The storage engine preserves two views at once

• The internal log stores data and selected state-machine batches.
• Offset translation keeps internal batches out of Kafka-visible positions.
• Sparse indexes reduce scans but do not eliminate parsing.
• Append and fsync are separate operations.
• Recovery trusts complete batches with valid CRCs and rebuilds index state.
• Compaction reduces storage on a best-effort schedule; it does not deduplicate application events.

Previous: Redpanda Raft: From Acknowledgement to Durable Media ←

Sources

• storage::log at v26.1.13
• segment_appender durability contract
• segment_index at v26.1.13
• offset_translator at v26.1.13
• log_replayer at v26.1.13
• Feature gate for validated batch timestamps
• Pinned compaction settings
• Pinned sizing and I/O model
• Pinned disk-throttling behavior
• Live Redpanda documentation — secondary, moving reference