Redpanda Tiered Storage: Local Writes, Remote Reads

A consumer, a Kafka client that reads records, rewinds to an offset that every broker—the server process that hosts partition replicas—removed from its local NVMe storage, directly attached solid-state media, days ago. Its Fetch request, the Kafka operation used to read records, is ordinary. The latency is not: the broker may need metadata from object storage, remote key-addressed blobs such as S3 objects, one or more byte ranges, and space in a local cache, bounded broker storage that retains fetched remote data for reuse, before it can answer.

Tiered Storage keeps recent log data on broker disks and copies older stable history to object storage. It extends the readable log beyond local retention; it does not make object storage behave like NVMe.

In Redpanda v26.1.13, Tiered Storage writes locally first and uploads stable history asynchronously. Remote retention, local retention, and cache capacity are separate controls. Cloud Topics use a different write path and must not be collapsed into the same architecture.

This post uses Redpanda v26.1.13 and the pinned Tiered Storage documentation. Both Tiered Storage and Cloud Topics are listed as Enterprise Edition features.

Redpanda now has three storage modes

Starting in Redpanda 26.1, the redpanda.storage.mode topic property names the intended path directly. A storage mode selects where a topic's durable data lives and how the broker reads it.

The definitions and precedence rules come from Redpanda's storage-mode table.

For new work, redpanda.storage.mode=tiered is clearer than combining the legacy redpanda.remote.read and redpanda.remote.write properties. The legacy properties still apply when the storage mode is unset; they have no effect when the mode is explicitly local, tiered, or cloud. Cluster-level remote-read and remote-write settings are creation-time defaults for unset topics; changing them later does not rewrite existing topics.

That distinction belongs in every runbook. A command that changes redpanda.remote.write may do nothing because the topic is already in an explicit storage mode.

Tiered Storage is an asynchronous copy, not the produce commit path

With Tiered Storage, the normal produce path still commits through the partition's local Raft group. A background archiver running with the partition leader selects uploadable log history and sends it to the configured object store.

Redpanda's documentation says remote write uploads only offsets below the last stable offset. That keeps open transactions, groups of Kafka writes that become visible or abort atomically, and otherwise unstable history out of the remote copy.

The tagged source divides the work between:

• the partition-level archival service in ntp_archiver_service.cc;
• object operations in cloud_storage/remote.h;
• partition and segment metadata in partition_manifest.h.

The sequence is:

producer acknowledgement
  → local Raft history becomes stable
  → archiver uploads segment data
  → manifest records remote range
  → local retention may remove the local copy

The upload does not block the original producer acknowledgement. This is why Tiered Storage is not, by itself, a synchronous remote backup of the newest records.

Local deletion waits for upload progress

Redpanda does not normally delete a Tiered Storage segment from local disk before the remote copy is available. The pinned remote-write guide states that local segments can exceed their configured retention while an upload remains pending.

That safety rule creates a useful failure mode:

If object storage slows down, the broker protects remote completeness by retaining more local data.

The consequence is disk pressure. Redpanda's upload-backlog controller raises archival priority when the unuploaded byte count grows, but it cannot create object-store bandwidth. A prolonged outage can fill local disks and cause produce throttling or rejection.

Pausing uploads through cloud_storage_enable_segment_uploads preserves this no-gap behavior: in-flight uploads finish, new uploads stop, and local data remains. Disabling remote write is not the safe pause control; the docs warn that toggling it can create gaps.

Remote and local retention answer different questions

Remote retention decides how long data remains in object storage. Local retention decides how much of that history stays on broker disks for low-latency reads.

For Tiered Storage topics:

• retention.ms and retention.bytes govern the total retained history, including object storage;
• retention.local.target.ms and retention.local.target.bytes govern the local target;
• cluster defaults apply when topic values are absent.

The pinned retention documentation adds two conditions:

• time and size policies apply together, so whichever removes more history can become the effective boundary;
• cleanup.policy=compact does not delete object-store segments through retention, while compact,delete does.

That last rule does not mean remote objects remain untouched. The same pinned guide says Redpanda can upload uncompacted segments and later upload compacted replacements. Compaction is still best effort, so compacted and uncompacted remote segments can coexist; compact only removes retention-based remote deletion from the policy.

Topic deletion has another independent control. With redpanda.remote.delete=true, deleting a Tiered Storage or Cloud Topic also deletes its object-store data; false preserves those objects. New topics default to true, but tagged source preserves legacy_remote_delete=false for upgraded legacy topics. Inspect the effective topic property before treating topic deletion as remote data deletion.

The two-tier model is:

Kafka retention window
┌────────────────────────────────────────────────────────┐
│ remote history                                         │
│ ┌──────────────────── local target ──────────────────┐ │
│ │ recent segments on broker disks                    │ │
│ └────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────┘

Local retention is a target rather than a promise that every partition keeps exactly that many bytes or hours. Disk pressure and skew affect which segments remain local.

Use the planner below to place one historical scan against those two retention windows. Its fixed ingest rate and read size keep the arithmetic inspectable.

A larger cache can absorb hydrated chunks, but it cannot make an expired remote range readable or turn an object-store fetch into a local-log read.

A cold fetch goes through manifests and a disk cache

A remote read starts with the Kafka offset. The broker translates that offset, finds the corresponding remote segment metadata, and serves the required range through a local object-storage cache.

Redpanda's read path includes:

• remote partition lookup in remote_partition.cc;
• archived and spillover manifests for old segment metadata;
• chunk-based hydration in segment_chunk_data_source.cc;
• cache trimming and eviction.

The current default cloud_storage_cache_chunk_size is 16 MiB. If a fetch needs a small range inside a much larger segment, Redpanda can cache only the necessary chunks instead of downloading the whole file.

Three eviction strategies are documented:

None guarantees a cache-hit percentage. Hit rate comes from the workload, cache budget, segment and chunk sizes, and the distance between repeated reads.

Object storage is a dependency with its own limits

Tiered Storage supports Amazon S3, Google Cloud Storage, and Microsoft Azure Blob or Data Lake Storage, according to the pinned setup guide. Redpanda uses persistent HTTP connections and a per-core connection pool shared by uploads and downloads.

The product docs list several boundaries:

• migrating a Tiered Storage topic between providers is unsupported;
• migrating between buckets or containers is unsupported;
• multi-region buckets or containers are unsupported;
• repeatedly disabling and re-enabling Tiered Storage can create inconsistent remote history;
• a credentials or endpoint failure can delay both uploads and cold reads.

The last point matters because remote read and write share object-store connection capacity. A recovery scan or cold-read burst can compete with the upload backlog if the pool, network path, or provider is saturated.

Tiered Storage is not Cloud Topics

Cloud Topics became available in Redpanda 26.1. They use object storage as the primary durable backing store, while local storage acts as a cache. The tagged implementation lives under src/v/cloud_topics and carries the Enterprise RCL header.

The pinned Cloud Topics documentation gives a different acknowledgement contract:

• Cloud Topics do not acknowledge data until the object-store upload completes, according to the vendor architecture description;
• Redpanda's vendor documentation says to expect 1–2 seconds end to end on public object stores;
• a Cloud Topic cannot be converted to or from a local or Tiered Storage topic, and Shadow Links do not support Cloud Topics, per the pinned limitations.

That architecture targets throughput-oriented, latency-tolerant workloads. It is not a faster form of Tiered Storage.

The distinction also changes failure analysis:

Redpanda v26.1.13 itself is a reminder to keep the paths separate. Its release notes include Cloud Topics fixes for compaction and retention behavior, including #31111. A fix in the Cloud Topics path is not evidence about classic Tiered Storage behavior.

Whole Cluster Restore is not a point-in-time snapshot

Tiered Storage can feed topic recovery and Whole Cluster Restore, but the pinned restore documentation states an important limit: Whole Cluster Restore does not provide snapshot-style consistency.

Different partitions can restore to different points, and committed transactions are not guaranteed to remain atomic across the restored cluster. In-flight transactions are treated as aborted.

Object storage therefore improves recovery options; it does not replace:

• a recovery-point objective;
• a recovery-time objective;
• restore drills;
• cross-partition consistency analysis;
• an active secondary cluster when the business requires one.

Object-storage failures surface on local disks

The documented behavior above produces five operational failure modes.

Upload lag consumes the local safety margin

Watch pending upload bytes, upload failures, local disk free space, and produce throttling together. Increasing local retention does not repair a provider or credentials failure.

A cache benchmark measures the wrong path

A warm-cache replay measures local disk. A first historical read measures manifest and object-store latency. Report both, and state whether the cache was empty, primed, or naturally warmed.

Retention removes remote data sooner than expected

retention.ms and retention.bytes apply at the same time, but only cleanup policies with deletion remove remote segments through retention. A byte limit can shorten a time-based promise during a traffic spike. Capacity-plan both per partition and record the cleanup policy.

An operator “pauses” Tiered Storage with the wrong property

Use cloud_storage_enable_segment_uploads for a controlled pause. Disabling remote write can create a gap between local and remote history.

Restore succeeds but the application state is inconsistent

Whole Cluster Restore can recover partitions to different points. Applications that require cross-partition atomicity need a reconciliation procedure after restore.

Trace one storage mode in source and metrics

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "class ntp_archiver" -- src/v/cluster/archival
git grep -n "class remote_partition" -- src/v/cloud_storage
git grep -n "segment_chunk" -- src/v/cloud_storage
git grep -n "Licensed as a Redpanda Enterprise" -- \
  src/v/cloud_storage/remote.h src/v/cloud_topics/app.h

On a test cluster, use rpk topic describe-storage <topic> --print-all. The pinned command reference explains its sections: summary reports storage mode and upload age; offsets show inclusive local and cloud ranges; size reports local/cloud bytes and segment counts; sync reports pending manifest or segment updates. The command describes metadata known to Redpanda—it does not benchmark a cold fetch.

Retention stops being a single number

• Tiered Storage commits locally, then uploads stable history in the background.
• Local deletion waits for upload progress, so object-store trouble can become local disk pressure.
• Total retention and local retention use different settings.
• Remote reads use manifests, range fetches, and a bounded disk cache.
• Tiered Storage and Cloud Topics have different acknowledgement paths.
• Object storage enables recovery, but Whole Cluster Restore is not a consistent snapshot.

Previous: Redpanda Storage: Logs, Offsets, Recovery, Compaction ←

Sources

• Pinned Tiered Storage docs
• Redpanda storage modes
• Pinned Cloud Topics architecture
• Pinned Cloud Topics limitations
• Pinned licensing overview
• partition_manifest.h at v26.1.13
• remote_partition.cc at v26.1.13
• remote.delete defaults at v26.1.13
• Pinned describe-storage reference
• Pinned Whole Cluster Restore limitations
• Live Redpanda documentation — secondary, moving reference