Benchmarking Redpanda: A Reproducible Systems Method

Consider a hypothetical benchmark whose p99 latency, the value 99% of observations complete at or below, stays clean at 200 MiB/s. The offered load—the rate clients attempt to send—reaches 240 MiB/s, a replica, one broker-held copy of a partition log, starts recovering, and p99 rises sharply. The benchmark was not false. It answered a smaller question than the deployment needed.

Fast-path architecture matters. So does queueing, work waiting behind finite CPU, disk, network, or scheduler capacity. Durability is the failure contract an acknowledged write must survive. Cache state records whether relevant data is already in faster local storage or memory. TLS provides transport encryption and peer authentication, while compression trades CPU for fewer bytes. Partition placement assigns leaders and replicas to brokers and cores. Saturation begins when added offered load no longer produces proportional completed throughput.

A useful Redpanda benchmark is a reproducible experiment that connects one workload change to one resource boundary while preserving the failure contract.

This post pins code to Redpanda v26.1.13 and the pinned July 2026 docs snapshot used throughout the series.

Disclosure: José has not run the benchmark described below. This post defines a method and example inputs; it reports no Redpanda result.

Start with a falsifiable question

“How fast is Redpanda?” has no useful experimental answer.

Choose one question whose result can change a decision:

• How much produce throughput can this three-broker cluster sustain before p99 exceeds the service objective?
• Does enabling TLS move the bottleneck from disk to CPU?
• Does Zstandard reduce network pressure without exceeding producer and consumer CPU budgets?
• How much latency headroom remains while one replica recovers?
• What cache size keeps a historical-read workload inside its p99 target?

Then write the rejection condition before running the test.

Accept configuration X only if:
  offered load = 300 MiB/s
  produce-to-consume p99 <= 40 ms
  error rate <= 0.01%
  no shard exceeds 80% Redpanda CPU busy over the measurement window
  one broker restart completes without breaching the 2 min recovery objective

The values above are an example decision contract, not a recommendation. Replace them with the application's objectives.

Pin every layer that can change the result

A benchmark result needs enough metadata for another engineer to recreate the run.

Redpanda v26.1.13 itself pins Seastar to 7e67971d789c6642de7ec6225d37fabaa197b184. Record that dependency when a result depends on scheduling or I/O behavior; “Redpanda 26.1” is less precise than the code that ran.

Match durability by failure outcome, not familiar config names

The hardest comparison error is believing that identical Kafka property names produce identical durability.

The Raft episode traces the tagged acknowledgement path in detail. A benchmark only needs to carry the resulting contract forward:

Redpanda maps Kafka acknowledgements this way:

• acks=0 → no-ack Raft mode;
• acks=1 → leader-ack mode;
• acks=all → quorum-ack mode.

By default, write_caching_default=false, so Redpanda's acks=all path requires the flush-aware committed boundary. Enabling write caching lets acks=all complete after majority replication and flush later.

Meanwhile, Redpanda v26.1.13 classifies Kafka's min.insync.replicas and flush.messages as accepted no-op topic properties.

The pinned Redpanda benchmark guide reproduces an OpenMessaging Benchmark topic configuration containing min.insync.replicas=2, flush.messages=1, and flush.ms=0. In the pinned source, the first two do not alter Redpanda behavior. For effective behavior, the runtime source and resulting topic configuration matter more than the familiar labels in a copied YAML file.

Use an outcome table before comparing systems:

No row becomes equivalent merely because both configs contain acks=all. State the allowed broker, host, zone, power, and storage failures.

Apache Kafka's acks=all waits for in-sync replica progress, not a coordinated fsync on a majority. Kafka has log flush policies, but it has no single producer setting equivalent to Redpanda's flush-aware majority acknowledgement. flush.messages=1 changes local flush behavior; it does not turn the Produce response into proof that a majority completed fsync. A stable-media comparison must state that semantic mismatch instead of claiming exact equivalence.

Measure the hardware before measuring the broker

Redpanda uses its own I/O scheduler and needs an accurate model of disk capability. The pinned rpk iotune reference measures read and write IOPS, input/output operations per second, and bandwidth, then writes io-config.yaml.

Run it on the same storage path and hardware class as production:

rpk iotune \
  --directories /var/lib/redpanda/data \
  --duration 10m \
  --out ./io-config.yaml

The 10m duration is an experiment choice shown explicitly here. Longer runs can expose burst-credit and thermal behavior that a short run misses.

Redpanda also provides rpk cluster self-test start for disk, network, and object-storage checks. The docs warn that self-test consumes significant resources. Use it on an idle test cluster or a controlled maintenance target, not beside a live latency benchmark.

Hardware tests answer:

• what the disk and network can do in isolation;
• whether one node differs from its peers;
• whether cloud burst capacity decays over time.

They do not answer what a replicated Kafka workload can sustain.

Use a fixed-rate workload before a saturation sweep

Holding offered load constant lets you compare latency and queueing at the same demand.

The OpenMessaging Benchmark can drive fixed-rate Kafka workloads. Redpanda publishes a fork and pinned setup guide; that makes the experiment reproducible, but the fork remains vendor-maintained tooling rather than independent evidence.

Check out the exact OMB revision before building or running it:

git clone https://github.com/redpanda-data/openmessaging-benchmark.git
cd openmessaging-benchmark
git checkout --detach 5296ad06bd4d7d93896509d36b33fd2a99012f6b

Start with a workload file that makes the load shape visible:

name: redpanda-fixed-rate-1k
topics: 1
partitionsPerTopic: 144
keyDistributor: "NO_KEY"
messageSize: 1024
useRandomizedPayloads: true
randomBytesRatio: 0.5
randomizedPayloadPoolSize: 1000
producersPerTopic: 4
subscriptionsPerTopic: 1
consumerPerSubscription: 4
producerRate: 200000
consumerBacklogSizeGB: 0
warmupDurationMinutes: 10
testDurationMinutes: 20

producerRate is the workload-wide target in messages per second, divided across the configured producers; Swarm mode also divides it across workers. With messageSize: 1024, producerRate: 200000 offers about 195.3 MiB/s of value bytes:

200,000 messages/s × 1,024 bytes ÷ 1,048,576 bytes/MiB
  = 195.3125 MiB/s

That excludes keys, headers, protocol framing, replication, compression, and retries. The numbers are example inputs, not a target.

Invoke the workload with driver-redpanda/redpanda-ack-all-group-linger-1ms.yaml at 5296ad06, the same driver named in the pinned guide:

sudo bin/benchmark -d \
  driver-redpanda/redpanda-ack-all-group-linger-1ms.yaml \
  workloads/redpanda-fixed-rate-1k.yaml

The OMB source at commit 5296ad06bd4d7d93896509d36b33fd2a99012f6b defaults warm-up to 30 min when the field is omitted. Setting it explicitly keeps the YAML and the written method aligned.

Run two experiment families:

1. Fixed-rate comparison. Hold the offered rate below known saturation and change one variable, such as TLS or compression.
2. Saturation curve. Raise the offered rate in recorded steps until throughput stops following demand, errors rise, or p99 crosses the predeclared limit.

A maximum-throughput result without the curve hides where latency stopped being acceptable.

Use the envelope planner below to separate offered load, measured broker capacity, load-generator capacity, and an explicit recovery reservation. The inputs are assumptions you supply, not vendor benchmark results.

The planner deliberately predicts no p99. Any publishable conclusion still needs raw results, pinned versions, and the same durability contract on both sides.

Warm-up, measurement, and repeats need separate labels

Use an explicit protocol. One defensible example is:

1. initialize topics and verify replica placement;
2. run 10 min of warm-up;
3. measure for 20 min at fixed offered load;
4. repeat three times;
5. randomize configuration order with a recorded seed;
6. reset or preserve state according to the stated test.

These durations and repeat counts are methodological choices, not universal minimums. Increase them when compaction cycles, cloud burst credits, garbage collection in clients, or recovery behavior operate on longer timescales.

Report every run, not only the best one. At minimum, publish the median and range for throughput, p50, p95, p99, errors, and resource use.

One benchmark should include steady and degraded states

Redpanda's architecture makes background work part of foreground performance. A defensible evaluation needs more than a clean steady state.

Do not combine these changes in one run. A TLS-plus-Zstandard-plus-broker-restart result cannot identify which boundary moved.

Thread per core changes what to measure

Redpanda's thread-per-core model removes many shared locks, but it also makes per-shard imbalance visible.

The pinned Seastar tutorial describes:

• one cooperative scheduler per core;
• explicit messages between shards;
• preemption checks at defined yield points;
• scheduling groups that divide CPU shares;
• reactor stalls when CPU work does not yield.

The relevant source is the pinned Seastar tutorial, not a generic “lock-free” diagram.

Measure:

• redpanda_cpu_busy_seconds_total by shard;
• reactor-stall logs;
• partition leaders and replicas by shard;
• cross-core request pressure where exposed;
• scheduling-group and I/O-queue pressure;
• application p99 at the same time.

The pinned public-metrics reference defines redpanda_cpu_busy_seconds_total as a counter labelled by shard. Calculate its rate over the measurement window; a raw cumulative value is not a utilization percentage.

A broker average can hide one saturated shard and 15 idle ones.

Batching buys throughput with queue time

Batching reduces per-request serialization, RPC, replication, and storage overhead. It can also hold the first record while the producer waits for more.

Keep these settings in the result:

• batch.size;
• linger.ms;
• producer count;
• key and partition distribution;
• record-size distribution;
• compression;
• maximum in-flight requests.

Compare end-to-end produce-to-consume latency, not only broker request time. A larger batch may improve broker throughput while increasing application delay before the request leaves the producer.

For compressed workloads, use the same serialized payload corpus. The Kafka compression material contains a fuller replay method; the principle applies unchanged to Redpanda.

TLS and compression change the byte path

TLS creates ciphertext, and compression creates a smaller representation. Neither operation is a free “zero-copy” toggle.

Hold the security contract constant in a product comparison. If one system runs plaintext and the other runs TLS, the test compares two security models as well as two brokers.

For each run, record:

• listener protocol and cipher;
• client and broker CPU;
• logical bytes and wire bytes;
• batch size after compression;
• p99 and error rate;
• network saturation and retransmissions.

If TLS dominates broker CPU, test the current supported runtime, cipher policy, and more broker capacity. Disabling production encryption to win a chart is not a performance fix.

Read adjacent signals before naming a bottleneck

Redpanda's pinned monitoring documentation provides public metrics for CPU, memory, disk, I/O, and Kafka traffic. Build a causal table before changing settings:

One signal is rarely enough. High CPU can be useful work. A deep queue can be a short burst. Name the bottleneck only when adjacent evidence agrees.

Publish a result another engineer can challenge

A useful result table looks like this:

Add configuration files, workload files, raw benchmark JSON, and the commands used to generate charts. A screenshot is not a reproducible artifact.

Benchmark traps that survive pretty charts

These traps invalidate a benchmark even when the chart looks persuasive.

The configuration contains impressive no-ops

Redpanda accepts several Kafka properties without applying Kafka semantics. Inspect the tagged source and effective Redpanda config rather than assuming a successful alter-config changed the system.

The load generator becomes the bottleneck

Measure client CPU, network, and request backlog. Use enough client hosts and connections to offer the target rate without saturating one producer process.

A warm cache hides the production path

Label local, warm-cache, and cold-cache reads separately. Do not average them into one latency number.

The run ends before background work begins

Compaction, segment rolling, object-storage housekeeping, and cloud burst credits can operate beyond a short test. Extend the run until the relevant cycles appear.

Two runs use different offered load

Lower latency at lower demand is not an implementation win. Compare at the same offered rate, then compare saturation curves.

The benchmark uses averages

Averages hide queueing. Publish distributions and tail percentiles with the sample count and run duration.

Reproduce the source and configuration record

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git rev-parse HEAD
git grep -n "write_caching_default" -- src/v/config/configuration.cc
git grep -n "allowlist_topic_noop_confs" -- \
  src/v/kafka/server/handlers/topics/types.h
git grep -n "name = \"seastar\"" -- bazel/repositories.bzl

Before each run:

TOPIC="benchmark-topic"
 
rpk version
rpk cluster health
rpk cluster config export --filename cluster-config.yaml
rpk topic describe "$TOPIC" -c
rpk redpanda admin brokers list

After each run, save the raw result and a timestamped copy of those files. Reproducibility is part of the benchmark, not cleanup after the chart looks good.

Run the baseline before tuning

• Redpanda's architecture creates testable performance hypotheses; it does not guarantee a number.
• Durability must match by failure outcome, not property spelling.
• Hardware, clients, workload, security, cache state, and background work all belong in the result.
• Thread per core requires per-shard measurement.
• Fixed-rate runs explain latency; saturation curves explain capacity.
• Degraded-state tests reveal whether the steady-state headroom is real.

The defensible conclusion is not “Redpanda is faster than Kafka.” It is: “Under this pinned workload and failure contract, this configuration met—or missed—these limits, and here is the evidence to reproduce it.”

The next action is concrete: export the effective configuration, run one fixed-rate baseline with explicit warm-up, and save the raw result before changing a single tuning property.

Previous: Redpanda Cluster Control: raft0, Moves, and Limits ←

Back to the Redpanda series →

Sources

• Redpanda v26.1.13
• Pinned Redpanda OMB guide
• Pinned rpk iotune reference
• Pinned cluster self-test reference
• Pinned Redpanda monitoring guide
• Pinned public metrics reference
• Pinned Redpanda sizing guide
• Pinned OMB workload model
• Pinned OMB Redpanda driver
• Kafka 4.3 log flush policy
• Pinned Seastar tutorial
• Redpanda write-caching configuration source
• Live Redpanda documentation — secondary, moving reference