Kafka Compression: Choosing a Codec With Real Workloads

Your Kafka brokers—the servers that own Kafka's logs—still have CPU headroom, but network traffic is near its limit and disk fills faster than planned. The review gets stuck on “Zstandard or LZ4?” while nobody asks what the producers actually batch.

A codec, the compression algorithm plus its byte format, never runs on abstract “Kafka traffic.” It runs on a particular mix of serialized records, grouped by partition, at a particular load and latency target. Change any of those and the winner can change.

My rule is simple: a codec recommendation without the real payload sample, batch-size distribution, and end-to-end latency results isn't an engineering recommendation. Compression is capacity planning, not a codec popularity contest.

Batch shape can matter more than codec choice

A producer, the client writing records, accumulates records for one topic-partition, an ordered shard of a topic. It encodes them as a record batch, a group that Kafka stores and transfers together. In Kafka's version 2 message format, compression applies to the records inside that batch rather than to each value independently.

That boundary matters. Repeated JSON field names, schema identifiers, or log prefixes only help the compressor when they land in the same batch. Kafka won't combine records from different partitions merely because their payloads look alike. A standalone test that concatenates records from 40 partitions into one file measures a batch Kafka will never create.

Two producer settings shape this boundary. Apache Kafka's current Java producer configuration lists batch.size=16384 bytes and linger.ms=5, and records that Kafka 4.0 changed the linger.ms default from 0 ms to 5 ms. Older clients, non-Java clients, and wrappers may use another default, so inspect the effective configuration rather than assuming it.

linger.ms=5 isn't a flat 5 ms toll. A busy partition can fill its batch and become eligible to send sooner. A quiet partition may reach the linger deadline with a small batch. More linger only improves compression when useful records arrive for that same partition during the wait; it can't repair sparse traffic split across too many partitions.

Before changing the codec, inspect batch-size-avg, batch-size-max, and record queue-time metrics. If most batches are tiny, test batching and codec changes separately. Otherwise, you'll know the result changed without knowing why.

Compression shifts capacity between CPU, network, and disk

With producer compression, the producer spends CPU encoding the batch. The broker validates incoming records and may decompress them during that work. When the topic preserves the producer's codec and Kafka doesn't need message-format conversion, the leader appends the encoded batch, followers replicate it, and consumers receive it in compressed form. The consumer, the client reading records, pays the final decompression cost. Kafka's compression design makes the byte savings apply to producer traffic, replication traffic, consumer fetches, and log storage.

The producer and topic configurations both use the name compression.type, but they solve different problems. The producer chooses how to encode new batches. The topic configuration either preserves that choice with producer or enforces gzip, snappy, lz4, zstd, or uncompressed. For a controlled producer fleet, I prefer producer compression with the topic set to producer; a fixed topic codec can still make sense when many unmanaged producers must follow one storage policy.

Here is a pinned starting point for one Zstd benchmark run. The topic setting belongs to a separate configuration scope, not the producer properties file.

# producer.properties
# ①
compression.type=zstd
# ②
linger.ms=5
# ③
batch.size=16384
 
# Topic configuration, applied separately
# ④
compression.type=producer

① Zstd is one candidate, not the conclusion. Repeat the run with none, lz4, snappy, and gzip.

② Pinning linger.ms prevents a client-version default from changing the experiment.

③ Pin batch.size for the same reason. Test larger batches in a separate phase.

④ producer tells the topic to preserve incoming compression when Kafka can do so. It doesn't force producers to compress: a producer using none can still write uncompressed batches.

If the topic enforces LZ4 while a producer sends Zstd, the broker may decompress and recompress the batch. If the producer sends uncompressed records, the broker performs compression itself. That policy can protect a disk or network budget, but it moves CPU onto the brokers. Producer-side compression distributes encoding work; it doesn't eliminate broker validation or consumer decoding.

Compression and zero-copy can coexist on one specific path

Zero-copy here means Kafka's classic plaintext fetch path can use sendfile()—through Java's file transfer APIs—to move log bytes from the kernel's page cache to a network socket without copying the payload through broker user space. Kafka's zero-copy design can use that path for a local log containing compressed batches because the broker can send the stored bytes unchanged.

That doesn't mean compression guarantees zero-copy. Transport Layer Security (TLS) changes the classic path because the broker must encrypt bytes before sending them. KAFKA-2517 documents that mechanism and an older Kafka SSL performance test, but its historical throughput result isn't a forecast for current hardware, JDKs, or TLS implementations. Kernel TLS, hardware offload, message conversion, and other fetch paths need their own measurement.

The precise claim is narrower: compression alone doesn't disqualify Kafka's classic plaintext sendfile path, while TLS requires a different path in Kafka's traditional implementation. Treat zero-copy as a property of the fetch path, not of the codec.

Wire-time arithmetic only settles one part of latency

A compression ratio here means uncompressed bytes divided by compressed bytes. At 4:1, four original bytes become one stored byte.

A 1 MiB batch contains 1,048,576 bytes, or 8,388,608 bits. On an otherwise idle 1 Gbit/s link, its ideal wire time is therefore 8.39 ms. If compression reduces it to 256 KiB, the ideal wire time becomes 2.10 ms—a difference of 6.29 ms. That calculation ignores Ethernet, IP, TCP, and TLS overhead, as well as queueing and contention, but it fixes the unit arithmetic.

Compression also consumes producer CPU, decompression consumes consumer CPU, and batching can add queue time. On a congested link, fewer bytes may reduce queueing enough to improve end-to-end latency. On a fast, idle link, codec CPU may dominate. Measure p99 latency, the threshold at or below which 99% of observations complete, from application send through consumer receipt. Producer request latency alone misses half the path.

Payload contents can overturn the result. JPEG images, compressed archives, and similar formats may offer little repetition. Application-level encryption, where the application turns values into ciphertext before Kafka sees them, should make those values look close to random and therefore hard to compress. Kafka TLS is different: the producer compresses the record batch before transport encryption.

Small or incompressible batches can even grow slightly because codec frames need metadata. Zstandard's format includes raw blocks for data it can't profitably compress, but framing still isn't free. Keep none in every benchmark rather than assuming some compression must win.

Cloudflare proves the stakes, not the winner

Cloudflare provides a useful bounded case study. In 2018, using a patched Kafka 1.0 deployment and eventually Zstd level 6 for its HTTP log workload, Cloudflare reported about 2.25x compression with Snappy and 4.5x with Zstd. The team reported no substantial producer CPU increase and canceled a pending Kafka hardware order.

Those figures belong to Cloudflare's payload schema, traffic, clients, batching, and machines. They don't predict Zstd's result for Protobuf records, encrypted values, or a latency-sensitive service. The case proves that compressed bytes can change a capacity plan. It doesn't prove a universal codec ranking.

Choose candidates from the resource that is running out

The codec projects provide useful starting hypotheses. LZ4 and Snappy emphasize speed. Zstandard offers a wider speed-to-density range. Kafka's KIP-390 shows that codec levels can change producer throughput, but its figures remain tied to the KIP's hardware and workload.

Use those sources to choose experiments, not winners:

Don't rank these rows by compression ratio alone. Eliminate any candidate that breaks latency or CPU limits. Among the survivors, choose the one that removes the capacity bottleneck you actually have. Tune compression levels only after that first decision.

A replay benchmark makes the choice reproducible

A useful benchmark must reproduce Kafka's batch boundaries and workload shape, not merely feed bytes to a standalone codec. Use this protocol:

1. Build an approved payload corpus. A corpus is the fixed record sample used in every run. Capture two 30 min production slices: one normal period and one peak period. Preserve each record's serialized key, value, headers, partition, and order after application compression or encryption. Use the first 10 min for warm-up and the remaining 20 min for measurement. Keep sensitive records in an access-controlled environment; random redaction can change repetition patterns and invalidate the comparison.

2. Record the complete environment. Save Kafka broker and client versions, JDK version, CPU model, instance type, partition count, TLS mode, acks, enable.idempotence, consumer count, and replication factor—the number of log copies Kafka maintains. Hash the corpus and configuration files so another run can prove it used identical inputs.

3. Create one topic per codec. Give all five topics the same partition count, replication factor, retention settings, and compression.type=producer. Run none, lz4, snappy, zstd, and gzip with linger.ms=5 and batch.size=16384. Change only compression.type during this phase.

4. Replay the original timing and partition distribution. Match the production offered load, the rate at which the application attempts to send records. Run the normal and peak corpora three times per codec, randomizing codec order with a recorded seed. For p99 claims, predeclare a minimum observation count and continue until a bootstrap confidence interval is narrower than the tolerance your decision requires. Test saturation separately by increasing the rate in fixed steps; don't compare one codec at peak production load with another at maximum throughput.

5. Measure every constrained resource. Record application p50, p95, and p99 produce-to-consume latency; records and logical MiB per second; batch-size and queue-time distributions; producer, broker, and consumer CPU; garbage-collection time; producer, replication, and fetch network bytes; retries and errors; and on-disk bytes after the same logical record count. Kafka's monitoring reference lists the client and broker metrics needed for this pass.

6. Test batching after the codec sweep. Take the two codecs that meet the first-round limits and test linger.ms values of 0, 5, 10, and 20, with batch.size values of 16384 and 65536. These are experiment points, not production recommendations. Stop testing settings whose intentional wait already exceeds the service's latency budget.

7. Set decision gates before reading the result. Use the service's existing latency objective and production CPU alert thresholds. Reject a codec that crosses either one. Then compare measured network and disk reductions using the real retention period, replication factor, and provider prices. Report medians and ranges across the three runs rather than selecting the best run.

Only after this process should you test Zstd or GZIP levels. KIP-390 found that levels could alter producer throughput more than compressed size in its specific test. Your payload may behave differently, which is exactly why level tuning belongs at the end.

What Could Go Wrong: Codec changes are compatibility changes

KAFKA-9203 documents a specific LZ4 consumption regression in Kafka clients 2.3.0 and 2.3.1. Kafka fixed it in 2.3.2 and 2.4.0. Treat it as evidence that codec implementations belong in the compatibility inventory, not as proof that mixed Kafka client versions generally disagree on the LZ4 frame format.

That bug isn't an argument against LZ4. It shows that a codec forms part of Kafka's wire compatibility surface. KIP-110 introduced Zstd support in Kafka 2.1, but non-Java client libraries added support on their own schedules.

Inventory every producer and consumer version before rollout. Run a canary, a deliberately limited deployment, on one topic and test reads with the oldest and newest supported clients. Changing the producer codec affects new batches only; old batches keep their original codec until retention removes them. A rollback can therefore leave multiple codecs in the log, and every consumer must read all of them.

The result should fit in a capacity plan

A defensible decision reports how each candidate changed producer CPU, consumer CPU, broker load, p99 latency, network bytes, and stored bytes under normal and peak traffic. “Zstd is popular” and “LZ4 is fast” aren't capacity inputs.

The key takeaways are:

• Kafka compresses records at batch level, so partitioning and linger.ms shape the result.
• Producer and topic compression.type settings control different costs.
• Pre-compressed or application-encrypted payloads can make none the right answer.
• Classic plaintext fetches can combine compression with sendfile; Kafka's TLS path changes that mechanism.
• Cloudflare shows what compression can change, not which codec your workload should use.
• Codec rollouts require compatibility testing until old batches expire.

Pick the codec that moves your limiting resource without moving latency or CPU past its boundary. That's capacity planning.

The next problem is no longer how many bytes one workload sends, but how a shared cluster decides who may send them.

Sources and references

Apache Kafka

• Kafka message format
• Kafka producer configuration
• Kafka topic configuration
• Kafka monitoring
• Kafka 4.3 producer configuration and linger.ms history
• KIP-110: Add Codec for ZStandard Compression
• KIP-390: Support Compression Level
• KAFKA-9203: LZ4 compatibility failure
• KAFKA-2517: SSL performance and zero-copy path

Codec specifications and production evidence

• RFC 8878: Zstandard Compression
• LZ4 project
• Snappy project
• Cloudflare: Squeezing the Firehose