Zero-Copy Starts by Removing Work That Adds No Value

Thought experiment: sendfile() removes a hot copy from an immutable file path. The next patch applies Linux MSG_ZEROCOPY to pooled response buffers. Under a slow receiver, the pool reuses memory before the kernel releases it.

The first change removed work. The second changed the lifetime contract.

Zero-copy is a family of techniques that avoids one or more payload copies. It never promises that no byte moves through caches, memory controllers, kernel data structures, or a network device. An owner is the component that decides when storage may be read, changed, reused, or released. My rule is more useful:

Keep bytes with their current owner until another owner must transform, inspect, isolate, or retain them.

That is a design discipline, not a count of application-to-kernel calls. It explains why sendfile() fits an unchanged file response, why registered buffers in io_uring, Linux's shared submission and completion interface, are not automatically zero-copy, and why a deliberate copy can be the safer design.

Draw the ownership ledger before choosing an API

A byte path is the sequence of representations and owners a payload crosses from source to sink. A representation is the actual byte sequence at one stage. The page cache is kernel memory that caches file contents.

The Linux contracts in this essay are pinned to man-pages 6.18, Linux 6.19, and liburing 2.15, all available by August 1, 2026. Distribution kernels can backport or disable features, so a release number is a review baseline, not runtime proof.

Start with a ledger, not an API name:

unchanged file payload
storage -> page cache [kernel] -> TCP transmit path [kernel] -> NIC
 
ordinary user-buffer send
application buffer [process] -> socket path [kernel] -> NIC
 
user-space TLS over file data
page cache -> plaintext buffer [process] -> ciphertext buffer [process]
           -> socket path [kernel] -> NIC

The arrows mark ownership or representation boundaries. They do not claim that every boundary executes one CPU copy. The network interface card, or NIC, still reads memory and emits frames. The machine still moves bytes.

For each arrow, ask:

1. Are the bytes identical on both sides?
2. Which component needs to inspect or change them?
3. Who may mutate or free the source storage?
4. What event proves that the next owner has finished with it?
5. Which measured resource does this crossing consume?

The fourth question is the correctness boundary. The fifth decides whether the work is worth removing.

Linux transfer mechanisms remove different crossings

A system call, or syscall, is an application entry point into the kernel. io_uring is a submission and completion interface for syscalls and related operations. Neither name tells you which payload copy disappeared.

The mechanisms below solve different problems:

copy_file_range() and sendfile() both perform in-kernel file transfers, but their sinks differ. splice() adds a pipe-backed handoff. Registered buffers amortize setup; they do not change an ordinary read into a file-to-NIC transfer. IORING_OP_SEND_ZC changes the socket-send mechanism, not every operation submitted through the ring.

That separation matters in code review. “We use io_uring” says how work is submitted. It does not say where the payload lives or which owner disappeared.

A completion proves release, not delivery

Copy avoidance usually turns a simple return value into a lifetime protocol. The simplified MSG_ZEROCOPY sequence makes that visible:

setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one)); // ①
send(fd, buf, len, MSG_ZEROCOPY);                           // ②
recvmsg(fd, &msg, MSG_ERRQUEUE);                            // ③

① The socket requires an explicit opt-in before zero-copy sends.

② A successful send accepts the zero-copy lifetime contract. The kernel may share the pages or take a deferred copy, but the application must keep the bytes stable until the notification arrives.

③ The error-queue notification says the kernel released its references. It does not say the peer received the bytes. Linux can batch notification ranges, and retransmission or socket teardown can make notifications arrive out of order. Inspect SO_EE_CODE_ZEROCOPY_COPIED to learn whether the accepted send used the copied path.

The same distinction appears in io_uring. A completion queue entry (CQE) is one result in the ring's completion queue. IORING_OP_SEND_ZC usually emits a send CQE first. Only when that CQE carries IORING_CQE_F_MORE will a later CQE with IORING_CQE_F_NOTIF report that the memory is safe to reuse.

If the first CQE has IORING_CQE_F_MORE, recycling the pool slot before the notification CQE is a correctness bug. If that flag is absent, no notification CQE follows and the first CQE is terminal for that request. The optimization can remove a copy, but only the flags tell the application which lifetime contract it received.

What could go wrong: the fast path is conditional

The API name is weak evidence. The result path is what matters.

• The kernel may copy after accepting MSG_ZEROCOPY. Linux documents deferred copies when a device lacks scatter/gather I/O, the ability to describe one operation with several memory fragments, or when the stack needs private data for work such as checksum calculation. The notification then carries SO_EE_CODE_ZEROCOPY_COPIED.
• A local benchmark can test the wrong branch. Linux 6.19 deliberately performs a deferred copy for TCP and UDP traffic looped back to the same host because notification latency could otherwise become unbounded. Test this mechanism between hosts before drawing a throughput conclusion.
• Short transfers are normal control flow. sendfile(), splice(), copy_file_range(), and socket sends can all complete fewer bytes than requested. The caller must advance offsets and retry without duplicating or skipping data.
• A successful file copy once meant less than it appeared. The copy_file_range(2) man page records a Linux 5.3 through 5.18 bug where some virtual filesystems could report success without copying the data. Current code should still validate the destination when correctness requires it.
• Pinned memory becomes backpressure, downstream delay that limits upstream progress. A slow socket extends the lifetime of shared pages. Registered buffers pin pages, create long-term mappings, and keep replaced resources alive until in-flight operations finish. Depending on the kernel, those pages consume RLIMIT_MEMLOCK or cgroup memory accounting. A design that removes copy bandwidth can make pool or accounted-memory exhaustion the next limit.

Fallback is not always a failure. It may be the kernel choosing the safe path for this device, payload, or route. The failure is treating the requested path as the observed path.

Copy when the new owner needs independence

Here is my default judgment: copy when the copy buys a smaller, independent, or easier-to-prove lifetime.

That includes:

• a parser retaining one field from a much larger frame;
• a task that must release a pooled request buffer before a slow consumer finishes;
• mutable input that downstream code must observe as a stable snapshot;
• compression, encryption, serialization, or transcoding that produces a new representation;
• small writes where page accounting and completion processing cost more than copying.

Linux 6.19's MSG_ZEROCOPY documentation says its page-pinning implementation is generally effective only for writes above roughly 10 KB. That number describes one kernel mechanism, not a portable application threshold. Payload shape, route, NIC, concurrency, and completion batching can move the break-even point.

A copy stops being the safer default when large immutable payloads dominate a measured CPU, cache, allocation, or memory-bandwidth limit and the application can bound their shared lifetime. That is the failure envelope. Do not pay lifetime complexity for a path that another resource already caps.

Measure the constrained boundary, not the syscall count

A hot path consumes a material share of the resource currently limiting the workload. Source inspection can prove that a copy exists. Only a controlled measurement can prove that removing it changes the result.

Use the same payloads, offered load, concurrency, cache state, route, and device configuration for both paths. Then record:

1. end-to-end throughput and latency;
2. CPU profiles and allocation rates;
3. short-transfer, fallback, and copied-notification counts;
4. buffer hold time, pinned or registered memory, and completion backlog;
5. page faults, cache events, and memory-controller events where the platform exposes reliable counters.

Linux perf stat collects counter statistics, while perf record samples the code consuming CPU. Hardware events differ across processors, and profiling perturbs the workload. Record the event set and tool overhead, then confirm the end-to-end result without the profiler attached.

One fewer syscall can leave the payload path unchanged. One extra completion can hold memory long enough to erase the saved copy. Stop when the first saturated boundary no longer moves.

Kafka and Valkey remove different owners

Kafka earns its file-transfer opportunity before Linux sees a file descriptor. A record batch is a group of records stored and transferred together. Kafka's shared binary format lets a broker serve those stored bytes without rebuilding each record. The Kafka compression essay shows why compressed batches can still use that plaintext file path when the broker sends them unchanged.

Valkey removes another crossing. Its reply copy-avoidance implementation borrows an eligible string object's storage and uses writev() to gather protocol framing with the payload. That removes the object-to-reply-buffer copy. It does not turn an ordinary socket writev() into sendfile() or MSG_ZEROCOPY.

Both systems follow the same discipline: remove the owner that adds no value, then keep the remaining ownership contract explicit.

Zero-copy is the question, not the answer

Ask who owns the bytes next, what value that owner adds, and which event ends the handoff. If the answer is “no one adds value,” remove the crossing. If the answer is transformation, isolation, or a shorter lifetime, keep the copy and stop apologizing for it.

Part two applies this ledger to Kafka's plaintext and TLS fetch paths, where the same record batch acquires a different owner before it reaches the socket.

Sources and References

Linux transfer contracts

• sendfile(2), man-pages 6.18: in-kernel file-descriptor transfer, partial results, fallback, and source mutation rules
• splice(2), man-pages 6.18: pipe requirement, page-reference model, partial results, and SPLICE_F_MOVE
• copy_file_range(2), man-pages 6.18: regular-file scope, filesystem acceleration, sparse-file behavior, and historical failure records
• Linux 6.19 MSG_ZEROCOPY documentation: page sharing, error-queue notifications, deferred copies, checksum limits, and loopback behavior
• writev(2), man-pages 6.18: gather writes from multiple user buffers

io_uring ownership contracts

• liburing 2.15 registered-buffer overview: long-term mappings, memory accounting, resource updates, and fixed-buffer semantics
• liburing 2.15 io_uring_prep_send_zc(3): send CQEs, reuse notifications, copied-use reporting, and error behavior
• liburing 2.15 io_uring_prep_splice(3): asynchronous submission of the splice() mechanism

Architecture and measurement

• Apache Kafka 4.3 design: efficiency: shared record batches, page-cache use, and the plaintext file-transfer path
• Linux 6.19 perf stat
• Linux 6.19 perf record