Zero-Copy Loses on Small Payloads, NUMA Misses, and Page Pinning

You turned on MSG_ZEROCOPY, moved a hot path onto direct buffers, or registered a pool because the profiler kept pointing at memcpy(). The lab result looked clean. Then production sent back the bill: a slow receiver held onto pages longer than expected, p99 latency, the 99th-percentile request time, got worse, and your “zero-copy” path quietly copied anyway.

TL;DR: Zero-copy wins only when the copy it removes costs more than the setup, pinning, mapping, notification, locality, and lifetime work it adds. Linux's own MSG_ZEROCOPY docs only call it generally effective above roughly 10 KB, and even that is a mechanism-specific heuristic, not a portable threshold per Linux 7.1.

Zero-copy is not a free speedup. It is a trade that wins only when the removed byte-copy cost exceeds the new ownership cost.

Episode seven ended with borrowed memory at the application layer. This capstone prices the same trade across the whole series. Its version anchors are Linux 7.1, liburing 2.15, the Data Plane Development Kit (DPDK) 26.07, Java 25, and Python 3.14.

The break-even point is a surface, not a size threshold

A break-even frontier is the set of conditions where copy and zero-copy tie. That frontier moves with payload size, page count, segment count, reuse, locality, and safe-reuse delay. If you only ask “what size wins,” you throw away the variables that usually decide the result.

A completion lag is the delay between submit and the release event that proves the source buffer is safe to reuse. A fallback probability is the chance that the nominal zero-copy path still sends via an ordinary copy. A β coefficient is the per-byte slope of a latency curve after fixed costs have already landed. Effective copy bandwidth is the real per-byte rate your explicit-copy path achieves once cache state, allocator cost, and runtime overhead are included.

For percentile q, I model the two paths like this:

T_{\mathrm{copy},q}(B) = \alpha_{c,q} + \beta_{c,q} B

T_{\mathrm{zc},q}(B) = \alpha_{z,q} + \beta_{z,q} B
+ \gamma_q N_{\mathrm{pages}}
+ \delta_q N_{\mathrm{seg}}
+ \varepsilon_q L
+ \kappa_q C_{\mathrm{pin}}
+ \mu_q C_{\mathrm{pressure}}
+ p_{\mathrm{fb}} \Delta_{\mathrm{fallback},q}

\Delta \alpha_q = \alpha_{z,q} - \alpha_{c,q}

B^{*}_q =
\frac{
\Delta \alpha_q
+ \gamma_q N_{\mathrm{pages}}
+ \delta_q N_{\mathrm{seg}}
+ \varepsilon_q L
+ \kappa_q C_{\mathrm{pin}}
+ \mu_q C_{\mathrm{pressure}}
+ p_{\mathrm{fb}} \Delta_{\mathrm{fallback},q}
}{
\beta_{c,q} - \beta_{z,q}
}

Here B is payload bytes, N_{\mathrm{pages}} is touched pages, N_{\mathrm{seg}} is fragment count, and L is completion lag. Linux 7.1 documents page pinning, notification handling, deferred copies, and copied completions for MSG_ZEROCOPY; io_uring_prep_send_zc(3) documents split completions, notification completion queue entries (CQEs), and copied-use reporting; and io_uring_registered_buffers(7) makes the amortization model explicit by paying verification, mapping, and pinning up front. Once tail latency matters, Google's “The Tail at Scale” is the right mental frame: you care about the percentile frontier, not the average one.

If \beta_{c,q} - \beta_{z,q} \le 0 in a hostile regime, no crossover exists there. Remote memory, heavy fragmentation, or runtime staging can make the zero-copy slope no better than the copy slope. For your service, the useful number is B^{*}_{99}(\mathrm{env}), not “zero-copy after 10 KB.”

Small payloads lose because setup lands before bandwidth matters

Below the local crossover, fixed cost dominates so completely that the removed copy barely matters. That's why Linux 7.1 says MSG_ZEROCOPY is generally only effective at writes over around 10 KB. The same page also says why: you traded a byte copy for page accounting and completion notification.

That 10 KB line is useful and easy to misuse. It is a statement about one Linux socket-send mechanism, on one implementation strategy. It is not a law of networking, storage, TLS, Remote Direct Memory Access (RDMA), or managed runtimes. The liburing 2.15 registered-buffer docs make the same point from the opposite direction: amortization helps repeated small I/O, while large I/O may already dominate the setup cost.

\Delta T(B) =
\frac{B}{BW_{\mathrm{copy,eff}}}
-
\left(
T_{\mathrm{setup,zc}}
+ \frac{T_{\mathrm{pin}}}{R_{\mathrm{reuse}}}
+ T_{\mathrm{notif}}
+ p_{\mathrm{fb}} \Delta_{\mathrm{fallback}}
\right)

Zero-copy helps only when \Delta T(B) > 0. The saved copy time grows with bytes. The control-path bill lands even when B is tiny. Reuse lowers the intercept. Fallback raises it.

For small, latency-sensitive RPCs, I think copy should be your default until measured p99 evidence beats it.

A 1% fallback branch can dominate p99

A mixture distribution combines two latency populations: real zero-copy completions and copied fallbacks that went through the slow branch after you already paid some of the zero-copy control cost.

\mathbb{E}[T] =
(1 - p_{\mathrm{fb}})\mathbb{E}[T_{\mathrm{zc}}]
+ p_{\mathrm{fb}}\mathbb{E}[T_{\mathrm{fb}}]

F_T(t) =
(1 - p_{\mathrm{fb}}) F_{\mathrm{zc}}(t)
+ p_{\mathrm{fb}} F_{\mathrm{fb}}(t)

The expectation moves linearly. The tail does not. If copied fallback is only 1% but materially slower, p99 can already sit on the fallback branch. That is why fallback belongs in the steady-state model, not the exception path. Linux may accept a zerocopy send and later report SO_EE_CODE_ZEROCOPY_COPIED. io_uring exposes the same distinction in notification CQEs via IORING_NOTIF_USAGE_ZC_COPIED. And “The Tail at Scale” explains why a rare slow subpopulation still punishes shared services.

Fast in the mean. Slower where you care.

Page pinning turns one saved copy into a memory bill

A page pin keeps memory stable while the kernel or a device still references it. That isn't bookkeeping. It is a memory-management choice.

Linux 7.1 distinguishes ordinary references from FOLL_PIN and FOLL_LONGTERM because long-term pins interact with migration, compaction, and device DMA in ways ordinary references do not. io_uring_registered_buffers(7) adds two operational details teams often miss: registered memory can count against RLIMIT_MEMLOCK or cgroup memory accounting, and huge pages pin the entire huge page even if only part is used. mlock(2) explains the memlock boundary from the process side.

With steady operation rate \lambda, Little's-law-style first-order estimates are:

\mathrm{PinnedBytes}_{\mathrm{in\ flight}} \approx \lambda B L

\mathrm{PinnedPages}_{\mathrm{in\ flight}} \approx \lambda L N_{\mathrm{pages}}

\mathrm{PoolSlots}_{\mathrm{needed}} \approx \lambda L_{99}

Completion lag is the hidden multiplier. If one congested receiver doubles L_{99}, your pool requirement doubles too. No extra application work happened. The ownership window simply stayed open longer.

That is the same shape we saw on the receive side in episode 3, io_uring Zero-Copy Receive Trades Copies for Buffer Ownership: recycle lag, not raw copy cost, decides whether the pool stays healthy.

A local copy can beat remote zero-copy

Two 64 KiB payloads are not economically equivalent if one sits in local memory and one sits on the wrong socket. Non-Uniform Memory Access (NUMA) means memory latency and bandwidth depend on which CPU socket owns the page. A segment is one contiguous fragment the kernel or device has to describe separately. Scatter-gather lets one I/O operation reference many such fragments.

The Linux numa(7) page exists because placement policy matters. The DPDK 26.07 guides are blunt about this: remote memory is slower and shared read-write memory causes expensive cache misses, huge pages reduce translation lookaside buffer (TLB) pressure, and the Environment Abstraction Layer (EAL) memory model shows how topology and I/O virtual address (IOVA) shape leak into datapath behavior. Linux 7.1's DMA API also makes mapping and cacheline constraints explicit.

T_{\mathrm{copy\rightarrow local}}
\approx
\frac{B}{BW_{\mathrm{copy,local}}}
+ T_{\mathrm{touch,local}}

T_{\mathrm{remote\ zc}}
\approx
T_{\mathrm{numa}}(B)
+ T_{\mathrm{map}}(N_{\mathrm{seg}})
+ T_{\mathrm{tlb}}(N_{\mathrm{pages}})
+ T_{\mathrm{frag}}(N_{\mathrm{seg}})

That is why a copy-to-local bounce buffer, a deliberately owned local staging region, is often an optimization rather than a defeat. You pay one predictable local copy now to avoid a longer tail of remote reads, more segments, more page translations, and more device descriptors later.

Episode six, RDMA and GPUDirect Move the Bottleneck to Memory Topology, showed the same thing at a more extreme boundary: once the copy disappears, topology starts writing the bill.

P99 remembers completion lag, not your microbenchmark mean

A synchronous copied socket send usually ends source-buffer ownership when the syscall returns; an asynchronous copied send ends it at its completion. A zero-copy send can keep ownership open until a later notification. That difference is what your p99 remembers.

Linux 7.1 delivers zerocopy release notifications through the socket error queue, and recvmsg(2) defines the MSG_ERRQUEUE path that drains it. socket(7) documents how readiness and POLLERR interact with queued async errors. io_uring_prep_send_zc(3) separates the send result from the safe-reuse notification, while io_uring(7) warns that completion ordering and unsafe overlap on stream sockets need care.

The stream-overlap hazard is easy to miss. On a TCP socket, one CQE does not serialize the whole byte stream for you. If you queue SEND_ZC operations whose buffers overlap, or if you reuse a buffer after the send CQE instead of after the notification CQE, you created a race: the kernel may still need the earlier region even though one request already completed. Treat the notification CQE as the release event. Keep in-flight stream regions disjoint unless you add explicit ordering.

That is why episode two, Kafka TLS: Where the JVM Zero-Copy Fetch Path Ends, still matters here. The win or loss was never just “TLS adds copies.” TLS changed who owned the bytes and when Kafka could safely reuse memory.

Managed runtimes keep copying because safety is part of performance

In a garbage-collected (GC) runtime, the real comparison is usually not copy versus magic zero-copy. It is copy versus a native-pool redesign that gives the runtime a stable ownership contract.

Java's ByteBuffer docs say direct buffers have higher allocation and deallocation costs and should be used primarily for large, long-lived native I/O buffers. Java's MemorySegment API makes spatial and temporal bounds explicit, and the Arena API distinguishes deterministic lifetimes from GC-managed ones. Python 3.14's buffer protocol makes the same ownership point from another angle: borrowed views must be released, and exporters can impose resize restrictions while views exist.

Those APIs are telling you something. The copy often exists because moving heaps, borrowed native memory, and delayed release events are hard to compose safely. Episode seven, Zero-Copy Serialization Borrows Memory Instead of Owning It, covered the semantic side. This post adds the cost side: a borrowed slice that keeps a large direct buffer, mmap, or foreign segment alive is not free just because the CPU skipped one copy.

If your payload lives on the heap, the honest baseline may be “copy now” versus “re-architect around an off-heap pool later.”

A benchmark design should derive B^{*}_{50}, B^{*}_{99}, and B^{*}_{999}

A publishable benchmark has to prove which path ran, under what locality, and with what fallback rate. Otherwise you only measured your story.

For file-backed paths, the topology restrictions and short-result rules in sendfile(2), splice(2), and copy_file_range(2) decide what you can compare honestly. For socket zerocopy, loopback is a known trap because Linux 7.1 uses deferred copy there. Cross-host runs matter.

The loop below is deliberate pseudocode. It shows the measurement contract, not a drop-in binary.

for case in sweep_matrix():                           # ①
    obs = run_case(case)                              # ②
    verify_actual_path(obs)                           # ③
    write_csv(obs.summary())

① Sweep the variables that actually move the frontier, not just bytes.

② Keep route, offloads, cache-state class, and concurrency fixed across paths.

③ Fail the run if counters prove you benchmarked the wrong branch.

Alignment matters because page count changes piecewise:

N_{\mathrm{pages}} =
\left\lceil \frac{o + B}{P_{\mathrm{size}}} \right\rceil

where o is the starting offset within a page.

The minimum sweep I would trust looks like this:

And the output set has to include more than throughput:

Record copied-completion counts explicitly: SO_EE_CODE_ZEROCOPY_COPIED on the socket path and IORING_NOTIF_USAGE_ZC_COPIED on io_uring notification CQEs per Linux 7.1 and liburing 2.15.

Fit separate curves for copy and zero-copy, then solve for B^{*}_{50}, B^{*}_{99}, and B^{*}_{999}. If no crossover exists in the tested region, say so. That's a result, not a failure.

Sourced failure modes that move the frontier

These are not hypotheticals. They are the places where the model breaks in production.

• The fast path can vanish behind an abstraction. Kafka lost its direct transfer branch in KAFKA-2517 when the abstraction hid the concrete socket type that the optimized path needed. Your measured \beta_z does not matter if the branch is gone.
• A zerocopy send can still copy. Linux 7.1 documents copied completions for MSG_ZEROCOPY, and liburing 2.15 documents copied-use reporting. If you don't log that rate, you don't know which distribution produced p99.
• Localhost can benchmark the wrong branch. Linux 7.1 says TCP and UDP loopback use deferred copy in this path. A localhost win or loss may tell you very little about a real NIC path.
• Long-term pins can hurt the rest of the node. Linux 7.1 calls out long-term pin interactions, and liburing 2.15 warns that huge-page registrations pin the entire huge page. A tiny hot slice can still tie up a large page budget.
• io_uring can break your ownership proof if stream operations overlap unsafely. io_uring(7) warns that stream ordering and overlap need care, and liburing 2.15 splits send completion from release notification. Reusing a buffer on the first CQE instead of the notification CQE is a correctness bug first and a latency bug second.
• Custom transport work can fail correctness before it proves a win. Kafka recorded a handshake stall with a custom SSLEngine in KAFKA-16305. Throughput charts do not cover handshake, partial-write, and closure state machines.

Pick the owner, not the slogan

Most production systems need four choices, not two:

There is no global winner. B^{*}_{99}(\mathrm{env}), N_{\mathrm{seg}}, L_{99}, locality, and representation change decide it.

A decent production rule usually looks like this:

if bytes transform, mutate, or must be retained independently:   // ①
    copy
else if B < B*_99(env) or Nseg is high or memory is remote:      // ②
    copy_to_local_or_coalesce
else if p_fb is unstable or L_99 breaks the pool budget:         // ③
    copy
else:
    request_zero_copy

① Independence is value. Buy it.

② A local, contiguous buffer can be cheaper than a remote shared one.

③ If the frontier moves under load, the fallback path should move with it.

For a latency-sensitive service, I would rather pay one predictable local copy than let a 1% copied fallback decide p99.

Revisit the right episode when the culprit changes

Key takeaways

• Zero-copy wins only when the added lifetime, locality, notification, and fallback costs stay below the removed copy cost.
• The threshold is a measured frontier over bytes, pages, segments, reuse, NUMA placement, completion lag, and fallback rate.
• Tail latency and memory pressure often erase mean CPU savings.
• A copy-to-local fallback is often better than a copy-versus-zero-copy binary switch.

What's next: Rerun your own benchmark with remote memory, forced reclaim, and one slow receiver; if the winner changes, your production policy should too.

Sources and References

Linux zero-copy and transfer mechanics

• MSG_ZEROCOPY trades copy cost for page accounting and notifications, may defer to copied sends, and uses deferred copy on loopback: Linux 7.1 msg_zerocopy
• SEND_ZC uses separate completion and notification CQEs, and notification usage can include IORING_NOTIF_USAGE_ZC_COPIED: liburing 2.15 io_uring_prep_send_zc(3)
• Registered buffers amortize verification, mapping, and pinning, and huge-page registrations pin the whole huge page: liburing 2.15 io_uring_registered_buffers(7)
• Error-queue release notifications are drained with recvmsg(MSG_ERRQUEUE): man-pages - recvmsg(2)
• Socket readiness and POLLERR interact with queued async errors: man-pages - socket(7)
• File-backed transfer APIs have topology limits and allow short results: man-pages - sendfile(2), splice(2), copy_file_range(2)
• io_uring ordering and stream-socket overlap need care: man-pages - io_uring(7)

Memory management, locality, and DMA

• Long-term page pins have distinct kernel semantics and MM interactions: Linux 7.1 pin_user_pages
• Locked-memory budgets constrain pinned memory from user space: man-pages - mlock(2)
• DMA mapping and cacheline behavior add real per-segment cost and correctness constraints: Linux 7.1 DMA-API-HOWTO
• NUMA placement is a first-class performance control: man-pages - numa(7)
• Remote memory is slower and shared read-write memory causes expensive cache misses: DPDK 26.07 Writing Efficient Code
• Huge pages reduce TLB pressure and are recommended on supported systems: DPDK 26.07 System Requirements
• Memory topology and IOVA shape leak into datapath performance: DPDK 26.07 Environment Abstraction Layer

Managed runtimes and borrowed memory

• Direct buffers cost more to allocate and free and are best for large, long-lived native I/O buffers: Java SE 25 - ByteBuffer
• Memory segments make bounds, lifetime, and alignment explicit: Java SE 25 - MemorySegment
• Arenas distinguish deterministic lifetime from GC-managed lifetime: Java SE 25 - Arena
• Borrowed buffers must be released and may constrain exporter behavior while views exist: Python 3.14 Buffer Protocol

Queueing and failure records

• Tail percentiles punish variance more than mean savings: Google Research - The Tail at Scale
• A transport abstraction once removed Kafka's direct-transfer fast path: Apache Kafka JIRA - KAFKA-2517
• A custom TLS engine integration stalled handshake progress in Kafka: Apache Kafka JIRA - KAFKA-16305

The fastest byte path is not the one with the fewest copies. It is the one with the cheapest ownership.