Zero-Copy Storage Is More Than O_DIRECT

You turned on direct I/O with O_DIRECT because “zero-copy storage” sounded like the obvious next win. The benchmark got stranger, not better: hot files slowed down, one path started failing on alignment, and the static-file server that had looked wasteful was still ahead on warmed data.

The mistake was not one bad flag. It was collapsing four different questions: did the path avoid a copy, bypass the page cache, avoid host dynamic random-access memory (DRAM), or reduce CPU data movement? O_DIRECT answers only the cache question.

Zero-copy storage is a byte-ownership decision across those four axes: where bytes live, who owns reuse, whether the page cache participates, and which completion proves the next owner is done. Episode four moved that boundary at the NIC. Here, the bytes already live on a filesystem or a Non-Volatile Memory Express (NVMe) namespace.

The version-sensitive claims use Linux 7.1, man-pages 6.18, liburing 2.15, and the Storage Performance Development Kit (SPDK) documentation available on August 1, 2026.

O_DIRECT is a cache policy, not a zero-copy certificate

O_DIRECT answers one question: should file data go through the page cache, the kernel's file-data cache? It does not answer the other three.

When engineers say “zero-copy storage,” they usually mean four different things at once:

Linux documents O_DIRECT as direct file I/O that may impose alignment restrictions and may vary by filesystem and kernel support, which is a long way from “always faster” or “always zero-copy” per open(2). Since Linux 6.1, statx() can expose those runtime alignment rules through STATX_DIOALIGN, which is the kernel telling you not to guess per statx(2).

That distinction matters because the storage stack already uses direct memory access (DMA) in ordinary buffered I/O. The device can DMA file data into page-cache pages, and the kernel can still copy from those pages into your user buffer after that. DMA happened. A copy still happened. Those are different claims.

The guard pattern below is pseudocode around the real statx() fields. The fallback helper stands for your existing buffered or filesystem-specific path.

struct statx stx = {0};
 
int rc = statx(AT_FDCWD, path, 0, STATX_DIOALIGN, &stx);  // ①
if (rc < 0 || !(stx.stx_mask & STATX_DIOALIGN) ||
    stx.stx_dio_mem_align == 0 ||
    stx.stx_dio_offset_align == 0) {
  return use_buffered_path(path);                         // ②
}
 
printf("mem_align=%u off_align=%u\n",
       stx.stx_dio_mem_align,
       stx.stx_dio_offset_align);
 
int dfd = open(path, O_RDONLY | O_DIRECT);                // ③

① STATX_DIOALIGN, available since Linux 6.1, asks the filesystem for the direct-I/O contract at runtime instead of relying on folklore.

② A successful statx() call does not guarantee that the filesystem returned the requested fields. Check stx_mask, then reject zero alignments; statx(2) uses zero to report that direct I/O is unsupported.

③ Open the direct path only after the guard passes. Memory alignment and file-offset or segment-length alignment are separate constraints, and neither proves anything about network copies, host-memory staging, or durability.

For your database, export job, or one-pass scan, this means you should stop treating O_DIRECT as a performance toggle and start treating it as a cache-ownership choice. If the page cache was helping, O_DIRECT can make you slower. If the page cache was polluting memory with one-pass data, it can help a lot. The syscall name won't decide that for you.

mmap(), O_DIRECT, and DAX create different owners for the same file

The same file can be read through copied user buffers, shared file-backed pages, direct DMA into process memory, or DAX mappings that skip the page cache entirely. Each path changes who owns coherency.

A buffered read() gives your process an independent copy. A memory map from mmap() usually does not. Instead, the process sees the file through file-backed cached pages, and the cost moves into page faults, invalidation, and shared lifetime rules per mmap(2). That is often copy avoidance into user space, but it is not page-cache bypass.

Direct Access (DAX) is the narrow exception that earns special treatment. On supported DAX filesystems and hardware, Linux can bypass the page cache and map storage directly into the process address space per the Linux 7.1 DAX docs. That is real. It is also not a general NVMe SSD trick. DAX is mostly about persistent-memory-style media and DAX-capable filesystems, not ordinary block storage.

buffered read
SSD/NVMe -> DMA -> page cache [kernel] -> CPU copy -> user buffer
 
mmap()
SSD/NVMe -> DMA -> file-backed pages [kernel]
                         <-> CPU loads/stores through mapping
 
O_DIRECT
SSD/NVMe -> DMA -> application buffer [process-owned pages]
 
DAX mmap()
storage media -> mapped address space [no page-cache path]

The hard part is not the diagram. It is the ownership rule behind each arrow. mmap() often removes an explicit read() copy, but it replaces that simplicity with shared visibility, truncation hazards, and writeback rules. DAX removes the page-cache copy too, but now you inherit DAX-specific failure boundaries and interop limits from the same Linux 7.1 documentation.

Durability stays separate. Linux exposes msync() because dirty mapped bytes do not become persistent by magic per msync(2). O_DIRECT also does not imply synchronous persistence; Linux documents O_SYNC and related flags separately from O_DIRECT per open(2).

void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); // ①
memcpy(p, src, len);                                                  // ②
msync(p, len, MS_SYNC);                                               // ③
fsync(fd);                                                            // ④

① MAP_SHARED means the mapping participates in the file's shared state. You did not create a private durable buffer.

② The copy into the mapping may save a write() syscall, but it did not settle persistence or coherency.

③ msync(MS_SYNC) waits for the mapped range to synchronize according to the mapping contract.

④ fsync() adds the file-level data and metadata synchronization your durability policy may require. Do not infer persistence from memcpy() into a mapping; whether you need msync(), fsync(), or both depends on the exact visibility and durability contract.

For analytics engines, embedded indexes, and “we'll just mmap() it” designs, this is the usual mistake: you removed the visible copy and bought the invisible problems. Of course, mmap() still fits some workloads well. It just solves a different problem than direct I/O.

sendfile() still wins when unchanged file bytes should stay kernel-owned

For unchanged file payloads, the honest fast path is usually page-cache-backed sendfile(), because it avoids user-space staging without forcing your process to own file-data lifetimes.

This is why static-file serving still embarrasses a lot of “direct I/O” experiments. If the payload is immutable and hot, the page cache is not overhead. It is the reuse mechanism. sendfile() lets the kernel move file-backed bytes to the socket path without bouncing them through a user buffer per sendfile(2).

Production software already encodes that judgment. NGINX documents a hybrid policy where sendfile on; serves the hot path, while aio on; directio 8m; can switch large file reads away from the page cache. The same docs also call out Linux direct-I/O alignment limits, including 512-byte alignment in general and 4 KiB on XFS, with unaligned ends handled in blocking mode per the NGINX core module docs.

sendfile on;   # ①
aio on;        # ②
directio 8m;   # ③

① Small and warm immutable files stay on the kernel-owned file-to-socket path.

② Asynchronous file reads help when the request path would otherwise block on storage.

③ directio 8m changes the strategy only for large reads, which is much safer than pretending every file wants direct I/O.

That layout is more honest than “direct I/O everywhere.” It matches two very different byte paths:

• Hot immutable file: NVMe -> page cache [kernel] -> sendfile path [kernel] -> NIC
• Large transformed payload: NVMe -> app buffer -> transform/check/encrypt -> socket/NIC

sendfile() also keeps its own limits. Linux documents short counts and a maximum transfer of 0x7ffff000 bytes per call, so your loop still needs normal retry logic per sendfile(2). But that is still a simpler contract than dragging every file payload into the process just to hand it back unchanged.

splice() is the nearby pipe-based mechanism: at least one endpoint must be a pipe, a kernel buffer exposed through file descriptors, and the call avoids copying payload bytes between kernel and user address spaces. Do not use SPLICE_F_MOVE as proof that Linux moved pages rather than copied them. The man-pages 6.18 contract says the flag has been a permitted no-op since Linux 2.6.21 after the original implementation proved buggy.

For static assets, artifact downloads, and model weights that you do not rewrite in flight, the storage zero-copy win is often “do not make the process own the bytes at all.”

io_uring fixed buffers remove setup work before they remove a copy

io_uring changes how you submit and complete I/O. Fixed buffers mostly remove repeated registration cost, while buffered-vs-direct semantics still decide where payload bytes live.

A buffered io_uring read still fills the page cache first and then copies into your buffer. The ring changed the submission path, not the storage ownership path. Registered or fixed buffers help when you reuse the same memory often enough that buffer validation, pinning, and mapping overhead start to matter per liburing 2.15's registered-buffer overview.

The memory contract is narrower than “register any pointer.” liburing 2.15 documents that fixed buffers must come from anonymous memory such as malloc() or mmap(MAP_ANONYMOUS); file-backed mappings are unsupported. A provided buffer ring solves a different problem: it lets the kernel choose one buffer from an application pool for operations such as socket receive or pipe read. Provided buffers use IOSQE_BUFFER_SELECT; fixed buffers use the fixed-buffer operation variants, and the two mechanisms cannot be combined.

That is why io_uring and zero-copy are orthogonal. A fixed-buffer read can still touch host DRAM twice: once in page-cache pages, then again in your registered pages. If you pair direct I/O with fixed buffers, you can remove the page-cache leg. You still did not remove host-memory staging, because the device now DMAs into your process-owned pages instead.

On the network side, IORING_OP_SEND_ZC is best-effort and usually returns two CQEs. The first reports the send result. If it carries IORING_CQE_F_MORE, a later CQE with IORING_CQE_F_NOTIF releases the source memory. With IORING_SEND_ZC_REPORT_USAGE, that notification reports IORING_NOTIF_USAGE_ZC_COPIED when the path copied instead per liburing 2.15. If the first CQE lacks F_MORE, no notification follows and that CQE is terminal for the request.

submit SEND_ZC
  -> send CQE: bytes or error, optionally F_MORE               ①
  -> notification CQE: F_NOTIF, safe to reuse source memory   ②
                       res = IORING_NOTIF_USAGE_ZC_COPIED       ③

① The initial CQE reports submission progress, not necessarily release.

② IORING_CQE_F_NOTIF is the ownership event when F_MORE promised a second CQE.

③ Copied fallback is an observed result, not a failed API call. Count it in the benchmark.

# ① Page-cache baseline.
fio --name=buf --filename=/mnt/test/file \
    --rw=read --bs=128k --size=8G \
    --ioengine=io_uring --direct=0 --iodepth=64 \
    --runtime=60 --time_based=1 --group_reporting=1
 
# ② Direct I/O plus fixed buffers.
# ③ Polling is an explicit CPU-for-latency trade.
fio --name=diofixed --filename=/mnt/test/file \
    --rw=read --bs=128k --size=8G \
    --ioengine=io_uring --direct=1 --iodepth=64 \
    --fixedbufs=1 --registerfiles=1 \
    --hipri=1 --sqthread_poll=1 \
    --runtime=60 --time_based=1 --group_reporting=1

① This run uses io_uring, but the page cache still owns the file data path.

② This run changes the storage path by adding direct I/O and long-lived registered buffers.

③ hipri and sqthread_poll trade interrupts and syscall overhead for dedicated CPU time. They save a different resource than copy avoidance.

The steady-state win can be real. So can the lifetime tax. Registered buffers pin memory and create long-term mappings. liburing documents RLIMIT_MEMLOCK accounting on kernels before 5.12; newer kernels with native workers use cgroup memory accounting instead. Huge-page registrations pin the entire huge page even when the operation uses only part of it per the same registered-buffer overview. On a storage-to-network path, you may need one proof that the read completed and another that the socket path released the buffer. That is not “more async.” That is a stricter ownership protocol.

ublk and SPDK move the storage boundary, not the laws of memory

ublk and SPDK shift more of the storage stack into user space, but they still make you own memory topology, queue lifetime, and completion.

ublk is a Linux framework for implementing block devices in user space on top of io_uring per the kernel ublk docs. That is a big layering change. It is not an end-to-end zero-copy guarantee. SPDK's own ublk documentation says the path still copies once between userspace buffers and bio pages, even though descriptor exchange uses shared rings and mmap() per the SPDK ublk docs.

SPDK moves the boundary farther. Its userspace drivers map PCI BARs into user space, allocate DMA-safe memory from hugepage-backed pools, and submit NVMe queue-pair I/O without the kernel block layer on the hot path per the SPDK userspace docs, memory docs, and NVMe docs. In its polling hot path, that can remove per-I/O syscalls, interrupt handling, and page-cache participation. It does not remove host memory. Ordinary SPDK I/O still DMAs into host-resident buffers that your process must allocate, pin, place, and retire correctly.

For Linux deployments, SPDK's memory guide strongly recommends Virtual Function I/O (VFIO) with the I/O memory management unit (IOMMU) enabled. vfio-pci lets the kernel constrain device DMA through an IOMMU-backed address space; Userspace I/O (UIO) does not provide the same memory-safety boundary. SPDK calls VFIO plus IOMMU its future-proof, hardware-accelerated foundation for userspace DMA in the SPDK memory guide and userspace-driver guide.

ublk
filesystem/app -> bio pages [kernel] -> one copy -> ublk target buffer [userspace]
 
SPDK NVMe
NVMe controller -> DMA -> hugepage-backed buffer [userspace] -> app or userspace net stack
 
storage-to-network P2P niche
NVMe controller -> NIC peer-to-peer DMA -> wire

Only the last line has a chance to avoid host-DRAM staging, and even that path is narrow. Linux's peer-to-peer (P2P) DMA support is conservative across PCIe hierarchies and can reject unsafe routing across host bridges per Linux 7.1's P2P DMA docs. SPDK marks its own peer-to-peer support as experimental and warns about topology and Access Control Services (ACS) limits per the SPDK peer-to-peer docs.

If you want to test the lower boundary instead of debating it, SPDK's own tools are the right first stop. bdevperf needs an SPDK JSON configuration that creates a bdev, plus -T to select that bdev. This minimal configuration attaches one PCIe NVMe controller; replace the address with the device assigned to the lab:

{
  "subsystems": [
    {
      "subsystem": "bdev",
      "config": [
        {
          "method": "bdev_nvme_attach_controller",
          "params": {
            "name": "Nvme0",
            "trtype": "PCIe",
            "traddr": "0000:65:00.0"
          }
        }
      ]
    }
  ]
}

Save that as bdev.json, then run:

sudo ./scripts/setup.sh                                             # ①
./build/examples/spdk_nvme_perf -q 64 -o 131072 -w read -t 60     # ②
./build/examples/bdevperf -c ./bdev.json -T Nvme0n1 \
  -q 64 -o 131072 -w randread -t 60                                # ③

① SPDK expects hugepages and userspace device access to be configured first. The setup step unbinds storage devices from the kernel, so run it only on a host and device reserved for the test.

② spdk_nvme_perf measures queue-pair I/O against the NVMe path directly.

③ -c creates the Nvme0n1 bdev from JSON; -T prevents bdevperf from silently testing every available bdev. That makes the command reproducible when a host exposes more than one SPDK device.

For a dedicated appliance or storage service, SPDK can be the right answer. For a normal backend that still wants the kernel filesystem, cgroups, and ordinary sockets, it may be the wrong boundary move.

Choose the path by the boundary you need to remove

Most storage mistakes happen when you pick the API first and the byte path second. Pick the path that removes the specific owner or copy that hurts you.

The table below separates the four claims people collapse into “zero-copy”:

My default picks are boring on purpose:

• Unchanged file bytes: keep them kernel-owned with sendfile().
• One-pass data that would pollute the cache: use direct I/O, then earn the alignment and lifetime cost.
• Reuse-heavy direct I/O: add io_uring fixed buffers after the direct-I/O case already makes sense.
• User-space block or appliance-style storage: consider ublk or SPDK.
• Host-DRAM avoidance: treat it as a hardware-topology project, not a Linux flag.

An extra copy is often cheaper than the lifetime proof you need to avoid it.

Benchmark cache state and locality, not the brand name

If you don't separate warm-cache, cold-cache, memory placement, and registered-memory pressure, your storage benchmark will tell you a fairy tale.

The baseline matrix should stay simple: buffered read, mmap(), direct I/O, then direct I/O with io_uring fixed buffers, then SPDK if you are actually willing to run a user-space storage stack. Report warm-cache and cold-cache numbers separately. Otherwise the page cache dominates the story and hides the real trade per Linux 7.1's contrast with ordinary page-cache-backed files.

Then pin locality before you chase microseconds. Non-Uniform Memory Access (NUMA) means that a buffer on the remote CPU socket costs more to reach. Linux P2P DMA depends on PCIe topology per the kernel P2P DMA docs. SPDK performance work assumes hugepages, userspace device access, and careful queue placement per bdevperf and SPDK's performance reports index. A cross-socket buffer target can erase the gain you thought you got by removing one copy.

numactl --cpunodebind=0 --membind=0 \
  fio --name=buf --filename=/mnt/test/file --rw=read --bs=128k \
      --size=8G --ioengine=io_uring --direct=0 --iodepth=64 \
      --runtime=60 --time_based=1 --group_reporting=1              # ①
 
numactl --cpunodebind=0 --membind=0 \
  fio --name=diofixed --filename=/mnt/test/file --rw=read --bs=128k \
      --size=8G --ioengine=io_uring --direct=1 --fixedbufs=1 \
      --registerfiles=1 --hipri=1 --sqthread_poll=1 --iodepth=64 \
      --runtime=60 --time_based=1 --group_reporting=1              # ②
 
numactl --cpunodebind=0 --membind=0 \
  ./build/examples/bdevperf -c ./bdev.json -T Nvme0n1 \
      -q 64 -o 131072 -w read -t 60                                # ③

① This is the page-cache baseline. Keep it, even if you expect to beat it.

② This isolates the direct-I/O plus fixed-buffer case on the same NUMA node.

③ This tests one named userspace NVMe bdev instead of assuming your application behaves like SPDK or accidentally measuring every configured bdev.

After that, record what the copy-avoidance story usually skips: p50 and p99 latency, CPU per core, page faults, memory-lock usage, cache-miss rates, short I/O counts, and whether the path actually touched the page cache. If your claimed zero-copy path spends one full core polling, that belongs in the result.

What could go wrong

• DAX mappings can fail with SIGBUS on media errors — Linux's DAX docs call out media-failure handling explicitly and note that recovery may require delete/restore or hole-punch/truncate workflows. That is a very different failure envelope from an ordinary buffered read in Linux 7.1.
• DAX interop is narrower than many teams expect — the kernel DAX docs say get_user_pages() (GUP) fails when a DAX mapping has no struct page. That blocks Remote Direct Memory Access (RDMA), sendfile(), splice(), and direct I/O from a non-DAX file into that mapped range. Direct I/O of the DAX file itself can still work; the destination memory is the deciding factor per the Linux 7.1 DAX docs.
• ublk recovery can change write behavior after failure — the kernel ublk docs describe recovery and flags that can reissue writes or fail I/O, which means your backend needs a clear double-write policy before you call the path “safe” kernel ublk docs.
• SPDK hot-remove has a real fault boundary — SPDK's NVMe docs describe how hot removal would otherwise leave BAR access paths exposed to SIGBUS, so SPDK remaps placeholder memory and completes I/O with errors instead SPDK NVMe docs.
• PCIe P2P may not exist on your machine at all — Linux blocks unsafe P2P routing across host bridges, and SPDK labels its own peer-to-peer support experimental with topology and ACS caveats in the Linux 7.1 P2P DMA docs and SPDK peer-to-peer docs.
• Directness does not buy durability — Linux documents O_DIRECT separately from O_SYNC, and mapped writes still need msync()/fsync() discipline. A direct write that returned is not the same thing as a durable write per open(2) and msync(2).

Key takeaways

• O_DIRECT answers a page-cache question. It does not prove zero-copy, no-host-DRAM, or no-CPU storage.
• mmap(), DAX, io_uring fixed buffers, ublk, and SPDK remove different kinds of work and create different lifetime rules.
• For unchanged file payloads, page-cache-backed sendfile() often remains the best storage-to-network path.
• The expensive part of advanced storage paths is usually not the API call. It is coherency, durability, NUMA placement, pinned memory, and safe buffer reuse.

What's next: Episode six moves past storage stacks into RDMA and GPUDirect, where registration, I/O memory management unit (IOMMU) mode, and PCIe topology decide whether “direct” can skip host DRAM at all.

Sources and References

Linux man pages and kernel documentation

• O_DIRECT is a direct-I/O request with filesystem-specific alignment and semantics: open(2) - man7
• Applications can query direct-I/O alignment since Linux 6.1, but must check stx_mask and zero alignment values: statx(2) and STATX_DIOALIGN
• mmap() shares file-backed state and documents truncation and SIGBUS hazards: mmap(2) - man7
• Mapped writes need explicit synchronization to reach backing storage: msync(2) - man7
• DAX bypasses the page cache on supported filesystems and hardware, while no-struct page mappings break GUP-dependent integrations: Linux 7.1 DAX filesystem docs
• sendfile() avoids user-space staging but still returns short counts and per-call limits: sendfile(2) - man7
• splice() avoids a kernel/userspace copy through a pipe, while SPLICE_F_MOVE has been a no-op since Linux 2.6.21: splice(2), man-pages 6.18
• Registered io_uring buffers require anonymous memory and create long-term pin/mapping and accounting costs: liburing 2.15 registered buffers
• Provided buffer rings are selection pools and cannot be combined with fixed registered buffers: liburing 2.15 provided buffers
• PCIe peer-to-peer DMA is topology-constrained and conservatively routed: Linux 7.1 P2P DMA docs
• ublk is a user-space block framework with documented recovery behavior: ublk docs - Linux kernel docs

User-space storage and I/O stack documentation

• IORING_OP_SEND_ZC usually returns a send CQE followed by an IORING_CQE_F_NOTIF release CQE and can report copied fallback: liburing 2.15 io_uring_prep_send_zc(3)
• SPDK maps storage control paths into user space instead of using the kernel block layer on the hot path: userspace driver docs - SPDK
• SPDK recommends vfio-pci with the IOMMU enabled for a safer, future-proof userspace DMA model: SPDK memory docs and userspace-driver docs
• SPDK NVMe queue-pair I/O changes submission/completion ownership, not the need for host buffers: NVMe docs - SPDK
• SPDK ublk still copies once between userspace buffers and bio pages: ublk target docs - SPDK
• SPDK peer-to-peer storage paths are experimental and topology-sensitive: peer-to-peer docs - SPDK
• bdevperf accepts an SPDK JSON configuration and -T <bdev> target: bdevperf docs
• SPDK publishes performance methodology and reports separately from product claims: performance reports index - SPDK

Production server documentation

• NGINX documents the production interplay between sendfile, aio, and directio, including Linux alignment caveats: core module docs - NGINX