RDMA and GPUDirect Move the Bottleneck to Memory Topology

You finally removed the hot host-side copy from a training ingest path. CPU dropped. Throughput barely moved. Then nvidia-smi topo -m explained the embarrassment: the "fast" node kept the GPU and NIC under the same PCIe switch, and the slow node crossed a socket boundary. The copy disappeared. The bottleneck moved.

TL;DR: Remote Direct Memory Access (RDMA), GPUDirect RDMA, and GPUDirect Storage (GDS) remove specific host staging copies, not byte movement itself. Memory registration is the real contract, and taking the remote CPU off the payload path trades copy work for protocol work. The next limit is usually topology, pinning, congestion, or fallback behavior.

My bias: if you cannot draw the data route from source memory to destination memory and name the event that returns buffer ownership, you do not yet have a zero-copy result. You have a hopeful benchmark.

After sockets and page-cache tricks, zero-copy becomes a memory-topology problem.

Episode five separated cache policy, copy avoidance, and DMA placement. This episode asks what happens when host memory itself stops being the obvious rendezvous point. The Linux-facing claims use Linux 7.1 and current rdma-core manuals; the NVIDIA claims use the GPUDirect RDMA and GDS documentation available on August 1, 2026.

Direct DMA removes a staging copy, not the ownership ledger

"Direct" is a narrower claim than most benchmark charts imply.

A direct memory access (DMA) engine moves bytes without a CPU copy loop. That still does not mean the bytes skipped memory controllers, DMA address translation, Peripheral Component Interconnect Express (PCIe) switches, retries, or congestion queues. RDMA's contract is that an RDMA-capable network interface controller (RNIC) can place bytes into a registered target under the transport and Linux verbs rules described by the rdma-core manuals and the RDMA Aware Networks Programming User Manual. RFC 5040 is the narrower RDMAP/DDP protocol reference, not a universal hardware contract.

That is why I still use the ownership ledger from episode 1. A bounce buffer is intermediate staging memory inserted because the real source or destination is not directly usable by the device. GPUDirect RDMA and GDS can remove a bounce buffer on supported paths, but NVIDIA's own docs still frame both as conditional data paths with platform and fallback constraints for GPUDirect RDMA, GDS overview, and GDS best practices.

Ordinary host path
app buffer [CPU]
  -> kernel networking buffer
  -> NIC DMA
  -> wire
 
Host-memory RDMA
registered host MR [CPU]                                 ①
  -> RNIC DMA
  -> wire
  -> RNIC DMA
  -> registered host MR [CPU]                            ③
 
GPUDirect RDMA
GPU memory
  -> RNIC DMA over PCIe                                  ① ③
  -> wire
  -> RNIC DMA
  -> GPU memory
 
GDS
NVMe DMA -------------------------------> GPU memory     ①
NVMe DMA -> host bounce buffer -> GPU copy engine        ②

① One ordinary host staging copy disappeared, not all byte movement.

② The API can stay the same while the physical path falls back.

③ Registration, topology, and synchronization still decide correctness.

The useful question is not "did a vendor page say direct?" It is "who owns these bytes now, who may reuse them next, and what event proves that reuse is safe?" For your service, that means a zero-copy review starts with route proof and buffer ownership, not a product claim.

Registered memory is the real RDMA contract

RDMA does not touch arbitrary process memory; it touches a registered range with explicit rights.

A memory region (MR) is a registered address range bound to one protection domain, the verbs object that scopes which queue pairs (QPs) may use that memory. ibv_reg_mr(3) defines the range, access flags, and the local key (lkey) and remote key (rkey) that authorize later operations. That is the real admission ticket to RDMA.

This is where most sloppy "zero-copy" claims break down. Linux treats DMA-pinned pages as a special VM concern because long-term pins change reclaim and migration behavior. CUDA's page-locked host memory is related, because it keeps host pages stable for device access, but it is not the same thing as a verbs MR. Verbs still needs its own registration object, access policy, and lifetime.

ibv_query_device_ex(ctx, NULL, &attr)                         ①
mr = ibv_reg_mr(pd, buf, len,
  IBV_ACCESS_LOCAL_WRITE |
  IBV_ACCESS_REMOTE_WRITE |
  IBV_ACCESS_REMOTE_READ)                                     ②
 
odp_mr = ibv_reg_mr(pd, NULL, SIZE_MAX,                       ③
  IBV_ACCESS_LOCAL_WRITE |
  IBV_ACCESS_REMOTE_WRITE |
  IBV_ACCESS_ON_DEMAND)
 
ulimit -l                                                     ④
grep -E 'VmLck|VmPin' /proc/$PID/status                       ⑤

① Query capability first. On-Demand Paging (ODP) is a device feature, not a safe assumption, as ibv_query_device_ex(3) documents.

② Conventional registration pins a specific range and creates the access contract.

③ An implicit ODP MR uses addr = 0, length = SIZE_MAX, and IBV_ACCESS_ON_DEMAND per ibv_reg_mr(3). That is not a typo.

④ RLIMIT_MEMLOCK can stop the experiment before the wire does.

⑤ VmLck and VmPin tell you whether pinning became the new bottleneck.

That is why you should separate cold-path registration from steady-state transfer. If your benchmark registers and deregisters on every request, you are mostly measuring setup churn. ODP can reduce up-front pinning by moving some cost into faults and invalidations the hardware can replay later, but that shifts work into runtime behavior and tail latency, exactly the trade described by ibv_reg_mr(3), ibv_query_device_ex(3), and the kernel's pin_user_pages docs. In shared environments, the I/O memory management unit (IOMMU), the DMA remapping layer, and Virtual Function I/O (VFIO) isolation model add one more constraint: protection and performance now pull on the same knob.

Removing the remote CPU adds protocol work

A remote read or write can remove the destination CPU from payload placement, not from correctness.

A two-sided operation like SEND/RECV only completes because the receiver posted a buffer first. That posted receive is part of the ownership protocol for you. A one-sided operation like RDMA_WRITE or RDMA_READ can bypass the remote CPU on the payload path, but then your application has to define when the bytes are valid, how long the target MR stays exposed, and how the other side learns the extent. The verbs model and work request types are spelled out in ibv_post_send(3), ibv_poll_cq(3), rdma_cm(7), and RFC 5040.

That is the real trade. You did not just choose a faster opcode. You chose whether the transport owns part of the receive contract or whether your application does.

SEND/RECV
receiver: post RECV buffers                                ①
sender:   SEND payload
RNIC:     place bytes into posted buffer
receiver: poll CQ; buffer ownership returns to app
 
WRITE + publish
receiver: register target MR and share rkey               ②
sender:   RDMA_WRITE payload into that MR
sender:   WRITE_WITH_IMM / doorbell / tail update         ③
receiver: control event publishes extent and validity

① Posted receives encode part of the ownership rule in the transport itself.

② Sharing an rkey is sharing a capability, not just an address.

③ The publish step is what turns "bytes landed somewhere" into "record is valid."

A local completion queue (CQ) event only proves local progress. It does not prove that the remote application consumed the data or even agrees on where one record ends. That is why WRITE_WITH_IMM, a tail pointer, or another doorbell signal often becomes the practical bridge between one-sided data placement and two-sided control. After reconnect, stale keys and replayed control messages become your problem too. For your queue, cache, or model-serving path, one-sided verbs usually replace a copy with a publication protocol.

This is also where IBV_ACCESS_RELAXED_ORDERING stops being a harmless tuning bit. ibv_reg_mr(3) explicitly says it relaxes write-after-write ordering for inbound RDMA writes, even though completion still guarantees visibility. I only trust it on bulk buffers whose readers do not infer meaning from the order of separate writes. If your protocol writes payload and then a header or tail pointer in the same MR, relaxed ordering can publish the small control write first.

Ethernet RDMA moves the fast path onto congestion policy

InfiniBand and RoCE can expose similar verbs while demanding very different fabric behavior.

An application can call the same verbs on InfiniBand or RDMA over Converged Ethernet (RoCE), but the operational story changes. InfiniBand carries its own fabric assumptions. RoCE rides on Ethernet, so loss behavior, queueing, switch buffering, and mark-or-pause policy become first-order design inputs. You cannot copy an InfiniBand benchmark into a RoCE cluster and call it evidence.

That is why Priority Flow Control (PFC), Explicit Congestion Notification (ECN), and Data Center Quantized Congestion Notification (DCQCN) belong in the same paragraph as your latency chart. RFC 3168 defines ECN marking. DCQCN is different: it is the de facto congestion-control loop many RoCE operators inherited from the original Microsoft Research paper, not a universal standard you can assume across vendors or fabrics.

ibv_devinfo -v                                          # ①
rdma link show                                          # ②
ethtool -S eth0 | egrep -i 'pfc|ecn|cnp|pause|drop'     # ③

① Confirm the link layer and device capabilities before you compare results.

② Map verbs devices to actual interfaces.

③ Publish pause, ECN, and Congestion Notification Packet (CNP) counters with the benchmark, not in an appendix nobody reads.

PFC can protect RNIC assumptions and still widen the failure domain with pause propagation and head-of-line blocking. ECN plus DCQCN can localize the response better, but only if the switch and NIC settings are actually documented. If you publish a RoCE benchmark without the ECN, PFC, and DCQCN settings, you published half a benchmark.

GPUDirect RDMA only wins when topology cooperates

GPUDirect RDMA is a topology-conditional fast path, not a universal feature bit.

NVIDIA's GPUDirect RDMA docs say the quiet part out loud: paths that stay under the same PCIe switch are best, paths that cross the same CPU root complex are worse, and paths that cross a Non-Uniform Memory Access (NUMA) boundary can be severely limited or unreliable on some systems in the GPUDirect RDMA guide. Linux's PCI P2PDMA docs make the same point from the kernel side: peer DMA legality depends on topology and on which device acts as provider, client, and orchestrator.

Two more constraints sit under the benchmark. Access Control Services (ACS) can force traffic upstream instead of allowing a local peer route, which is why NVIDIA's NCCL troubleshooting guide calls ACS out for GPU-to-NIC communication. The IOMMU translates and isolates device DMA. NVIDIA requires it to be disabled or configured for 1:1 pass-through mappings from the peer devices' point of view in the GPUDirect RDMA guide. That pulls directly against the isolation goals behind VFIO and IOMMU groups. It is a design trade, not a BIOS footnote.

nvidia-smi topo -m                                        # ①
lspci -tv                                                 # ②
cat /sys/bus/pci/devices/0000:65:00.0/numa_node          # ③
lspci -vv -s 65:00.0 | grep -i acs                       # ④
cat /proc/cmdline | egrep -o \
  'intel_iommu=[^ ]+|amd_iommu=[^ ]+|iommu=[^ ]+'        # ⑤
dmesg -T | grep -i iommu                                  # ⑤

Favorable
GPU0 ----\
          +--- PCIe switch --- NIC0
NVMe0 ---/                    NUMA node 0
 
Hostile
GPU0 -- RC0 -- CPU0 == CPU1 -- RC1 -- NIC0

① nvidia-smi topo -m gives labels like PIX, PXB, PHB, and SYS; nvidia-smi documents them, and they are your first clue about locality.

② lspci -tv shows whether the devices really share a switch or only look close on paper.

③ Device NUMA placement tells you where to pin CPUs and memory for the test.

④ ACS can quietly turn a hoped-for local route into an upstream detour.

⑤ Boot flags are only the request. Confirm the active IOMMU mode in kernel logs before you claim the route. For GPUDirect RDMA, translation other than 1:1 pass-through can block the path.

Pinning GPU memory is not free either. NVIDIA recommends lazy unpinning and registration caches because peer mappings consume Base Address Register (BAR) resources and pin/unpin operations are expensive in the GPUDirect RDMA guide. The GDS troubleshooting guide calls out the same pressure from another angle: too much pinned GPU memory can end in BAR exhaustion and ENOMEM.

One more caveat matters for correctness and recovery: a NIC completion does not mean a GPU kernel may safely read those bytes yet. NVIDIA explicitly says only CPU-initiated CUDA synchronization and work-submission APIs order GPUDirect operations with GPU work on the same memory, so concurrent kernel access plus RDMA writes is a data race in the same guide. That guide also documents a synchronous unpin callback used for revocation. If teardown hits that callback while a driver stack holds the wrong locks, you can deadlock in the cleanup path. For GPU racks, slot placement and deregistration strategy belong in architecture review. They are not cabling details.

GPUDirect Storage is direct only on the supported bulk path

GDS can keep bulk payload bytes off host DRAM, but it does not remove the CPU from the whole story.

In the previous episode, we covered storage paths and why bypassing one copy still leaves submission, metadata, and failure handling somewhere. GDS follows the same rule. NVIDIA's overview guide, best practices guide, and cuFile API reference all describe a narrow claim: the supported bulk-data path can move between Non-Volatile Memory Express (NVMe) storage and GPU memory without staging payload bytes in host DRAM.

That still leaves filesystems, alignment, O_DIRECT intent, transport support, and topology. NVIDIA's O_DIRECT guide and troubleshooting guide both describe compatibility and fallback behavior. Same API. Different route.

gdscheck -p                                              # ①
cat /proc/driver/nvidia_fs/stats                         # ②
cat /sys/block/nvme0n1/device/numa_node                 # ③

Fast path:
NVMe DMA -------------------------------> GPU0 memory
 
Dynamic route:
NVMe DMA -------------------------------> GPU1 staging buffer  ④
GPU1 peer path -------------------------> GPU0 memory
 
Fallback:
NVMe DMA -> host bounce buffer -> GPU copy engine -> GPU0     ⑤

① gdscheck -p tells you whether the stack even sees a supported configuration.

② Runtime stats help you catch compatibility mode and fallback instead of guessing from throughput.

③ NVMe locality still matters. A "direct" storage path can lose to a worse NUMA route.

④ NVIDIA's overview guide describes a dynamic-routing path through a better-located intermediate GPU when that route beats a hostile direct topology.

⑤ That can still outperform host DRAM staging, but it is not the same claim as endpoint-to-endpoint direct DMA.

This is the same lesson as section 5: the physical route decides whether the claim survives production. GDS can even choose a clever intermediate-GPU route and still fail later on alignment, filesystem support, or BAR pressure. For checkpoint loading or feature ingest, the payload path may improve while the control path stays on the CPU.

Security, recovery, and observability decide whether the direct path survives production

A direct path without isolation boundaries, re-registration strategy, and alerts is still a demo.

An rkey is not descriptive metadata. It is a capability that authorizes remote access to a memory range in ibv_reg_mr(3). GPUDirect RDMA has a legacy token path too. NVIDIA has considered p2pToken and vaSpaceToken deprecated since CUDA 6.0, but still documents them as authentication material when that legacy API is used in the GPUDirect RDMA guide. Treat those tokens like credentials if your stack still exposes them.

Recovery is just as explicit. Queue-pair (QP) errors, stale keys after reconnect, retry exhaustion, and completion queue pressure are not hidden by the verbs layer. The control plane you chose in section 3 has to define re-registration, re-keying, replay, and idempotence. In multi-tenant systems, VFIO's IOMMU-group model decides which devices can actually be isolated together, so "turn protections off for speed" is rarely a serious production recommendation.

One subtle production limit now shows up in cloud environments: Confidential Computing (CoCo) can reintroduce a bounce path below you. ibv_query_device_ex(3) exposes IBV_DEVICE_CC_DMA_BOUNCE when a device requires DMA bounce buffering unless the application opts into shared or unprotected memory. If your guest memory is encrypted, "direct" can disappear before the first packet leaves the host.

Async events are the bridge between correctness and observability. ibv_get_async_event(3) surfaces CQ_ERR, QP_FATAL, PORT_ERR, and DEVICE_FATAL, and applications must acknowledge those events. That should feed the same metrics and alerts as your timeout counters. If the process polls CQs but never drains async events, the first sign of a broken fast path becomes "latency looked weird yesterday."

rdma resource show                                                   # ①
find /sys/kernel/iommu_groups -mindepth 3 -maxdepth 3 -type l \
  -print                                                             # ②
devlink health show pci/0000:65:00.0                                # ③
dmesg -T | egrep -i 'mlx5|rdma|nvidia-fs|iommu|acs|pcie|aer'        # ④

① Inventory live QPs, CQs, and MRs.

② Show the actual isolation boundaries before you claim tenant safety.

③ Pull NIC health reports if the driver exposes them.

④ Kernel logs often explain the "mysterious" slow path: ACS, IOMMU, PCIe Advanced Error Reporting (AER), resets, or storage-driver fallback.

One Linux-specific trap is easy to miss on multi-NIC hosts. librdmacm warns that ordinary Address Resolution Protocol (ARP) behavior can map the remote IP to the wrong RDMA device unless arp_ignore=2 is set. If a benchmark flips between ports or resolves to the wrong NIC, you are not measuring verbs semantics. You are measuring routing luck.

No single tool proves the path. You need verbs state, async events, PCIe topology, GPU topology, NUMA placement, storage diagnostics, and logs that explain fallback. If you cannot draw the route and show the counters, you have not explained the speed.

What could go wrong

• MR registration becomes the outage — Linux treats RDMA pins as long-term DMA pins, and memory-lock limits or pin pressure can make registration fail or stretch latency in the Linux 7.1 pinning docs and ibv_reg_mr(3).
• ODP moves pain into the tail — implicit ODP MRs cut up-front pinning, but faults and invalidations now show up during request handling in ibv_reg_mr(3) and ibv_query_device_ex(3).
• A one-sided design publishes half a record — RDMA_WRITE completion and CQ polling do not define application-level validity; you still need a publish step and reconnect plan per ibv_post_send(3), ibv_poll_cq(3), and rdma_cm(7).
• IBV_ACCESS_RELAXED_ORDERING breaks your publish assumption — the man page says it relaxes inbound RDMA write ordering even though completion still guarantees visibility in ibv_reg_mr(3).
• GPUDirect takes the slow route or no route — Linux 7.1 P2PDMA docs, NVIDIA's NCCL troubleshooting guide, and the GPUDirect RDMA guide all document topology, ACS, and IOMMU constraints that can erase the gain.
• GPU memory pinning runs out of BAR space — NVIDIA documents expensive GPU pin/unpin behavior, and the GDS troubleshooting guide calls out BAR exhaustion and ENOMEM.
• The unpin callback deadlocks teardown — NVIDIA documents a synchronous revocation callback in the GPUDirect RDMA guide; wrong lock ordering can turn cleanup into an outage.
• A CoCo guest quietly reintroduces bounce buffers — ibv_query_device_ex(3) exposes IBV_DEVICE_CC_DMA_BOUNCE for that exact reason.
• GDS falls back to staging memory — NVIDIA documents compatibility mode, unsupported filesystems, alignment issues, and fallback behavior in the overview, O_DIRECT guide, and troubleshooting guide.
• RoCE protects throughput by spreading pain — RFC 3168 defines ECN, but DCQCN is still a de facto operating practice rather than a universal standard; PFC alone can widen the blast radius, which is why the DCQCN paper exists.
• Linux ARP picks the wrong RNIC — librdmacm documents the multi-NIC routing pitfall.
• The NIC finished DMA, but the GPU still read stale bytes — NVIDIA explicitly documents that only CPU-initiated CUDA synchronization orders GPUDirect RDMA with GPU kernels in the GPUDirect RDMA guide.

The simplest option that clears the bottleneck usually wins

You do not get points for the fanciest path. You get points for the simplest path that clears the measured limit.

The choice is rarely "copy" versus "no copy." It is "which ownership boundary is hot, and which extra contract can your team actually operate?" Small or irregular flows often prefer a deliberate copy. Large, hot, predictable flows can justify RDMA. GPU and storage direct paths only earn their keep when topology is favorable and fallback is visible.

Use the table as a constraint filter, not as a badge chart.

If topology is uncertain, recovery budgets are tight, or your publish protocol is still hand-wavy, the safer answer is often a deliberate copy or two-sided RDMA. The winner is the path that clears the bottleneck and still lets your team sleep.

Benchmark the boundary you moved, not just the bandwidth

After sockets, a believable benchmark separates setup cost, route, and reuse.

Measure four phases independently: allocation, registration or pinning, steady-state transfer, and completion or reuse. If you lump them together, a pretty chart can hide the fact that registration churn, ODP faults, or fallback dominated the result. Use perftest for host RDMA baselines, then add GPU-aware or storage-aware tools only after you have route proof.

Pin the benchmark to the same NUMA node as the tested devices. Publish the topology capture next to the result. Without that, "RDMA beat copy" is not evidence. It is fan fiction. The final comment below is deliberate: gdsio arguments depend on the file, direction, device, and I/O shape, so publish the exact command from your benchmark manifest instead of a fake universal invocation.

# ① Local CPU and memory placement for one-sided RDMA.
numactl --cpunodebind=0 --membind=0 \
  ib_write_bw -d mlx5_0 --report_gbits
 
# ② Two-sided baseline.
ib_send_bw -d mlx5_0 --report_gbits
 
# ③ Run the recorded gdsio command under the same numactl policy.

① Bind CPUs and memory to the device's NUMA node first, or you are benchmarking a worse route.

② Compare primitives, not just implementations: SEND/RECV, WRITE, and READ answer different questions.

③ Run storage-to-GPU tools only after gdscheck, topology, and fallback stats say the path is real.

Then report throughput, p50, p99, and p999 latency; registration latency; pinned-memory and BAR footprint; retries or receive-not-ready events; async event counts; pause, ECN, and CNP counters; PCIe link and AER warnings; GPU activity; and GDS fallback indicators. The official docs already tell you which layers can lie: Linux 7.1 pinning, ibv_query_device_ex(3), the GDS docs, and nvidia-smi topology output all belong in the appendix.

Key takeaways

• Registered memory, not the verbs call alone, is the real RDMA contract.
• One-sided verbs remove remote CPU work and add publication, replay, and key-lifetime work.
• GPUDirect RDMA and GDS only win when PCIe, NUMA, IOMMU, ACS, and fallback behavior cooperate.
• "Direct DMA" is much narrower than end-to-end zero-copy. Always publish the route, the counters, and the fallback story.

What's next: Episode 7 on zero-copy serialization moves the same ownership question up the stack, where borrowed memory can be faster than copying right up until validation and lifetime rules catch up with you.

Sources and References

Linux Kernel and RDMA Documentation

• DMA-pinned pages are a distinct VM concern: Linux 7.1 pin_user_pages() documentation
• PCI peer-to-peer DMA depends on legal topology: Linux 7.1 PCI P2PDMA documentation
• IOMMU groups define device isolation boundaries: Linux 7.1 VFIO documentation
• MR registration, access flags, ODP, relaxed ordering, and keys: ibv_reg_mr(3)
• Verbs capability discovery, including ODP and CoCo DMA-bounce flags: ibv_query_device_ex(3)
• One-sided and two-sided work requests: ibv_post_send(3)
• CQ completion semantics: ibv_poll_cq(3)
• Async QP, CQ, port, and device failure events: ibv_get_async_event(3)
• Connection management and reconnect events: rdma_cm(7)
• Linux multi-NIC ARP routing caveat for RDMA CM: rdma-core librdmacm.md
• Standard host RDMA microbenchmarks: linux-rdma/perftest

NVIDIA GPU, NIC, and Storage Documentation

• GPUDirect RDMA topology, IOMMU mode, BAR pressure, legacy tokens, unpin callbacks, and synchronization rules: NVIDIA GPUDirect RDMA
• CUDA page-locked host memory is related to, but separate from, verbs MR registration: CUDA C Programming Guide
• GPU-to-NIC communication can be limited by ACS and topology: NCCL troubleshooting
• PIX, PXB, PHB, and SYS topology labels: nvidia-smi documentation
• GPU, NIC, and storage placement guidance for DGX-style systems: DGX BasePOD deployment guide
• GDS bulk-data fast path, dynamic routing, and compatibility model: GPUDirect Storage Overview Guide
• Operational caveats and fallback guidance for GDS: GPUDirect Storage Best Practices Guide
• cuFile API control-path semantics: GPUDirect Storage API Reference Guide
• O_DIRECT requirements and fallback behavior in GDS: GPUDirect Storage O_DIRECT Guide
• Troubleshooting compatibility mode, BAR exhaustion, and mapping failures: GPUDirect Storage Troubleshooting Guide
• RDMA verbs programming model and ownership basics: NVIDIA RDMA Aware Networks Programming User Manual

Standards and Research

• RDMAP and DDP semantics for iWARP: RFC 5040
• ECN signaling on IP networks: RFC 3168
• RoCE congestion-control trade-offs at scale: DCQCN paper