Multi-GPU Inference Starts With Per-Rank Placement

A deployment spreadsheet says the model fits because total cluster memory is larger than the saved weight files. Use one mental model before the arithmetic: the cluster is a fleet of separate trucks, not one giant cargo hold. Unused space on one device cannot hold a tensor assigned to another.

The spreadsheet says:

model weights: 1.6 TB
cluster HBM:   32 × 80 GB = 2.56 TB

A rank is one distributed worker process, usually bound to one graphics processing unit (GPU). High-bandwidth memory (HBM) is that GPU's local memory, where the rank must hold its resident model and runtime state.

The model bytes fit in the sum. The deployment can still fail.

Suppose tensor parallelism (TP) splits each layer across eight ranks, while pipeline parallelism (PP) assigns layer ranges to four stages. That TP8 × PP4 plan gives an ideal 50 GB shardable weight share per rank. Most ranks need another 4 GB of unsharded tensors, 8 GB of key/value (KV) cache, the stored attention state for prior tokens, and 12 GB of runtime reserve:

50 + 4 + 8 + 12 = 74 GB

One stage owns an 18 GB unsharded embedding or output tensor:

50 + 18 + 8 + 12 = 88 GB

That rank cannot load on an 80 GB device. Aggregate nameplate HBM never exposed the failure.

This is the first invariant: the largest rank must fit. A collective is a coordinated data exchange among ranks, and goodput is completed work that meets the declared service objective. Those questions matter only after residency passes.

Multi-GPU inference has four gates: per-rank residency, runtime-supported parallel groups, topology-aware communication, and workload goodput with a recovery plan.

The calculations and runtime documents in this post were checked on 2026-08-03. vLLM and Megatron references are pinned to retained commits; NVIDIA Collective Communications Library (NCCL) behavior is versioned separately. Recheck all support claims before reusing this plan with a newer runtime.

Use the planner below as a rejection tool for a uniform-rank lower bound. Enter one candidate parallel configuration, then increase replicated tensors or KV allocation until the modeled rank fails. The widget does not assign specific layers or experts to ranks, so it cannot identify the real largest rank or prove an uneven placement safe.

After the planner runs, complete the downloadable rank and stage ledgers: record the real rank with the highest peak, the bytes whose placement is still assumed, and the runtime measurement that could overturn the arithmetic. A green bar without those notes is not an approval.

Every rank needs its own memory ledger

For rank i:

stage-owned weights_i
+ replicated or unsharded tensors_i
+ KV or recurrent state_i
+ peak activations_i
+ communication buffers_i
+ graph and kernel workspaces_i
+ allocator and failure reserve_i
<= usable HBM_i

Read the ledger as a packing list. Weights are mostly persistent. KV grows with retained context. Activations are intermediate layer values, while workspaces are temporary kernel buffers; both can peak at particular phases. Reserve covers allocator behavior and recovery uncertainty. Prefill processes the input prompt; decode produces later tokens one step at a time.

usable HBM is what the selected runtime can safely allocate under the chosen memory policy. It is not automatically the device's marketed capacity.

The shortcut:

sum(nameplate HBM) >= saved weight bytes

is only an early rejection test. It ignores:

• uneven pipeline stages;
• unsharded embeddings, norms, routers, or output heads;
• KV-head replication and page rounding;
• quantization scales and repacking;
• prefill activation peaks;
• collective communication buffers;
• graph capture and kernel workspaces;
• allocator fragmentation;
• duplicated prefill and decode pools;
• spare capacity for recovery.

The planner on this page uses idealized shardable bytes plus entered replicated bytes, KV, and reserve. A green result is a lower bound, not runtime approval.

Parallelism dimensions solve different constraints

A replica is one complete serving copy of the model placement. Some parallelism modes make that copy fit; data parallelism creates more copies.

Choose the dimension that attacks the current bottleneck. Adding every parallelism mode because GPUs exist creates more groups, buffers, and failure paths without proving a useful result.

Tensor parallelism buys residency with layer-critical communication

Tensor parallelism splits matrix operations within a layer. Megatron-LM showed column- and row-parallel transformer projections whose partial results are combined with collectives.

For a homogeneous model:

ideal shardable weight share/rank ≈
shardable weight bytes / (TP × PP)

Then add tensors that the runtime does not shard across that group.

TP can reduce per-rank weight bytes and local computation. It also places frequent rank-to-rank combination and redistribution steps on the execution path of many layers. The next section names the exact collective operations.

Higher TP eventually loses:

• local matrix shapes become too small for efficient kernels;
• collective startup becomes visible;
• a group crosses a slower host boundary;
• tensor or head dimensions do not divide the requested degree;
• KV heads stop sharding and begin replicating;
• the selected runtime lacks a kernel for the shape and precision.

The deciding constraint is not GPU count. It is the slowest supported compute-plus-collective step under the target batch and sequence shape.

Collective names describe semantics, not achieved bandwidth

NCCL defines:

• all-reduce: reduce values and return the result to every rank;
• all-gather: gather rank-local pieces on every rank;
• reduce-scatter: reduce values, then distribute result pieces;
• all-to-all: send a different piece from every rank to every rank.

The NCCL collective documentation also requires ranks to call a collective with matching counts and dtypes; mismatches can hang, crash, or corrupt data.

A first communication model is:

time ≈ communication_steps × startup_latency
     + transferred_bytes / effective_bandwidth

For a ring approximation across p ranks, with m equal to the full logical tensor bytes:

all-reduce bytes/rank
≈ 2 × (p - 1) / p × m
 
all-gather bytes/rank
≈ (p - 1) / p × m
 
reduce-scatter bytes/rank
≈ (p - 1) / p × m

These equations estimate volume. NCCL may select rings, trees, direct paths, or topology-specific algorithms. Protocol, channel count, message size, congestion, and rank mapping decide observed latency.

“NVLink” and “InfiniBand” are topology labels, not benchmark results. Remote direct memory access (RDMA) lets a network adapter transfer data without ordinary host-copy staging when the hardware and software path support it.

The physical path crosses local buses before the network

Peripheral Component Interconnect Express (PCIe) is the host bus connecting devices and central processing unit (CPU) root complexes. A network interface card (NIC) connects the host to the scale-out network. Non-uniform memory access (NUMA) means processor sockets have different access costs to memory and attached devices. Draw the actual path through all three:

GPU
  -> NVLink/NVSwitch or PCIe switch
  -> CPU root complex and NUMA node
  -> NIC local to that root complex
  -> scale-out fabric
  -> remote NIC/root complex
  -> remote GPU

On NVIDIA/Linux hosts, these read-only inventory commands are useful when the tools are installed:

# Environment-specific inventory; no configuration changes.
nvidia-smi topo -m
nvidia-smi topo -p2p n
lspci -tv
numactl --hardware

The NCCL GPU troubleshooting guide calls out GPU peer access, GPU-to-NIC locality, PCI Access Control Services (ACS), input-output memory management unit (IOMMU) behavior, and GPUDirect RDMA prerequisites. The CUDA GPUDirect RDMA guide explains that paths crossing CPU interconnects can perform worse or fail to support the intended direct access.

ACS routes PCIe traffic under platform policy, while an IOMMU translates and isolates device memory access. Do not treat either setting as a generic performance toggle.

Do not disable ACS, IOMMU, or host security controls from a blog recipe. Capture the topology, involve the platform owner, and validate the approved configuration with vendor and operating-system guidance.

NUMA matters even when GPU kernels dominate. Tokenization, request handling, host-side buffers, page-locked memory, and NIC interrupts can run on a remote CPU socket. Record process CPU affinity, memory policy, GPU bus ID, and NIC affinity beside the benchmark.

Pipeline parallelism trades frequent collectives for stage boundaries

Pipeline parallelism assigns layer ranges to stages. A request moves through the stages instead of executing every layer on every rank.

For P stages and M forward microbatches:

makespan ≈
sum(stage_time)
+ (M - 1) × max(stage_time)

For equal stages:

utilization ≈ M / (M + P - 1)

With four stages and one microbatch:

1 / (1 + 4 - 1) = 25%

With 16 microbatches:

16 / (16 + 4 - 1)
= 16 / 19
≈ 84.2%

Those are ideal forward-pipeline ratios. Real scheduling, variable sequence lengths, stage imbalance, and communication add gaps.

PP changes placement:

• each stage owns its layers and their KV;
• stage-boundary activations cross links;
• embedding and output stages may be larger;
• one slow stage sets steady-state cadence;
• cancellation and failure require coordination across the pipeline.

A common two-host candidate is TP8 inside each host and PP2 across hosts. It keeps frequent TP collectives on a scale-up island, the GPUs connected by one host's fast local fabric, and crosses the network at a stage boundary. It is not automatically faster than TP16; it changes the traffic and bubble profile.

Data parallelism replicates capacity and cache state

Data parallelism sends different requests to model replicas.

DP=4, TP=2

uses eight GPUs as four two-GPU replicas.

For independent dense-model replicas, DP can improve aggregate throughput and fault isolation. It does not make one replica fit because each DP group owns its weights and KV.

Runtime details can break the simple picture. In vLLM commit 6c7e679, the data-parallel deployment guide documents independent KV caches per DP engine, while some MoE DP+EP configurations synchronize expert layers and even run dummy forward passes on otherwise idle ranks.

Routing needs:

• queue depth and deadline;
• current KV capacity;
• prefix-cache locality;
• tenant and priority isolation;
• replica health and failure domain.

Round robin can move a request away from the only replica that holds its warm prefix.

Context parallelism has phase- and runtime-specific meanings

Long-context serving can fail after weights fit because attention state dominates.

For equal key/value dimensions and dtype:

KV bytes/token =
layers × KV heads × (key_dim + value_dim) × bytes

Context parallelism (CP) splits sequence work or stored state across ranks. Prefill and decode need different algorithms.

At vLLM commit 6c7e679, the context-parallel deployment guide describes:

• prefill strategies with partial query and full or partial key/value state;
• decode-context parallelism (DCP) that shards KV along the token dimension;
• DCP reusing already launched TP ranks rather than adding GPUs;
• a DCP bound tied to TP size and KV-head count in that snapshot.

That support matrix can change. A memory model that divides KV by TP without checking head count, representation, backend, and DCP behavior can undercount replication.

Workshop three contains the context-state equations.

Expert parallelism is a routed communication problem

An expert is a routed feed-forward subnetwork inside a mixture-of-experts model. Expert parallelism distributes those experts across ranks.

For:

• S: source tokens on one rank;
• k: experts selected per token;
• d: hidden width sent to experts;
• b: bytes per activation element;

one-way logical dispatch payload from that source rank is approximately:

S × k × d × b

Combine returns a similar activation volume. Metadata, scales, padding, and redundant expert copies add bytes.

Average volume does not determine tail latency. The hottest destination or congested network cut gates the layer:

EP step time >=
max over destination ranks(
  queue and startup
  + received bytes / path bandwidth
  + expert compute
)

The DeepEP snapshot at dd758ca implements EP dispatch/combine paths with explicit hardware and dependency requirements. The MoonEP snapshot at 0f385f0 uses dynamic redundant experts to balance received token counts. These are implementation examples, not evidence that one EP layout works on another runtime, GPU generation, or network.

EP planning includes:

• expert-to-rank placement;
• all-to-all group boundaries;
• capacity, padding, and overflow policy;
• skew and hotspot handling;
• redundant expert memory;
• separate prefill and decode paths;
• scale-up versus scale-out cuts;
• failure and re-routing behavior.

Sparse activation reduces compute per token. It does not remove assigned expert weights from residency.

Runtime support is a versioned constraint

The table below describes pinned documents, not universal laws:

Megatron's cited guide focuses on training. Do not copy its GPU-count formulas or flags into an inference plan without checking the serving runtime.

Record:

• runtime release or commit;
• model adapter and quantization backend;
• attention, MoE, and collective kernels;
• supported tensor-, pipeline-, data-, expert-, and context-parallel combinations;
• divisibility and graph-capture constraints;
• CUDA or ROCm, driver, NCCL or ROCm Communication Collectives Library (RCCL), and container digest.

“The framework supports EP” is not specific enough to approve a topology.

Prefill and decode may need separate placements

Prefill processes many prompt tokens and often exposes large matrix multiplications. Decode repeatedly reads resident weights and growing state for small token steps.

One placement can be poor for both.

Prefill/decode disaggregation can:

• isolate prefill queue spikes from decode latency;
• choose phase-specific parallelism;
• place phases on different hardware;
• control tail inter-token latency.

It also adds:

• another weight pool or placement;
• KV transfer and possible conversion;
• cross-pool admission and backpressure;
• request ownership and retry questions;
• a failure boundary between phases.

The full path is:

prefill queue
+ prefill compute
+ KV transfer
+ decode queue
+ decode steps

Approve disaggregation only when the measured phase interference is worse than its transfer, duplication, and recovery cost.

Benchmark goodput under a named objective

Peak output tokens per second can reward a plan that misses the product's latency target.

Goodput is completed work that satisfies a declared service objective and correctness policy. A service-level objective (SLO) is the threshold that defines that promise.

For generation workloads, time to first token (TTFT) covers admission, queueing, and prefill before output begins. Inter-token latency is the delay between later generated tokens. The p50, p95, and p99 values are percentiles: 50%, 95%, and 99% of observations fall at or below them.

Every reported comparison needs this contract:

MLCommons' inference rules snapshot define a system under test as hardware and software, including memory and interconnect, and require replicable results. Their server scenario uses an arrival process and tail-latency constraint rather than an unconstrained batch. Your product benchmark can use different rules, but it needs the same clarity.

One 8K-input/1K-output result is evidence for that precision, hardware, topology, runtime, batch, and request shape. It is not a universal capacity number.

Rank failure is a placement and state-recovery event

NCCL's reliability, availability, and serviceability (RAS) subsystem has been available since NCCL 2.24 to diagnose unresponsive processes and collective mismatches. RAS improves detection and diagnosis. It does not reconstruct model placement or lost KV.

NVIDIA's fault-tolerant NCCL guidance describes aborting an affected communicator, NCCL's object for one rank group, and forming a new one. NCCL 2.27 added ncclCommShrink; the communicator documentation documents removing ranks with NCCL_SHRINK_ABORT.

Shrinking a communicator is not enough for inference continuity. After a rank disappears, the application must still answer:

• Do the remaining ranks hold every required weight and expert?
• Does the new TP, PP, EP, or CP degree divide model dimensions?
• Where does the lost KV or recurrent state come from?
• Can the request restart from retained input or a checkpoint?
• Has streamed output already reached the client?
• Is there enough HBM and queue headroom to absorb the work?

For a tightly coupled TP or PP replica, one failed rank usually invalidates that replica's current placement. A whole independent DP replica can isolate the failure only when another replica has spare capacity and request state can be replayed safely.

Reserve capacity according to the recovery unit. One spare GPU does not replace an eight-rank replica unless the runtime can form a valid smaller placement and reload the missing state.

Measure:

detection time
+ communicator teardown/rebuild
+ weight reload or remap
+ KV reconstruction
+ queue recovery
= service recovery time

Also count failed, retried, duplicated, and abandoned requests.

The workshop compares plans on a measured two-host topology

Before filling in the template, predict which candidate fails first and name the reason: residency, unsupported grouping, slow physical path, workload latency, or recovery. The measured run should either confirm that prediction or teach you which assumption was wrong.

The supplied JavaScript Object Notation (JSON) file is a planning template. It does not measure your links or prove runtime support. Its example stores exact bytes; do not mix its binary 80 GiB field with a rounded decimal “80 GB” claim.

For two eight-GPU hosts, compare candidates such as:

1. TP16;
2. TP8 × PP2;
3. TP4 × PP2 × DP2;
4. separate eight-GPU prefill and decode pools;
5. for an MoE model, EP16 versus two host-local EP8 groups.

For every candidate:

1. calculate weights, KV, activations, buffers, workspaces, and reserve per rank;
2. list every parallel group and divisibility rule;
3. mark collectives and stage transfers on the physical topology;
4. calculate approximate collective volume by message size;
5. record duplicated weights, KV, and cache locality;
6. define precision, batch, input/output distributions, and arrival process;
7. measure TTFT, inter-token latency, p99, throughput, and goodput;
8. inject a rank or host failure and measure recovery;
9. list every assumption that still needs runtime evidence.

Expected rejection logic:

• TP16 is suspect when frequent collectives cross the scale-out fabric.
• TP8/PP2 trades that traffic for stage boundaries and bubbles.
• TP4/PP2/DP2 is invalid when one eight-GPU replica still cannot fit.
• split prefill/decode pools are invalid when either pool cannot hold its own placement and reserve.
• EP16 is suspect when routed traffic crosses a low-bisection network, where limited cross-cut bandwidth constrains simultaneous transfers; EP8 needs a valid expert placement and load-balance policy.

No plan wins without the model, runtime, topology, and workload.

After the comparison, keep a rejection note for every losing plan. “Slower” is not enough; record the rank, collective, runtime constraint, SLO miss, or recovery gap that made the decision.

What could go wrong

Residency

• Aggregate HBM passes while one rank runs out of memory (OOM).
• Decimal GB, binary GiB, and runtime-reported usable bytes are mixed.
• Quantization metadata or repacking is omitted.
• KV is divided by TP even though heads replicate.
• Prefill activations overlap collective buffers.
• Expert skew grows receive buffers beyond an average estimate.
• Disaggregated pools duplicate more weights than planned.

Communication

• TP crosses hosts accidentally.
• NCCL falls back from RDMA to sockets.
• GPU ranks map to a remote NIC or NUMA node.
• Collective order, count, or dtype differs across ranks.
• A benchmark uses a different message size from production.
• Nameplate link bandwidth replaces measured collective bandwidth.

Runtime

• Tensor dimensions do not divide TP.
• PP stages are imbalanced.
• The runtime's EP backend rejects the selected TP/EP relationship.
• DCP is counted as extra devices.
• A source snapshot is cited for a later incompatible release.
• Graph capture or a quantization kernel changes peak memory.

Operations

• Nodes run different containers, drivers, or libraries.
• Locked-memory or device permissions break GPUDirect.
• Warm caches make the benchmark unlike failover.
• Average throughput hides p99 stalls.
• A rank failure has no valid placement or spare replica.
• Streamed output is retried without a client-visible duplication policy.

A topology plan is complete only when every byte has a rank, every collective has a physical path, every benchmark has boundaries, and every failure has a recovery unit.

Count GPUs last. Place the bytes, draw the links, pin the runtime, and define which requests still count after a failure.

Series completion

The reusable path is now:

1. pin and inspect the release;
2. make tool effects durable and authorized;
3. model context as stored and selected state;
4. place weights and state across physical failure domains.

Return to the Frontier Model Engineering roadmap when a new model changes the numbers but not the questions.

Sources and references

Parallelism and runtime snapshots

• Megatron-LM
• Megatron-LM parallelism guide snapshot
• DeepSpeed Inference
• vLLM parallelism and scaling snapshot
• vLLM data-parallel deployment snapshot
• vLLM context-parallel deployment snapshot
• vLLM disaggregated-prefill snapshot
• DeepEP source snapshot
• MoonEP source snapshot

Topology, collectives, recovery, and benchmarking

• NCCL collectives
• NCCL GPU troubleshooting
• NCCL RAS
• NCCL communicator resize and recovery
• CUDA GPUDirect RDMA
• NVIDIA fault-tolerant NCCL applications
• MLPerf Inference rules snapshot