Kimi K3 Hardware Requirements: Why 8 H100s Are Not Enough

You open the Kimi K3 model card, see 104B active parameters and MXFP4 weights, then sketch a deployment note: eight H100s should run it. Active parameters are the weights used for one token; MXFP4 is a grouped 4-bit floating-point storage format. Serving is the production path that loads those weights and generates responses for users.

The shortcut confuses work with inventory. Active parameters are the boxes opened for one order; checkpoint bytes are every box the warehouse must keep available. A sparse model can open only a few boxes per token and still need a very large warehouse.

It fails before the server starts.

Eight H100 SXM GPUs provide 640 GB of nominal high-bandwidth memory. Kimi K3's 96 released weight shards occupy 1,560,936,091,448 bytes. Those files exceed the cluster by about 921 GB before the runtime allocates cache, recurrent state, communication buffers, activations, CUDA graphs—captured GPU execution sequences—or allocator headroom. Repository code and metadata add disk bytes, not resident weights.

Kimi K3's active parameter count describes work performed for one token. It does not describe how many parameters the serving cluster must store.

TL;DR: The pinned Hugging Face revision reports 2,779,931,837,184 logical parameters. Its 96 visible weight shards total 1,560,936,091,448 bytes, and all 118 visible files total 1,560,998,984,390 bytes. Eight H100s cannot hold that checkpoint. Documented serving recipes start at 32 H100s or 16 H200s in SGLang and eight GB300 or MI350X/MI355X GPUs in vLLM. A 1M-token request then adds 27 GiB of BF16 per-rank global-attention state before any token-position sharding. Evidence and calculations were checked on August 3, 2026.

Four labels keep the rest of the post honest:

• Official fact comes from a pinned artifact, report, or serving recipe.
• Derived calculation follows directly from those published values.
• Planning estimate adds an explicit assumption, such as even placement.
• Deployment hypothesis is a configuration that still needs a benchmark.

Before using the planner, set only the GPU family and count. That first pass answers whether the weight lower bound fits. Then add context length, cache precision, and runtime reserve to expose request-level pressure.

After the planner, record the first resource that crosses its boundary. A red result rejects the configuration under the stated inputs. A green result is a deployment hypothesis, not proof of throughput, useful work completed per second; latency, time per request; or reliability.

The released bytes settle the eight-H100 question

High-bandwidth memory (HBM) is the fast memory attached directly to an accelerator. A server can only use the model after every required weight is resident somewhere in the serving topology: the GPUs, links, and process layout that act as one deployment.

Official fact — pinned artifact. The pinned Kimi K3 artifact metadata reports:

Planning estimate — resident placement. For HBM, the weight-shard total is a conservative file-size proxy. A serving engine can transform the representation during load, so only a measured per-rank ledger proves the resident layout.

The Hugging Face API's usedStorage field is 19,259,278 bytes larger than the sum of the visible files in the pinned revision. The API does not explain that difference, and no visible file accounts for it. Treat usedStorage as repository-level accounting rather than a downloadable snapshot manifest. For download and disk planning, use the visible-file sum and then add the artifact client's cache, staging, conversion, and rollback behavior.

Derived calculation — hard rejection. Eight H100s provide:

8 GPUs × 80 GB/GPU = 640 GB nominal HBM
 
1,560.936 GB released weight shards
- 640 GB aggregate HBM
= 920.936 GB shortfall

The conclusion does not depend on treating file bytes as resident HBM. Even the more generous flat 4-bit counterfactual fails:

2,779,931,837,184 parameters × 4 bits ÷ 8
= 1,389,965,918,592 bytes
= 1.390 TB

That is more than twice the HBM in eight H100s. The actual checkpoint is larger because it carries quantization scales and leaves selected modules at higher precision. Part four builds a transparent checkpoint reconstruction and labels what still needs a tensor-level audit.

Active parameters reduce computation, not residency

Kimi K3 is a mixture-of-experts (MoE) model, a model that routes each token through a subset of specialist feed-forward networks instead of executing all of them. Routing is that per-token assignment to selected experts.

The released configuration contains 896 routed experts, 16 selected experts per token, and two shared experts. Moonshot reports 104.2B activated parameters.

The tempting calculation is:

2.7799T × 16 / 896 ≈ 49.6B

That multiplication applies the routing fraction to the entire model. The entire model is not routed expert weight.

Every token still uses attention layers, embeddings, routers, latent projections, the vision stack when media is present, one dense layer, and two shared experts. Only the routed-expert portion follows the 16-of-896 rule. That is why Moonshot's disclosed count is about 104.2B rather than 49.6B.

The same distinction matters operationally:

• Computation: only selected routed experts execute for a token.
• Weight traffic: only selected expert weights need to participate in that token's routed calculation.
• Residency: all experts must remain available across the cluster because a later token may select any of them.
• Communication: the serving engine must move token activations to the ranks that own the selected experts.

Sparse execution makes a 2.8T model possible. It does not turn a 1.56 TB checkpoint into a 104B dense model.

Published topologies start where the checkpoint has room to breathe

The launch recipes tell a more reliable story than community arithmetic. Their GPU counts are official recipe facts. The headroom columns below are derived estimates that assume an even split.

An even split gives a lower bound for per-rank placement. A rank is one participating runtime process, usually bound to one GPU:

Real placement can be worse. Dense modules may replicate, expert counts may not divide evenly across the chosen parallelism, and each rank still needs cache, activations, collective workspaces, and allocator headroom.

The HBM capacities come from NVIDIA's H100, H200, and HGX B200/B300 documentation plus AMD's MI355X system guide. The SGLang K3 cookbook lists 32 H100s, 16 H200s, 16 B200s, and eight B300- or MI35x-class accelerators. The vLLM recipe starts at eight GB300s or eight MI350X/MI355X GPUs and recommends multi-node deployment for real production traffic.

SGLang still marked every recipe cell Final Verification In Progress in the July 30 revision. Treat these as documented bring-up shapes, not independently reproduced minimums or production guarantees.

Moonshot goes further. Its launch post recommends supernodes with 64 or more accelerators for efficient production inference. That recommendation is about throughput and communication, not the minimum number of devices required to place the bytes.

This is the first boundary to keep clear:

A checkpoint-fitting topology answers “can the weights be placed?” It does not answer “will the service meet my concurrency, latency, or availability target?”

Nominal HBM is also not fully allocatable. Firmware, runtime contexts, communication workspaces, kernels, graphs, and fragmentation consume memory. The table therefore reports weight-only headroom, not usable cache capacity.

Fixed and token-growing attention state create two memory limits

Use a two-drawer mental model. Every request gets one fixed-size drawer for recurrent state and one drawer that grows as the conversation gets longer. Either drawer can fill first.

Kimi K3 interleaves 69 Kimi Delta Attention (KDA) layers, recurrent layers that update a fixed state, with 24 Multi-Head Latent Attention (MLA) layers, global attention layers whose serving cache grows with tokens.

That creates two independent pools:

1. a relatively large fixed KDA state allocation for every running request;
2. an MLA key-value cache (KV cache), stored attention state that grows with context length.

The SGLang engineering report describes this split directly. A short-request workload can exhaust KDA state slots while leaving most of the MLA pool unused. A few very long requests can fill MLA KV while the state pool has space.

One KDA state slot is about 53.6 MiB at TP8

Tensor parallelism (TP) divides compatible tensors and attention heads across GPUs; TP8 means eight participating ranks. K3 has 69 KDA layers, 96 heads, and a 128 × 128 recurrent state per head. With TP8 and FP32 recurrent state:

local heads = 96 / 8 = 12
 
recurrent state
= 69 layers × 12 heads × 128 × 128 × 4 bytes
 
short-convolution state
= 69 × 3 projections × 3 retained positions × 12 × 128 × 2 bytes
 
total
= 56,171,520 bytes
≈ 53.6 MiB per state slot per rank

SGLang's normal buffering strategy can reserve five slots per running request, or roughly 268 MiB per rank before speculative-decoding state. Its day-zero report describes a unified-memory design because static KDA and MLA pool splits can strand free memory on the wrong side.

One full-context BF16 MLA cache is 27 GiB per rank

K3's 24 MLA layers cache a 512-wide latent representation plus 64 additional channels:

24 layers × (512 + 64) values × 2 bytes
= 27,648 bytes/token
= 27 KiB/token

At 1,048,576 tokens:

27 KiB/token × 1,048,576 tokens
= 27 GiB per request per rank

FP8 KV halves that to about 13.5 GiB. Decode context parallelism (DCP), a serving technique that divides token positions across ranks, can divide the MLA cache further. DCP8 reduces the 27 GiB figure to about 3.375 GiB per rank.

DCP does not shard the checkpoint. It also does not remove the per-request KDA state. It solves one side of the memory equation.

A 1M context window says nothing about how many such requests fit concurrently. Raising a server's accepted context limit cannot create cache capacity.

Worked per-request ledger — derived, TP8, one 1M-token request:

This ledger excludes weights, activations, workspaces, communication buffers, and allocator headroom. It explains the request's two main state buckets; it does not claim that either total is the runtime's complete allocation.

Optional architect depth: if you only need a fit/no-fit decision, skip to the deployment preflight. The next two sections explain transient memory and why GPU count alone does not choose a parallelism plan.

Prefill activations can consume gigabytes before kernels add workspaces

Prefill is the phase that processes the incoming prompt; decode generates new tokens one at a time. Weight and cache arithmetic still misses transient activations, the intermediate tensors created while a batch moves through either phase. Their size depends on live tokens, parallelism, kernel fusion, and whether the engine keeps or recomputes intermediates.

The released 7,168-wide backbone gives a useful lower-bound calculation. For a 16,384-token prefill chunk in BF16, Attention Residuals (AttnRes)—K3's learned selection among representations from earlier depth blocks—can retain nine sources near the top of the model:

one hidden buffer
= 16,384 tokens × 7,168 values × 2 bytes
= 234,881,024 bytes
= 224 MiB
 
nine Attention Residual sources at the top of the model
= 9 × 224 MiB
= 1.969 GiB
 
one materialized 16-way routed latent payload
= 16,384 × 16 experts × 3,584 values × 2 bytes
= 1.750 GiB

These are logical aggregate payloads, not measured per-rank allocations. Sequence parallelism divides live tokens across ranks, expert parallelism divides expert work and routed copies, and fused kernels can avoid some materialization. Conversely, input/output pairs, routing metadata, alignment, collective buffers, and CUDA graphs add memory. A runtime trace—not a fixed percentage of checkpoint size—settles the activation budget.

Parallelism choices solve different constraints

The acronyms become easier once each one is tied to the resource it changes.

The vLLM release post describes separate MoE backends for tensor- and expert-parallel deployments. The SGLang report splits prefill and decode strategies because the two phases stress hardware differently.

An eight-GPU aggregate-memory calculation cannot tell you which combination is valid. The model's 896 experts make all-to-all bandwidth a first-order design constraint. A cluster can have enough HBM and still produce unusable throughput because the topology cannot move routed tokens efficiently.

Disk, load time, and cold starts belong in the hardware plan

The visible pinned revision is roughly 1.561 TB. That creates three separate storage questions:

1. Artifact storage: where the 96 shards and model files live.
2. Staging behavior: whether the download cache and working directory share files or duplicate them.
3. Runtime expansion: whether a loader materializes higher-precision or converted tensors.

At theoretical line rate, downloading 1.561 TB takes:

Real transfers include protocol overhead, source throttling, shard startup, storage writes, and verification. These figures are lower bounds, not observations from Hugging Face.

The vLLM launch instructions required a dedicated K3 Docker image because several dependencies were still prerelease. That is another cold-start dependency: pinning the model revision is not enough when the engine image and kernels are changing at the same time.

What could go wrong even after the bytes fit

Several launch-day failure modes were documented by the serving projects:

• The H100 topology has the least post-weight headroom. SGLang's cookbook uses 32 devices, but that does not make large contexts or high concurrency automatic.
• DCP can conflict with other cache features. The SGLang recipes document restrictions around higher cache tiers and speculative state because token ownership changes.
• Speculative decoding and pipeline parallelism are not freely composable. Published K3 configurations constrain which combinations can start.
• The wrong KDA/MLA pool split can produce an out-of-memory error while the other pool remains mostly empty. SGLang introduced unified memory to address that workload sensitivity.
• A model parser can fail even when inference succeeds. The vLLM recipe warns that K3 can emit a tool-call form its parser does not expect, so production agents should validate and retry.
• A context limit can pass validation and fail allocation. The configured maximum checks request length; it does not prove that current cache pools can serve the request.

None of these failures is fixed by adding the word “sparse” to the architecture diagram.

Use this deployment preflight before reserving GPUs

1. Pin the artifact. Record the Hugging Face revision, stored bytes, and shard count.
2. Reject impossible aggregate HBM. If nominal HBM is below the 1.390 TB flat four-bit lower bound, stop. Use the 1.561 TB weight shards as the next planning proxy, not a measured resident layout.
3. Reserve runtime headroom. Treat the percentage as an explicit planning assumption, then replace it with measurements.
4. Choose TP, EP, PP, and DCP from the serving engine's supported matrix. Do not infer compatibility from the acronyms.
5. Model both request pools. Calculate KDA state slots and MLA KV for the expected context distribution.
6. Verify the fabric. Expert routing makes NVLink or remote direct memory access (RDMA) configuration part of model performance.
7. Plan disk and cold starts. Include caches, staging, container images, and recovery downloads.
8. Benchmark your workload. Input length, output length, concurrency, prefix reuse, reasoning effort, and latency target must match production.

Kimi K3 is open-weight. It is not laptop-scale, workstation-scale, or an eight-H100 deployment.

The first honest hardware requirement is simple: store the model that was actually released. Everything after that is systems engineering.

Part two traces how KDA, Attention Residuals, and 896 experts divide work across sequence, depth, and width.

Sources and References

Released model

• Artifact counts and stored bytes: Pinned Hugging Face model API
• Architecture and active parameters: Pinned Kimi K3 model card
• Quantization and dimensions: Pinned Kimi K3 config
• Architecture and training disclosure: Kimi K3 technical report v1

Serving

• Minimum vLLM topology and operational notes: vLLM Kimi K3 recipe
• vLLM architecture and cache implementation: Kimi K3 day-zero support
• Published GPU shapes and memory settings: SGLang K3 cookbook
• KDA/MLA memory management: SGLang day-zero report
• Moonshot production recommendation: Kimi K3 launch post

Accelerator memory

• H100 and H200 HBM: NVIDIA H100 and H200
• B200 and B300 HBM: NVIDIA HGX components
• MI355X HBM: AMD Instinct system guide