How 2.8T Parameters Fit in a 1.56 TB Kimi K3 Checkpoint

You reserve 1.4 TB because the model is described as 4-bit. The download finishes at 1.561 TB, and the extra 171 GB looks like unexplained packaging.

Part three tracked the messages a client must replay. This post tracks the bytes a deployment must store before serving—running the trained model to produce responses—can begin.

The shortcut was:

2.8 trillion parameters × 4 bits ≈ 1.4 TB

Then you download Kimi K3 and Hugging Face reports 1.561 TB.

Nothing is wrong with the files. The estimate assumed every model value used four bits, with no scales, no higher-precision modules, no headers, and no repository metadata.

Use a packed-box mental model. Most weights are tiny 4-bit items, but each group needs a scale label, and some larger items remain at higher precision. A checkpoint is the saved weights plus the metadata needed to interpret them. Kimi K3 is a mixture-of-experts (MoE) model: routing assigns each token to specialist feed-forward modules. Most routed expert weights use MXFP4, a grouped 4-bit floating-point format, inside a mixed-precision checkpoint produced through quantization-aware training (QAT), training that exposes the model to its low-precision representation before export.

The released counts support a useful storage reconstruction. They do not, by themselves, prove every tensor shape, scale tensor, alignment byte, or safetensors header in the 96 shards.

The arithmetic below uses the sum of visible files in the pinned revision. Hugging Face's separate usedStorage field does not equal that snapshot manifest, and the API does not explain the difference. The metadata was rechecked on August 3, 2026.

Read the evidence in four buckets:

• Official fact: released parameter counts, files, formats, and byte sizes.
• Derived calculation: arithmetic from those published values.
• Reconstruction estimate: a modeled payload that should reconcile closely with the files but is not a parsed tensor manifest, the inventory of every stored array, shape, dtype, and offset.
• Hypothesis: a possible source of unexplained bytes, kept separate until a tensor-level audit confirms it.

Before using the explorer, compare the released snapshot with the three uniform-precision counterfactuals. Then add group scales and higher-precision modules to see which assumptions close the gap.

After the explorer, leave the unexplained remainder visible. A balanced equation is not the same thing as a tensor manifest, and the widget does not turn a reconstruction estimate into an official fact.

“2.8T” rounds an exact released count

Official fact — released metadata. The pinned Hugging Face model API reports:

2,779,931,837,184 total logical parameters

The metadata divides them into three categories:

The U8 label does not mean “8-bit model.” Reading it as one stored byte per listed item would exceed the shard total. The config, tensor format, and near-exact reconstruction below support using it as the compressed logical weight count for this ledger; only a tensor-manifest audit can prove how every stored U8 array contributes. The released config sets these values under text_config.quantization_config:

{
  "format": "mxfp4-pack-quantized",
  "group_size": 32,
  "num_bits": 4,
  "scale_dtype": "torch.uint8",
  "type": "float"
}

K3 groups 32 compressed MXFP4 weights under one 8-bit scale.

The distinction between logical parameters and stored bytes is the whole article.

Uniform precision gives useful counterfactuals

Derived calculation — counterfactual floor. Before reading the mixed format, calculate three simpler models:

These are storage comparisons, not runnable alternative checkpoints. Moonshot did not release a uniform BF16 K3 artifact.

The BF16 result explains why a full unquantized conversion would need more than 5.5 TB before metadata. The flat 4-bit result explains why “4-bit” gets close to the release. The 171 GB gap between flat 4-bit and the actual repository is where the format and module choices live.

MXFP4 uses 17 bytes for every 32 compressed weights

Picture 32 small weights sharing one label. Packing two 4-bit weights into each byte uses 16 bytes. The shared 8-bit scale adds one more byte.

The vLLM compressed-tensor implementation stores:

• two 4-bit values in each byte;
• one 8-bit E8M0 scale per group of 32 values.

For one group:

32 weights × 4 bits = 128 bits = 16 packed bytes
+ 1 scale byte
= 17 bytes per 32 weights

That is:

17 bytes × 8 / 32
= 4.25 stored bits per compressed weight

Apply it to K3's 2,722,740,830,208 compressed parameters:

packed FP4 values
= 2,722,740,830,208 / 2
= 1,361,370,415,104 bytes
 
E8M0 scales
= 2,722,740,830,208 / 32
= 85,085,650,944 bytes
 
compressed payload
= 1,446,456,066,048 bytes

The scale overhead alone is about 85.1 GB.

“Divide by two” undercounts a grouped 4-bit format. The values use four bits; the groups still need scales.

Microsoft's microxcaling reference implementation documents and emulates the broader OCP MX family. The model config and vLLM code establish the specific K3 packing choices used in this byte reconstruction.

Higher-precision modules add another 114.4 GB

K3 leaves 57,179,884,544 parameters in BF16:

57,179,884,544 × 2 bytes
= 114,359,769,088 bytes

The F32 payload adds:

11,122,432 × 4 bytes
= 44,489,728 bytes

Now add the tensor payload:

1,446,456,066,048 compressed bytes
+ 114,359,769,088 BF16 bytes
+      44,489,728 F32 bytes
= 1,560,860,324,864 bytes

A safetensors shard is one file that stores a subset of the model's named arrays. The 96 visible shards occupy 1,560,936,091,448 bytes. The difference, 75,766,584 bytes, is the part this simplified logical-parameter ledger does not explain.

Hypotheses, not confirmed categories: safetensors headers, tensor alignment or padding, scale-tensor shape details, and mismatches between reported logical counts and stored arrays can contribute to that remainder.

The repository then adds 62,892,942 bytes outside the weight shards, including the unusually large safetensors index and model code:

1,560,936,091,448 visible weight-shard bytes
+      62,892,942 other repository bytes
= 1,560,998,984,390 visible snapshot bytes

Worked byte ledger:

The arithmetic reaches the repository total only after treating that shard remainder as unresolved overhead. A reproducible tensor-manifest audit must parse every tensor shape, dtype, offset, scale tensor, and header before calling the split exact.

The same API response reports:

usedStorage = 1,561,018,243,668 bytes
 
usedStorage - visible snapshot
= 19,259,278 bytes

The API does not explain the difference. Do not add it to a download plan: it is not represented by a visible file in the pinned revision, so the static evidence does not identify it as model payload.

The quantizer intentionally ignores selected modules

The junior takeaway is simple: “4-bit checkpoint” does not mean “every module is 4-bit.”

Optional artifact depth: the patterns below explain which module families the quantizer skips. The released BF16 category total is enough for the byte ledger; proving each tensor requires the full manifest audit.

K3's quantization config excludes patterns for:

.*self_attn.*
.*shared_experts.*
.*mlp\.(gate|up|gate_up|down)_proj.*
.*lm_head.*
.*vision_tower.*
.*mm_projector.*

These expressions need context.

The config targets Linear modules for compression, then excludes attention, shared experts, selected dense MLP paths, the language-model head, the vision tower, and the multimodal projector. The expert-specific compressed path is therefore not the entire model.

This also resolves an apparent contradiction in config.json:

{
  "dtype": "bfloat16",
  "text_config": {
    "dtype": "bfloat16",
    "quantization_config": {
      "format": "mxfp4-pack-quantized"
    }
  }
}

The top-level dtype describes the model's default tensor dtype and higher-precision paths. The quantization section describes compressed target modules. “BF16 model config” and “MXFP4 checkpoint” can both be true.

Do not infer that every ignored regex maps one-to-one to the 57.18B BF16 count without enumerating tensors. The artifact metadata gives the exact category total; the patterns explain the intended scope.

The released average is about 4.492 bits per logical parameter

Divide the visible snapshot by the logical parameter count:

1,560,998,984,390 bytes × 8
/ 2,779,931,837,184 parameters
= 4.49219 bits per logical parameter

That figure includes:

• packed FP4 values;
• group scales;
• BF16 and F32 tensors;
• shard headers;
• the index and repository files.

It is an artifact-level average, not the arithmetic precision used by every kernel.

The technical report says routed expert weights use MXFP4 and expert input activations use MXFP8 during quantization-aware post-training. At runtime, an inference engine schedules model execution, and its precision choices can differ by hardware:

• one engine may execute W4A8;
• another may use W4A16;
• attention and shared paths remain at higher precision;
• key-value (KV) cache precision—the format of per-request attention state—is a separate serving choice.

Checkpoint storage, activation precision, and cache precision are different dimensions.

Quantization-aware training started before export

K3's quantization is not described as a final offline conversion. Moonshot says quantization-aware training (QAT), training that simulates deployment quantization during optimization, starts with supervised fine-tuning and continues through reinforcement learning.

The stated objective is to avoid a mismatch where:

1. the policy learns through high-precision rollouts;
2. the final export changes numerical behavior;
3. agent trajectories degrade after quantization.

The released checkpoint confirms the resulting MXFP4 module layout. It cannot independently prove how much quality QAT preserved. Moonshot does not publish a full high-precision K3 checkpoint or an independently reproducible comparison.

Treat “QAT was used” as an official training disclosure and “QAT preserved quality by X” as unsupported unless a cited experiment supplies X.

The release contains no optimizer state

Optional training depth: serving the checkpoint does not require the optimizer ledger below. It matters when estimating what a comparable training job would own.

The downloadable snapshot contains inference weights and model metadata. It does not contain the optimizer state used to train K3.

The technical report discloses Muon for matrix parameters, Per-Head Muon for query/key/value projections, a cosine learning-rate schedule, 1% warmup, weight decay of 0.1, and Kimi K2's weight-clipping mechanism. It does not publish the full matrix/non-matrix optimizer split, momentum dtype and coefficients, Newton-Schulz settings, full-precision weight-copy policy, or state-sharding layout.

Counterfactual full-parameter ledgers show why that omission matters:

These are reproducible planning scenarios, not K3's measured training memory. Muon workspaces, QAT representations, activations, communication buffers, and allocator headroom sit outside the table. Sharding optimizer or tensor state can divide resident bytes per rank, one distributed worker process usually attached to one GPU; it does not change the aggregate state that the training job owns.

Active parameters cannot be derived from the expert ratio

K3 selects 16 of 896 routed experts:

16 / 896 = 1 / 56

Multiplying the entire 2.7799T model by that ratio gives about 49.6B. Moonshot reports 104.2B active parameters.

The shortcut fails because only routed-expert tensors follow the selection ratio. Every token also uses:

• attention and recurrent state updates;
• embeddings and output projections;
• routers and latent projections;
• two shared experts;
• one dense layer;
• normalization;
• multimodal components when media is present.

The 16-of-896 ratio is a routing fact. The 104.2B figure is a reported whole-model execution count.

Neither number tells you the stored checkpoint size. Inactive experts remain stored because another token may route to them. The hardware post starts from 1.561 TB rather than 104B.

The 59.8 MB index is part of a 96-shard execution map

Large safetensors checkpoints include an index that maps tensor names to shard files. K3's model.safetensors.index.json is about 59.8 MB.

That size is not weight data. It is consistent with a model that has many named tensors across 93 layers and 896 experts, but file size alone does not prove how many tensor entries or bytes each structure contributes.

The index matters to operations because:

• loaders need it before they know which shard owns a tensor;
• a damaged or mismatched index can make valid shard files unusable;
• artifact verification includes more than counting .safetensors files;
• copying the repository without all code and config files can break custom loading.

The Hugging Face repository uses custom model code. Loading through a generic transformer path can require trust_remote_code=True, while production engines carry their own optimized K3 implementation.

A checkpoint is a set of weights plus the metadata needed to interpret them.

Disk planning depends on how your artifact client stores snapshots

The verified physical floor for one visible snapshot is:

1,560,998,984,390 bytes

The required free disk can be higher:

• a Git LFS working tree may coexist with an object cache;
• a container build can copy the checkpoint into another layer;
• a conversion job can create a second output artifact;
• a rollout can keep old and new revisions side by side;
• a failed partial download can leave temporary files;
• an engine may build local indexes or kernel caches.

It can also be close to one copy when the snapshot uses links into a shared cache.

Therefore, “you need exactly 3.2 TB of disk” would be another invented universal. Plan disk by:

1. identify the artifact client's cache layout;
2. determine whether snapshots link or copy;
3. record conversion outputs;
4. retain rollback copies intentionally;
5. measure peak disk during a real deployment.

Host RAM follows a similar rule. The model occupies 1.56 TB on disk, but a loader can stream or memory-map shards. Do not claim that 1.56 TB of host RAM is always mandatory.

What the checkpoint cannot tell you

The released files expose model structure. They do not disclose the training ledger.

You cannot derive:

• total pre-training tokens;
• data cutoff;
• dataset names or weights;
• language mix;
• text, image, and video proportions;
• contamination results;
• training FLOPs;
• hardware fleet;
• wall-clock training time;
• energy or cost;
• stage-by-stage context token allocation;
• safety-training volume.

The Kimi K3 report describes data domains, optimizer families, parallelism, post-training stages, and the context curriculum. It does not supply the missing totals.

File size is not a proxy for training compute. Active parameters are not a proxy for data volume. Parameter count is not a cutoff date.

What could go wrong

• A loader treats U8 metadata as an ordinary 8-bit checkpoint. The config requires the MXFP4 compressed-tensor scheme and group scales.
• A conversion doubles peak disk. Source and destination artifacts coexist during the job.
• A generic transformer runtime loads reference code but misses production kernels. Semantic compatibility does not imply acceptable throughput, useful model work completed per second.
• The top-level BF16 dtype is mistaken for uniform BF16 storage. The quantization section and safetensors metadata contradict that reading.
• The checkpoint fits aggregate HBM but fails per-rank placement. Dense modules, caches, and communication buffers do not necessarily shard evenly.
• The active parameter count is used to size storage. Sparse execution does not remove expert weights.

The safest byte calculation is the one you can reconcile against the files.

Kimi K3 fits in 1.56 TB because most routed weights use grouped MXFP4, not because all 2.8T parameters became a flat 4-bit array.

The final Engineering Guide post turns these artifact and hardware facts into a decision model for comparing the Kimi API with a measured self-hosted deployment.

Sources and References

Released artifact

• Exact parameters, files, shards, and repository size: Pinned Hugging Face model API
• Quantization scope and group size: Pinned Kimi K3 config
• Model summary and deployment links: Pinned Kimi K3 model card

Format and training

• K3 architecture and QAT disclosure: Kimi K3 technical report v1
• MXFP4 packing implementation: vLLM compressed-tensor MXFP4 scheme
• Microscaling reference implementation: Microsoft microxcaling

Deployment context

• Supported vLLM topology: vLLM K3 recipe
• Published SGLang shapes: SGLang K3 cookbook