Inside Kimi K3: How KDA, AttnRes, and 896 Experts Work

You open Kimi K3's serving configuration—the runtime description used to load and execute the model—expecting identical transformer blocks. By layer four, that picture no longer works.

Part one counted the memory. This part explains which mechanisms create its fixed state, token-growing cache, and expert communication.

Start with three directions instead. A token can gather information from earlier tokens, from earlier model depth, and from specialist feed-forward networks. K3 assigns a different mechanism to each direction:

• Kimi Delta Attention (KDA) keeps a fixed recurrent matrix instead of appending every prior token to a key-value (KV) cache, the per-request attention history.
• Multi-Head Latent Attention (MLA) periodically restores global token interaction through a compressed latent cache.
• Attention Residuals (AttnRes) let a token select representations from earlier depth blocks instead of accepting only the previous layer.
• Stable LatentMoE is a mixture-of-experts (MoE) feed-forward path. MoE routing assigns each token to 16 of 896 specialist networks, while two shared experts run for every token.

Its 93 layers do not use one attention mechanism. Its residual path does not only add the previous layer. Its 896 routed experts do not operate at the 7,168-dimensional backbone width. Even the phrase “16 active experts” omits two shared experts that run for every token.

Kimi K3 scales information flow along three axes: KDA and MLA move information across sequence length, Attention Residuals select representations across depth, and Stable LatentMoE distributes computation across expert width.

Here is one hypothetical token journey. The route IDs are illustrative; the released artifacts do not tell us that one expert owns one human-readable skill.

The evidence labels matter:

• Official fact: released config, code, or report text.
• Derived calculation: arithmetic from those released values.
• Estimate: a memory or serving approximation with stated assumptions.
• Reported result: Moonshot's measurement without independent reproduction.
• Interpretation or hypothesis: a stated reading that the artifacts do not isolate causally.

The pinned artifacts and technical report v1 were checked on August 3, 2026.

Use the explorer in three passes. First compare KDA and MLA across the layer schedule. Then move the depth slider to see which representation blocks become eligible. Finally switch among Route A, B, and C: the changing expert IDs are schematic, not evidence that experts learned semantic jobs.

After the explorer, keep its three axes but discard any imagined expert personality. The exact schedule and dimensions below are official facts; the animated routes are teaching aids.

The exact backbone is 23 hybrid groups plus one final MLA layer

Official fact — released config. The released configuration defines 93 transformer layers. The attention schedule is:

23 × [KDA, KDA, KDA, Gated MLA]
+ 1 final Gated MLA

That produces 69 KDA layers and 24 MLA layers.

The 1-indexed MLA layers are:

4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48,
52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 93

Every other layer from one through 92 uses KDA. Layer 93 is the standalone MLA layer, so “a perfect 3:1 ratio” is a useful approximation but not the exact topology.

The feed-forward schedule has a second exception. The config sets first_k_dense_replace=1: layer one is dense, while layers two through 93 use Stable LatentMoE.

These details matter because each layer type creates different state, communication, and kernel requirements. A serving engine cannot treat K3 as a standard transformer with a larger num_hidden_layers.

KDA writes corrections into a recurrent matrix

Think of KDA as an erasable association notebook. A normal attention layer keeps adding pages for prior tokens. KDA keeps one fixed-size page: it reads the current association, measures the error, and writes a correction.

Softmax attention stores keys and values for prior tokens in the KV cache, then compares each new query with those cached keys. KDA keeps a matrix state instead.

For one head, the technical report expresses the update as:

S_t = (I - beta_t k_t k_t^T) Diag(alpha_t) S_(t-1)
      + beta_t k_t v_t^T
 
output_t = S_t^T q_t

The public Flash Linear Attention reference makes the intuition clearer:

1. decay the old state
2. read what the state currently predicts for k_t
3. compute error = v_t - prediction
4. write beta_t × k_t × error into the state
5. read the updated state with q_t

This is a delta rule, an update that writes the difference between the desired value and the state's current prediction. If the state already maps the key to the right value, the correction is small. If the mapping is wrong, the update erases part of the old association and writes a new one.

Derived calculation — fixed state. K3 uses 96 heads with 128-dimensional keys and values. Each head therefore holds a 128 × 128 recurrent matrix. The state size does not grow with sequence length:

96 heads × 128 × 128
= 1,572,864 state elements per KDA layer

That fixed-size property is the main reason KDA helps at long context. It is also easy to overstate.

K3 is not constant-memory overall. The model still has 24 MLA layers with a token-growing cache, and each concurrent request needs its own KDA state plus short-convolution state. The hardware analysis in part one shows how those two pools create separate concurrency limits.

The lower bound on decay is a hardware decision

Optional kernel depth: if you only need the architecture mental model, skip to the short-convolution section. This section explains why a numeric bound appears in the model design.

KDA derives a channel-wise retention value from a log-decay g:

g = g_min × sigmoid(...)
alpha = exp(g)
g_min = -5

Therefore:

g is between -5 and 0
alpha is between exp(-5) and 1
exp(-5) ≈ 0.0067

The lower bound applies to g, not directly to alpha.

Why put a floor on forgetting?

The report groups work into 16-token secondary tiles. Without a bound, a cumulative decay can become too small to rescale safely in BF16. With g_min=-5, the sum across a 16-token tile remains above -80. Moonshot says that range lets both diagonal and off-diagonal causal tiles use dense Tensor Core matrix multiplications instead of a slower special path.

The technical report specifies the 16-token secondary tiles. The pinned FlashKDA interface confirms BF16 query/key/value/gate inputs, FP32 gate parameters, and the -5.0 lower-bound range. Reported speed and accuracy benefits remain first-party measurements. Public artifacts verify the mechanism and bound; they do not independently prove that -5 is globally optimal.

This is a recurring K3 pattern: an architecture choice is also a kernel choice. The mathematical operation, numeric range, and target hardware were designed together.

Short convolutions give KDA local order before recurrence

Before KDA updates the matrix, the released reference implementation applies separate short convolutions to query, key, and value projections. The kernel size is four.

The path is approximately:

x_t
 ├─ W_q → four-token convolution → Swish → L2 normalize → q_t
 ├─ W_k → four-token convolution → Swish → L2 normalize → k_t
 └─ W_v → four-token convolution → Swish                → v_t

The convolution exposes the recurrent update to a small ordered window. KDA then carries information forward through its matrix state.

Calling KDA “linear attention” hides these mechanics. Linear sequence scaling does not mean the layer is position-blind or a simple summation of keys and values. The convolution, decay, write strength, delta correction, and output gate all shape the update.

Periodic Gated MLA restores global token interaction

K3 does not replace every full-attention layer. Every fourth layer, plus the last layer, uses Gated MLA.

The plain-language role is a periodic full-room conversation. KDA carries a fixed summary forward; MLA lets the current token interact with the full stored sequence again.

In the table, low-rank adaptation (LoRA) rank and KV latent rank are compressed projection widths. They are model dimensions, not distributed ranks, worker processes usually attached to GPUs.

The released dimensions are:

The config retains a legacy field named qk_rope_head_dim, but it also sets mla_use_nope=true. The released implementation does not construct or apply rotary embeddings in MLA.

NoPE means the MLA layer does not apply an explicit positional embedding. It does not mean the 64 channels vanish. KDA's intervening recurrent and convolutional computation gives the MLA layer order-sensitive inputs. That is an interpretation, not an isolated causal result from Moonshot; the public artifacts do not measure how much each mechanism contributes.

MLA also applies a full-rank output gate:

output = W_o [sigmoid(W_g x) elementwise_mul attention_result]

That gate controls output channels after attention. It is not another token attention matrix.

The simple Hugging Face reference expands and appends per-head key/value tensors. Production engines such as vLLM and SGLang reorder projections and cache the compact latent representation. Reference correctness and production memory efficiency are different implementation goals.

Attention Residuals let a token choose depth

An ordinary residual stack is a relay race: each layer receives the baton from the layer immediately below. AttnRes adds a shelf of earlier snapshots and lets the current token mix the completed snapshots it finds useful.

An ordinary residual stack carries:

h_l = h_(l-1) + F_l(h_(l-1))

Every layer receives the accumulated result from the layer immediately below it. Earlier representations survive only through repeated addition.

Attention Residuals (AttnRes) changes the depth path. A learned pseudo-query scores normalized representations from prior depths, then takes a weighted sum:

score_(i→l) = w_l^T RMSNorm(v_i)
alpha_(i→l) = softmax_i(score_(i→l))
h_l = sum_i alpha_(i→l) v_i

The Attention Residuals repository and K3 reference implementation confirm this structure.

The pseudo-query is learned and fixed after training. The normalized representations depend on the current token, so the resulting depth weights remain token-dependent.

K3 uses Block AttnRes. It groups up to 12 transformer layers:

• seven complete 12-layer blocks;
• one final nine-layer block;
• the embedding as an additional source;
• ordinary summation inside the current partial block;
• learned selection across completed blocks and the embedding.

At the top of the model, a token can draw from as many as nine stored representations: the embedding plus eight depth blocks.

AttnRes does not attend over other token positions. KDA and MLA handle the sequence axis. AttnRes attends over representations at different depths for the same token.

Block storage changes the residual-bank memory from one representation per layer toward one representation per block. Moonshot reports the asymptotic change as O(Ld) to O(Nd), where L is layer count and N is block count. The public code confirms the block form. The quality benefit remains an official training result.

Stable LatentMoE routes at half the backbone width

Treat the MoE path as a switchboard. The model first compresses the token from 7,168 values to 3,584, sends that smaller representation to 16 routed experts, adds two always-on shared experts, then expands the result back to the backbone width.

K3's expert path begins with a projection:

x in R^7168
    ↓
latent projection
    ↓
z in R^3584
    ↓
16 selected routed experts, each with 3072-wide SiTU-GLU
    ↓
weighted sum + RMSNorm
    ↓
projection back to R^7168

Two shared experts run alongside the routed path at full width.

The model config and sparse block implementation verify:

• 896 routed experts;
• 16 selected per token;
• two shared experts;
• 3,584 routed latent width;
• 3,072 expert intermediate width;
• sigmoid router scores;
• renormalization of selected scores;
• a correction bias used for selection but not mixture weight.

Derived calculation — routing sparsity. The routing ratio is:

896 / 16 = 56

That is not an active-parameter ratio for the whole model. Attention, latent projections, routers, embeddings, shared experts, and other dense components remain active. Moonshot reports 104.2B active parameters in the technical report.

The word “Stable” refers to interventions for two scale problems described in the report:

1. a long chain of routed matrix multiplications can produce activation outliers;
2. balancing nearly 1,000 experts becomes difficult.

K3 addresses the first with post-aggregation RMSNorm and a bounded activation. It addresses the second with Quantile Balancing and a separate expert-placement system.

SiTU-GLU caps the routed multiplicative path

Optional numeric depth: this section explains the activation bound. The practical takeaway is that K3 limits extreme values inside routed experts.

K3 replaces SwiGLU in the routed experts with Sigmoid Tanh Unit GLU (SiTU-GLU):

gate(a) = beta1 × tanh(a / beta1) × sigmoid(a)
up(b)   = beta2 × tanh(b / beta2)
 
SiTU-GLU(a, b) = gate(a) × up(b)

The released constants are:

beta1 = 4
beta2 = 25

The product is therefore bounded in magnitude by 100. Near the origin, the shape behaves more like an ordinary gated activation. At large magnitudes, the tanh terms cap both branches.

The implementation contains those exact equations and constants. The trade-off is familiar: bounds suppress outliers but also saturate gradients for very large coordinates. Moonshot did not release an independent ablation that proves 4 and 25 are universally best.

Quantile Balancing and MoonEP solve different imbalance layers

These two mechanisms are easy to collapse into one phrase. They operate at different times.

Quantile Balancing changes tomorrow's routing tendency. MoonEP places today's already-routed work. One adjusts the decision; the other schedules its execution.

Quantile Balancing changes future routing decisions

K3 uses auxiliary-loss-free routing. “No auxiliary loss” does not mean “no balancing.”

The router computes sigmoid scores, adds a non-gradient correction bias for Top-16 selection, then uses the original selected scores for normalized mixture weights.

Quantile Balancing estimates how much bias each expert needs from the distribution of router-score margins. The update targets equal expected load on the next training step. Moonshot says global histograms and one all-reduce avoid gathering every token's margin.

The final correction bias is frozen for inference. The released checkpoint contains that bias. The training-time histogram procedure appears in the report, not in a released training pipeline.

MoonEP changes where current expert work executes

Expert parallelism (EP) distributes expert weights across ranks. Even with a balanced router, one batch can send more work to some ranks than others.

MoonEP accepts the current routing decision, replicates hot experts where needed, and places tokens so every rank receives the same routed-token count. Its public code confirms the planning and dispatch mechanism.

The difference is:

Moonshot's speed and memory comparisons for MoonEP are first-party results on its reported hardware. The planning algorithm is inspectable; the production performance claim is not an independent reproduction.

Per-Head Muon separates attention-head updates

Optional training depth: serving readers can skip this section. It describes how Moonshot says K3 updated attention matrices during training, not a runtime requirement.

The report says K3 applies Muon, an optimizer that orthogonalizes matrix updates with Newton-Schulz iterations, to matrix parameters. For query, key, and value projections, it orthogonalizes each attention head separately:

standard Muon:
orthogonalize(concatenated head matrix)
 
Per-Head Muon:
concatenate(orthogonalize(head_1), ..., orthogonalize(head_H))

Moonshot's stated motivation is to stop large-scale heads from dominating one shared matrix update and to reduce some orthogonalization work.

The report also discloses a cosine learning-rate schedule, 1% linear warmup, weight decay of 0.1, and the weight-clipping mechanism introduced for Kimi K2. It does not disclose the optimizer used for every non-matrix parameter, momentum precision and coefficients, Newton-Schulz iteration count and polynomial, state-sharding layout, or a per-head ablation table. The public evidence supports “Moonshot reports this training recipe,” not “the community can recreate the run.”

For a smaller implementation of Muon and its matrix math, see the NanoChat Muon post.

Context and vision were trained into the shared backbone

Moonshot reports a progressive context curriculum:

8K → 64K → 256K → 1M

The first two stages belong to pre-training; 256K and 1M appear during cooldown. NoPE avoids a separate RoPE-rescaling stage.

The report does not disclose how many tokens each stage consumed. Do not turn the sequence into a pie chart or claim that K3 trained at 1M throughout.

K3 also includes a 401M-parameter MoonViT-V2 encoder:

The released vision config confirms those dimensions. Moonshot reports that the encoder was trained from scratch with next-token prediction and shares parameters for images and video. The claim that this matched a contrastively initialized baseline comes from Moonshot's internal experiment.

Model capability and serving-interface capability are separate. The hosted API documents video input, while the model card's summary table says “Text, Image” and the launch-day SGLang processor accepts images but rejects video and audio. Treat video as a documented hosted capability, not proof that every self-hosted processor implements the same media path. Part three covers that boundary.

Nine RL teachers become one low/high/max model

Moonshot describes this post-training sequence:

supervised fine-tuning
    ↓
three task domains × three reasoning efforts
    ↓
nine reinforcement-learning teachers
    ↓
Multi-Teacher On-Policy Distillation
    ↓
one model with low, high, and max effort

The three domains are general tasks, general agents, and coding agents. Each domain has low, high, and max effort.

Multi-Teacher On-Policy Distillation (MOPD) trains the unified model from the matching teacher's token probabilities on student-generated trajectories. Moonshot reports that a more detailed Top-K objective did not produce a clear benefit, but it does not publish the numerical comparison.

Quantization-aware training spans supervised fine-tuning and reinforcement learning. It is not a final export step. Routed expert weights use MXFP4, a grouped 4-bit floating-point weight format, while selected modules remain at higher precision.

That training/inference match matters because a policy optimized in one numeric regime can behave differently after a separate low-precision conversion. K3's process aims to expose the policy to the deployment format during post-training. The reported quality benefit remains a Moonshot claim.

The architecture is inspectable; the training run is not reproducible

Released artifacts support precise statements about:

• layer schedule and dimensions;
• KDA recurrence and lower bound;
• Gated MLA and NoPE behavior;
• AttnRes block storage and aggregation;
• expert counts, latent dimensions, routing, and SiTU;
• frozen correction biases;
• checkpoint quantization;
• tokenizer and media-control code.

The public release does not include:

• total pre-training tokens;
• exact dataset inventory, weights, licenses, or cutoff date;
• language and modality proportions;
• contamination analysis;
• training FLOPs, duration, accelerator fleet, utilization, energy, or cost;
• global batch size or peak learning rate;
• actual context-stage token allocation;
• Muon settings;
• SFT volume;
• RL rollout count and compute;
• teacher checkpoint details;
• complete safety-training data;
• runnable pre-training code.

The headline 2.5× scaling-efficiency improvement over K2 is a fitted Moonshot claim. The underlying checkpoint sweep and compute ledger are not public.

That does not make the architecture uninteresting. It defines the evidence boundary.

What could go wrong

• Fixed KDA state can cap short-request concurrency. The state does not grow with tokens, but every concurrent request needs several large state slots.
• MLA still grows with context. K3 is hybrid, not cache-free.
• Prefix caching requires both state types to agree. The vLLM K3 implementation had to add snapshot and copy-on-write behavior for recurrent state.
• Speculative rejection cannot roll back KDA like append-only KV. SGLang's ReplaySSM design replays accepted inputs instead of storing a full state after every draft step.
• The Hugging Face reference is not a production engine. Its sparse expert path uses inference-only logic, and its MLA cache is not the compact serving representation.
• The model can act too proactively under ambiguity. Moonshot lists this as a limitation in the launch post.

K3's architecture is not one trick. It is a set of coordinated choices that move bottlenecks between computation, memory, numeric range, routing, and communication.

The useful mental model is three-dimensional: recurrence across tokens, selection across depth, and sparsity across experts.

Part three turns from model internals to the client contract: a production Kimi K3 integration must preserve reasoning, tool calls, and cache-sensitive history.

Sources and References

Model architecture

• Architecture, training, and evaluation disclosure: Kimi K3 technical report v1
• Exact released dimensions and layer schedule: Kimi K3 config
• KDA, MLA, AttnRes, MoE, and SiTU implementation: Kimi K3 reference model
• Model summary and active parameters: Pinned model card

Component implementations

• Attention Residuals: MoonshotAI/Attention-Residuals
• KDA reference operation: Flash Linear Attention KDA
• KDA production kernels: MoonshotAI/FlashKDA
• Expert placement and dispatch: MoonshotAI/MoonEP

Serving implications

• Hybrid cache and prefix-state support: vLLM Kimi K3 release
• KDA/MLA pools and ReplaySSM: SGLang Kimi K3 report
• Official claims and limitations: Moonshot Kimi K3 launch post