Kimi K3 Ideas Need Experiments Before Design Rules

You read an architecture report, open an issue, and start copying mechanisms into a smaller model before deciding what result would count as success.

The cost post ended with a decision rule: measure before comparing. The lab applies the same rule to architecture claims.

Use a recipe mental model. K3 changed many ingredients and the kitchen at the same time. Copying one ingredient into a smaller model does not tell you whether that ingredient caused the final result.

The series shorthand stays compact here:

• Kimi Delta Attention (KDA) carries prior-token information in fixed-size recurrent state.
• Multi-Head Latent Attention (MLA) periodically restores global attention through a compressed key-value (KV) cache, stored per-request attention history.
• Attention Residuals (AttnRes) let a token select representations from earlier model depth.
• A mixture-of-experts (MoE) layer uses routing to assign each token to a subset of specialist feed-forward networks.

Now copy one mechanism, bound one activation, or orthogonalize each attention head separately. If the 2.8T-parameter model used it, should the smaller model improve too?

That conclusion does not follow.

Kimi K3 changed architecture, optimizer, data, training scale, context curriculum, post-training, quantization, kernels, and distributed systems at the same time. The technical report explains the design but does not release the controlled experiments needed to isolate every intervention outside Moonshot's training environment.

The Kimi K3 Lab exists to test mechanisms before turning them into design rules. It will publish experiment outlines for review, freeze the design before execution, then release code, raw measurements, failures, and conclusions that stay inside the evidence.

This page is the experiment backlog. It contains no claimed result. Read each statement by label:

Official K3 facts and API contracts were rechecked on August 3, 2026.

Before using the planner, pick the conclusion you want to publish. Then select an experiment and inspect whether its evidence class can support that conclusion.

After the planner, compare the desired conclusion with the displayed transfer limit. If the conclusion is broader, redesign the experiment. The planner is an index; the protocol text below is authoritative for sample counts, repetitions, stopping rules, and invalid-run handling.

The lab has three evidence classes

Not every experiment can answer the same question.

Serving means running a trained model to answer requests. An ablation changes one mechanism while holding the comparison as constant as possible. The evidence class tells us whether we observe K3 during serving, ablate a smaller proxy model, or simulate a systems policy.

The separation is not academic.

A one-million-token API test can tell us whether K3 retrieves the right evidence in a constructed document. It cannot tell us whether KDA caused that success.

A 120M-parameter KDA model can tell us whether recurrent state reduces memory in that implementation. It cannot prove that the same quality/compute frontier holds at 2.8T parameters.

A memory allocator simulator can tell us when fixed pools strand bytes. It cannot produce an end-to-end tokens-per-second claim.

Every future lab post will show its evidence class near the top.

Direct tests ask what K3 does today

The first two experiments use the Kimi K3 API. They do not train a new model. They test two claims that directly affect production users.

Experiment one: useful context may end before the configured limit

K3 supports a 1,048,576-token context according to the official pricing and model page and released config. The report says context training progressed through 8K → 64K → 256K → 1M.

Supported length is not the same as useful recall.

The direct experiment will use several fixture families:

1. Single retrieval. One exact fact appears at controlled depth.
2. Distractor retrieval. Similar facts compete with the target.
3. Multi-hop retrieval. The answer requires evidence from distant sections.
4. Order-sensitive state. The answer changes when events are reordered.
5. Repository task. A permissively licensed code snapshot requires tracing definitions across files.

Hypothesis. At least one fixture family will lose validator pass rate as required evidence moves farther apart, before the configured 1M-token limit.

Controlled variables. Each fixture runs at 32K, 128K, 256K, 512K, and a 1,048,576-token total request budget. Prompt tokens, message framing, and max_completion_tokens must sum to no more than that budget; the 1M arm therefore reserves completion capacity instead of sending a full 1M-token prompt. The hosted model ID, 24-hour test window, reasoning effort, question wording, answer validator, evidence-placement rule, and output limit remain fixed. Cold-prefix and warm-prefix runs form separate strata; they are never averaged together.

Primary metrics:

• exact-match or deterministic validator pass rate;
• cited evidence coverage;
• time to first token;
• total latency, elapsed time from request to terminal response;
• prompt, cached, and completion tokens;
• cost per successful task.

The API caching guide requires special treatment. Warm-prefix and cold-prefix runs answer different questions, so results must report both rather than averaging them together. Give every scored cell a retained prefix_id. A warm cell reuses its priming call's exact prefix. Each cold repetition uses a fresh, semantically neutral cache-buster line and a prefix ID never used elsewhere. Accept a cold measurement only when usage.cached_tokens == 0.

Stopping rule. Freeze the call ledger before the first result:

scored calls
= 8 fixtures × 5 families × 5 lengths × 2 cache strata × 3 repetitions
= 1,200
 
warm-prefix priming calls
= 8 fixtures × 5 families × 5 lengths × 1 warm stratum × 3 repetitions
= 600
 
planned total before transport retries = 1,800 calls

Count priming tokens and cost. Accept a warm measurement only when the response reports usage.cached_tokens > 0; otherwise mark that cell invalid. Retry transport failures at most twice with the same request. A cold response with non-zero cached tokens is also invalid. If more than 5% of scored cells are invalid, stop the test window and rerun the complete matrix later. Do not add fixtures after inspecting accuracy.

Interpretation boundary. A successful experiment can say:

The Kimi K3 API in test window X retained Y behavior on these fixtures through context length Z under these settings.

It cannot say:

Kimi K3 has Z useful tokens for every workload.

The hosted API may not expose a weight revision. Record the response model field, request IDs, UTC timestamps, and any platform version identifier. A backend revision is the provider's deployed model-and-runtime version. If none is available, report the test window as an uncontrolled source of drift.

Experiment two: exact message replay should improve agent reliability

The K3 quickstart requires the client to replay complete assistant messages, including reasoning and tool calls. The tool guidance requires one matching result per tool-call ID.

Part three of this series explains the production protocol. The lab will test the operational impact with deterministic local tools and hidden validators.

Hypothesis. Complete message replay and exact tool-result correlation will produce more validated task completions and fewer duplicate mutations than histories with one deliberate protocol fault.

Control and interventions. The control is the documented protocol. Interventions intentionally introduce one fault:

• keep only assistant content;
• change reasoning effort midway;
• omit one of two tool results;
• truncate the completion;
• retry a mutating tool without an idempotency key;
• reorder a dynamic tool declaration.

Controlled variables. Keep the model ID, 24-hour test window, task order, tool schemas, deterministic tool results, maximum of eight model turns, completion limit, and validators fixed. Reasoning effort changes only in its named fault arm. The no-idempotency arm uses a resettable local ledger and injects exactly one deliberate duplicate dispatch; the fixture resets after the run so the fault cannot leak into another task.

Primary metrics:

• task success;
• duplicate mutations;
• unexpected tool names;
• unmatched tool calls;
• retries and error classes;
• cached-token ratio;
• cost per completed task.

The experiment will not publish raw reasoning_content. Retained data can record token counts, hashes, field presence, transitions, and validator outcomes without turning returned reasoning into article material.

The exact symptom after dropping reasoning history is not documented. The protocol post says so. The lab will report observed outcomes by API version without pretending they are a permanent contract.

Stopping rule. Freeze 24 tasks, one control plus six fault arms, and three repetitions: 504 planned agent runs. Stop every run after eight model turns or one validated terminal answer. Retry platform transport failures at most twice, but a transport retry must resume from the retained operation record and must not execute a mutation again. The no-idempotency fault arm is the only controlled exception: it injects one duplicate dispatch against resettable fixture state. If one arm exceeds 5% invalid runs, invalidate the whole test window rather than dropping that arm.

Interpretation boundary. The result can measure the reliability cost of these faults for the tested API window and fixtures. It cannot prove one permanent server error for a contract violation, expose causal internals, or justify publishing raw reasoning_content.

Proxy models ask whether the mechanism survives isolation

Five experiments move K3 ideas into smaller models. They are more causal than an API test because the intervention can be isolated. They also carry greater transfer risk.

Experiment three: hybrid KDA/MLA versus full attention

The plain comparison is three memory strategies: keep token-by-token attention history everywhere, keep only recurrent state everywhere, or alternate the two.

Official fact — K3 schedule. The released layer schedule uses 23 groups of three KDA layers followed by one MLA layer, then one final MLA layer. The FlashKDA repository provides the production-oriented KDA implementation, while Flash Linear Attention provides a readable reference path.

The ablation needs at least three models:

Hypothesis. The hybrid arm will retain more long-context retrieval quality than all-KDA while using less inference memory than full attention.

Controlled variables. Hold trainable parameter count at 120M within 0.5%, tokenizer, initialization family, examples and order, 1B-token training budget, optimizer search budget, batch tokens, hardware, and evaluation checkpoints fixed. Use three predeclared seeds per arm.

Primary metrics are not only validation loss:

• peak training and inference memory;
• throughput, tokens processed per second under the stated workload;
• long-context retrieval;
• recurrent-state drift;
• quality after context lengths beyond the training window.

Stopping rule. Train every valid run to exactly 1B tokens and evaluate every 25M tokens. Do not stop a weak arm early. A 72-hour safety timeout marks the run incomplete; it does not turn the partial checkpoint into a comparison. Complete all nine valid runs before reading aggregate results.

Interpretation boundary. The hybrid wins only if it moves the quality/memory frontier, not merely because it uses less memory. Any result applies to this proxy size, data, kernels, and training budget—not K3's 2.8T regime.

Experiment four: ordinary residuals versus Attention Residuals

An ordinary residual path passes one accumulated representation upward. AttnRes keeps earlier depth snapshots available for selection. The experiment asks whether that extra choice repays its memory and compute cost.

The Attention Residuals implementation lets each module select normalized representations from prior depth. K3 uses 12-layer blocks.

Hypothesis. Block AttnRes will recover part of Full AttnRes's quality gain with lower residual-bank memory, but the gain may disappear at NanoChat scale.

Controlled variables. Use at least 36 layers so the proxy contains three complete 12-layer blocks. Keep model width and depth, parameter count, tokenizer, data order, initialization, optimizer search budget, 1B-token budget, batch tokens, hardware, and checkpoint cadence fixed. Pair five seeds across all three arms.

The plain decision rule is “repeat across seeds and repay the systems cost.”

Optional statistical depth: the formal rule below prevents one lucky seed or a tiny quality gain from winning the headline.

Decision rule. The primary comparison is the paired per-seed validation loss at 1B tokens. Claim a gain only when the mean relative improvement over ordinary residuals is at least 0.5% and a paired bootstrap 95% confidence interval excludes zero. Memory and throughput remain co-primary engineering constraints: reject the mechanism if either regresses by more than 10%.

Primary metrics include:

• validation loss and downstream task quality;
• training throughput;
• peak stored residual memory;
• gradient norms by depth;
• depth-attention entropy;
• how often different tokens prefer different blocks;
• whether the learned weights collapse to one recent source.

Stopping rule. Train all 15 valid runs to exactly 1B tokens. A crash or OOM is a recorded arm failure unless it comes from infrastructure shared by every arm; do not silently shrink the batch for one design. Freeze the analysis script before opening final checkpoints.

Interpretation boundary. If AttnRes improves one seed and loses on the others, it did not establish a reliable gain. The result tests depth selection in this small model; it does not validate Moonshot's reported K3-scale improvement.

Experiment five: Quantile Balancing versus other router controls

K3 uses 896 routed experts and selects 16. The report introduces Quantile Balancing, which updates non-gradient correction biases from router-score quantiles.

Think of the router as a dispatcher. The four arms compare no traffic control, a training penalty, a hand-tuned correction, and a correction derived from the observed score distribution.

Hypothesis. Quantile Balancing will reduce load variance and bias oscillation relative to a fixed-step correction without worsening final primary loss by more than 0.5% relative.

Controlled variables. Keep router inputs, expert count, Top-K, token batches and order, capacity factor, optimizer, learning-rate search budget, and 50,000 post-warmup updates fixed. Pair five seeds across all four arms.

Primary metrics:

• expert-load coefficient of variation;
• maximum-to-mean load;
• dropped-token rate;
• primary training loss;
• correction-bias oscillation.

Stopping rule. Run the same 1,000-step warmup and 50,000 measured updates for every arm. Do not stop after the load curve looks flat. Treat dropped tokens, NaNs, and OOMs as outcomes; exclude a run only for a predeclared hardware or logging failure shared across arms.

Interpretation boundary. Balanced routing is not automatically better if it damages model loss. The experiment must plot both. Small-MoE balance does not establish K3-scale network throughput or MoonEP gains.

MoonEP belongs to a different layer: it places already-selected expert work across ranks, distributed worker processes usually attached to GPUs. A later distributed extension can combine the router result with a MoonEP-style planner, but the first ablation should not change both at once.

Experiment six: SiTU-GLU under quantization-aware training

SiTU-GLU is K3's bounded gated activation for routed experts. It limits extreme values in the two multiplied branches.

Quantization-aware training (QAT) exposes the model to simulated low-precision arithmetic during training instead of converting only after training ends.

K3's released implementation uses SiTU-GLU with beta1=4 and beta2=25. The bounded branches cap the product magnitude at 100.

Official fact — reported numeric regime. The technical report says routed weights use MXFP4, grouped 4-bit floating-point values, while expert input activations use MXFP8, grouped 8-bit floating-point values.

The hypothesis is not “bounded is better.” It is:

Bounding both branches may reduce activation tails and low-precision error, while saturation may slow optimization.

Control and intervention. The control is SwiGLU under the same quantization-aware training setup. The intervention replaces it with SiTU-GLU at the released beta1=4 and beta2=25.

Controlled variables. Hold model size, initialization, tokenizer, data order, quantization format, calibration, optimizer search budget, batch tokens, and 500M-token training budget fixed. Pair five seeds across the two arms.

Primary metrics:

• activation p99 and p99.9;
• quantization error;
• loss spikes;
• validation loss;
• downstream task quality;
• time spent in saturation regions.

Stopping rule. Train all 10 valid runs to exactly 500M tokens, with activation histograms sampled at fixed checkpoints. Do not change beta values after seeing tails. A numerical failure remains in the result unless a predeclared logging defect caused it.

Interpretation boundary. If SiTU reduces tails but loses quality, the result is a trade-off, not a win. The experiment tests one proxy QAT stack; it does not reproduce K3's MXFP4/MXFP8 training system.

Experiment seven: Per-Head Muon versus standard Muon

Muon is an optimizer that orthogonalizes matrix updates. The report says K3 applies that operation to query, key, and value momentum separately per attention head. It does not release optimizer code or the settings needed to reproduce Moonshot's training.

That makes the experiment an independent interpretation:

Standard Muon straightens one combined projection update. Per-Head Muon straightens each head's slice separately, then joins the slices again.

standard Muon:
orthogonalize(concatenated projection momentum)
 
Per-Head Muon:
concatenate(orthogonalize(momentum for each head))

The series already has a Muon explainer, making NanoChat a useful proxy environment.

Hypothesis. Per-Head Muon will reduce update-norm dispersion across heads; that mechanical effect will matter only if convergence or end-task quality also improves enough to repay optimizer overhead.

Controlled variables. Keep model, initialization, data order, token budget, batch tokens, hardware, checkpoint cadence, and total hyperparameter search budget fixed. Give AdamW, standard Muon, and Per-Head Muon the same eight-configuration search over two pilot seeds.

Primary metrics:

• update-norm dispersion across heads;
• optimizer step time;
• training and validation loss;
• head diversity;
• end-task quality.

Stopping rule. Freeze each arm's selected configuration by mean pilot validation loss, then run five fresh confirmatory seeds to exactly 1B tokens. The complete design therefore stops after 48 pilot runs and 15 confirmatory runs. Do not retune one optimizer after opening confirmatory results.

Interpretation boundary. Lower update dispersion is not enough. The optimizer must improve convergence or quality at an acceptable cost. Moonshot did not release its full optimizer settings, so this remains an independent interpretation rather than a K3 reproduction.

Systems simulation asks where memory policy breaks

The eighth experiment does not train a model.

The SGLang K3 report describes two memory types:

• large fixed KDA state blocks per request;
• small MLA KV pages that grow with tokens.

Separate pools require a startup guess about workload mix. A short-request burst can exhaust KDA slots while MLA memory sits unused. A few long contexts can do the reverse.

Admission is the allocator's decision to accept a request into available memory and the serving queue.

Hypothesis. Unified allocation will admit more requests than fixed pools when the short/long request mix shifts, unless compaction movement erases the admission gain.

Controlled variables:

• total memory;
• request arrivals;
• context distribution;
• service times;
• admission policy.

Primary metrics:

• admitted requests;
• stranded free bytes;
• memory-related rejections;
• queue delay;
• bytes moved during compaction.

Stopping rule. Run each policy on one retained trace plus 20 seeded traces for each of three workload mixes, with 1,000,000 arrivals per trace. Stop after the last arrival and drain the queue. If draining exceeds twice the original trace duration, count remaining queued requests as misses rather than extending the horizon.

Interpretation boundary. The simulation can establish whether unified allocation improves admission for these traces and assumptions. It cannot establish GPU throughput, copy cost, allocator-lock contention, or network behavior without device measurements.

This experiment is especially relevant to my systems background: it turns a model architecture into an allocation and queueing problem with explicit invariants.

Every experiment uses the same publication contract

Each outline must become a frozen protocol before execution. The result post will include this ledger:

The lab will not hide failed runs because they make a chart unattractive. Excluded data needs a retained reason.

For training experiments, the comparison should hold these constant whenever the intervention permits:

• parameter count;
• tokenizer;
• training examples and order;
• token budget;
• optimizer and learning-rate search budget;
• hardware;
• wall-clock budget;
• evaluation checkpoints.

Matching only training steps can favor a slower model. Matching only wall-clock time can give models different token exposure. Report both tokens and time.

Result pages should make reproduction cheaper than trust

Every lab post should ship with:

1. a pinned source revision;
2. one command that creates the environment;
3. one command that runs the experiment;
4. one command that regenerates tables and charts;
5. machine-readable raw results;
6. a README stating expected hardware and duration;
7. a failure log;
8. a license for original code and data.

The article can explain. The repository must let another engineer inspect.

For direct API tests, store request metadata and validator outcomes. Do not commit credentials, sensitive prompts, or raw reasoning content.

For generated long-context fixtures, publish the generator seed and the exact answer derivation. A retrieval test without inspectable ground truth is an opinion poll.

For code-repository tasks, use a pinned permissively licensed repository and record the commit. Repository drift can silently change the answer.

The lab will not use hosted outputs for distillation by default

The Kimi K3 open-weight license broadly permits modification and derivative work, subject to its commercial conditions. The hosted API has separate model-use terms.

Training a competing student from hosted outputs can create a terms problem even when the underlying weights are downloadable.

The default lab rule is:

• direct API experiments measure behavior;
• proxy-model training uses original or clearly licensed datasets;
• distillation from hosted outputs requires a separate terms review;
• open-weight distillation requires hardware, license review, and explicit provenance.

This is an engineering boundary, not legal advice.

Threats to validity

• The proxy model is too small. A mechanism that helps at 2.8T may add only overhead at 100M.
• Compute is not actually matched. One implementation sees more tokens or a larger tuning sweep.
• The benchmark leaks into training. Long-context fixtures or validators appear in the optimization data.
• The intervention changes several things. Replacing attention also changes parameter count, initialization, or kernel maturity.
• One seed becomes the headline. Training variance is mistaken for an architecture effect.
• The fastest kernel wins the model comparison. Implementation quality is reported as architecture quality without qualification.
• A direct API result drifts. The hosted model or backend changes after the verification date.
• Synthetic routing traces are too clean. The simulator misses bursty expert skew.
• Results disappear behind screenshots. No raw data or regeneration command exists.
• A null result is rewritten as a lesson about scale. “It may work at 2.8T” is not evidence.

The lab is successful even when an idea does not improve the proxy model. A well-controlled null result removes one bad assumption.

Start with the two experiments readers can use immediately

The first execution order should be:

1. useful one-million-token context;
2. agent-state reliability;
3. dual KDA/MLA memory-pool simulation;
4. Attention Residuals in NanoChat;
5. Per-Head Muon;
6. hybrid KDA/MLA;
7. Quantile Balancing;
8. SiTU-GLU under QAT.

The first two test the model readers can access. The simulator fits the blog's systems focus without a GPU cluster. The training ablations then increase in implementation and compute cost.

The engineering guide explains what Kimi K3 contains. The lab will test which ideas survive outside Kimi K3.

Continue from one model into reusable engineering

The Frontier Model Engineering workshop series turns the Kimi lessons into methods that apply to future releases:

• audit weights, code, licenses, and benchmark claims before loading;
• build tool-using agents around durable intent and recovery;
• treat long context as representation, allocation, caching, and retrieval;
• design multi-GPU inference from per-rank memory and physical topology.

Kimi remains the worked example. The workshops are the reusable skill and publish before this lab track begins.

The first result stays a draft until it is reproducible

The first lab result should be “Is Kimi K3's One-Million-Token Context Actually Useful?” It will remain a draft until the fixture count, repetitions, backend identity or bounded test window, stopping rules, runner, raw output, and validators are fixed.

Sources and References

Kimi K3 architecture

• Architecture, training, and experiment disclosure: Kimi K3 technical report v1
• Exact layer schedule, SiTU values, and dimensions: Pinned Kimi K3 config
• KDA, MLA, MoE, and SiTU reference model: Pinned Kimi K3 implementation
• KDA production implementation: MoonshotAI/FlashKDA
• Readable KDA reference: Flash Linear Attention KDA
• Attention Residuals: MoonshotAI/Attention-Residuals
• Expert placement: MoonshotAI/MoonEP

Direct K3 tests

• K3 context and streaming examples: Kimi K3 quickstart
• Prefix-cache behavior: Kimi context caching
• Tool protocol: Kimi K3 tool-calling practices
• Hosted pricing: Kimi K3 pricing

Systems and terms

• Hybrid KDA/MLA memory pools: SGLang Kimi K3 report
• Open-weight conditions: Kimi K3 License
• Hosted model-use terms: Kimi OpenPlatform terms