Model Distillation Breaks in the Same Ten Places Every Time

In 2023, researchers fine-tuned an aligned GPT-3.5 Turbo checkpoint on ten adversarial examples, spent less than $0.20 in compute, and reported that the resulting model would comply with nearly any harmful instruction in their test setup Qi et al., "Fine-tuning Aligned Language Models Compromises Safety". The experiment tested fine-tuning, not distillation — training a smaller student model to reproduce a larger teacher's behavior — but the uncomfortable lesson generalizes: safety, calibration, and capability are properties of the exact artifact that was tested. A smaller descendant doesn't inherit them for free.

Distillation is genuinely useful. DistilBERT shrank BERT by 40% and ran 60% faster while keeping 97% of its language-understanding performance Sanh et al.. Character.AI reports cutting serving costs by at least 33x since their 2022 launch, driven substantially by moving traffic onto smaller, distilled models Character.AI engineering blog. OpenAI ships a managed distillation product for exactly this reason OpenAI, "Model Distillation in the API". None of that is in dispute.

What's in dispute is whether the specific distilled artifact your team shipped actually kept what the benchmark said it kept. Below are ten failure points grounded in the cited literature, grouped by where the failure appears rather than repeated as ten identical templates.

What you'll find below:

• Why style imitation and capability transfer are different axes, and how to tell them apart before a demo fools you.
• The service envelope you need defined before you touch a loss function.
• Why a model that passes eval in FP16 can fail after 4-bit export, and the re-gating step that catches it.
• An audit table mapping each failure to its symptom, root cause, and fix.

Data and curriculum failures teach the wrong lesson

Style is not capability

The easiest student to overtrust is the one that sounds like its teacher. Gudibande and coauthors trained small open models purely on ChatGPT transcripts and found that crowdworkers initially rated about 70% of the imitation models' outputs as equal to or better than ChatGPT's. Then the paper adds the part that matters: "as we train on more imitation data, our models fail to further close the gap, and even begin to regress along other axes." Tone, formatting, and polite openings are cheap for a student to copy. Factual recall and logical consistency are not, and imitation training optimizes for the former long before it touches the latter.

The fix, and it's the same fix that shows up in the next two items: stop training on text transcripts alone. Add reasoning traces so the student practices the steps, not just the conclusion, and add on-policy correction so the student's own mistakes — not just the teacher's polished answers — enter the training loop.

An unfiltered generation loop trains on the teacher's worst day

Teacher models hallucinate, repeat themselves, and occasionally answer a question with a paraphrase of your eval set. Lee and coauthors found extensive duplication and train/test overlap in common LM training corpora, and deduplication measurably reduced memorized output in their experiments. Feeding a student unfiltered teacher output amplifies whatever the teacher gets wrong on a bad day, because there's no gate between "the teacher said this" and "the student trained on this."

The fix is mechanical, not conceptual: schema validation, deduplication, and decontamination, run in that order, before a single row reaches the optimizer. The previous post in this series builds all three gates end to end — it's worth reading as the companion piece to this one, since half of what breaks a distilled model breaks upstream, in the data.

Perfect transcripts teach a model that never learned to recover

Train exclusively on flawless teacher trajectories and the student never sees what happens after a small mistake — because the teacher never made one in the training set. This isn't a new LLM-era problem: Bengio and coauthors identified the same failure in sequence models back in 2015 — a train/inference mismatch where small early errors compound because the model was never shown how to recover from its own drift.

The modern fix is on-policy distillation: let the student generate its own rollout, have the teacher score or correct the tokens where the student went wrong, and train on that. Agarwal and coauthors show this directly reduces the train/inference distribution mismatch compared to training on teacher-only transcripts. DeepSeek's own distillation results back this up at scale: DeepSeek-R1-Distill-Qwen-32B, built on reasoning traces rather than static transcripts, scores 72.6% on AIME 2024 and 94.3% on MATH-500 — and the same report notes that direct distillation from R1 outperformed running reinforcement learning on the base model directly. Direct Preference Optimization and rejection sampling are the two most common mechanisms for turning corrected rollouts into a training signal; Zephyr-7B used exactly this recipe to beat a 70B chat model on MT-Bench with no human annotation at all.

Public benchmarks measure memorization more than they measure your product

GSM8K, HumanEval, and MMLU are useful, contaminated, and static — all three at once. Zhang and coauthors built GSM1k, a fresh benchmark matching GSM8K's difficulty distribution, and found accuracy drops of up to 8% across leading open- and closed-source models, with several families showing systematic overfitting to the original set. That's an 8-point gap between "passes the benchmark" and "actually generalizes," on existing frontier models — a distilled model trained partly on synthetic data that itself leaked from public benchmark phrasing has no reason to be more honest.

The fix is a benchmark you refresh, not one you trust forever. LiveCodeBench is a working example: it continuously pulls new problems from live coding contests specifically so contamination can't accumulate. Build (or budget for) the equivalent private, versioned holdout for your domain, and refresh it on a cadence contamination can't outrun.

Engineering decisions made before the constraints existed

A loss function doesn't know your p95 latency budget

It's easy to spend a week tuning a distillation temperature schedule and the cross-entropy/KL weighting between them, and only then discover the resulting checkpoint's KV cache blows your memory budget under real concurrency. PagedAttention's authors put it plainly: "the KV cache memory for each request is huge and grows and shrinks dynamically... this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size." A checkpoint with excellent eval loss and no defined memory or latency ceiling is not a deployment candidate — it's an unfinished spec.

My opinion, not a universal rule: define the service envelope — an accuracy floor, a memory ceiling, and a latency ceiling — as versioned configuration before you pick a loss function or a compression target, and wire deployment approval to check all three automatically. This works well when the envelope reflects real product SLAs; it stops working if the numbers are guessed rather than measured against your actual traffic shape.

Quantization is not a free re-encoding of the same model

A checkpoint that passes every eval gate in FP16 is not the artifact you're about to serve. Export to a 4-bit format — AWQ, GGUF, or a comparable scheme — and you've changed the model's numerical representation, not just its file size. Jacob and coauthors' foundational integer-quantization work had to co-design a training procedure specifically to preserve accuracy post-quantization, because a naive post-hoc conversion doesn't preserve it automatically. Hugging Face's own quantization documentation is blunt about this too: some methods "require calibration for greater accuracy," which is another way of saying the un-calibrated version can degrade.

The fix: re-run your full evaluation suite — including the safety and structured-output checks, not just the accuracy benchmark — against the exact exported artifact running on the target production hardware. The FP16 checkpoint's eval results describe the FP16 checkpoint.

Parameter memory is the floor, not the budget

"An 8B model in FP16 is 16 GB, so it fits comfortably on a 24 GB card" is true and irrelevant. Parameter weights are one term in the memory equation, not the whole thing:

KV cache bytes per token = 2 × layers × KV heads × head dimension × precision bytes

The factor of two is keys and values. Multiply by expected concurrent sequence length and batch size and the KV cache — the per-request memory that holds each sequence's attention keys and values — can dwarf the weights themselves under real traffic, exactly the dynamic-growth problem PagedAttention was built to manage. Using grouped-query attention — sharing key/value heads across multiple query heads to shrink the cache — changes the constant in that formula but doesn't remove the need to compute it; see TensorRT-LLM's attention documentation for how MQA/GQA change the shape.

The fix: size hardware from the memory inequality above, evaluated at your expected concurrency and sequence length — not from the weights-only number on the model card.

Organizational and safety failures

A teacher's safety evaluation describes the teacher, not your export

This is the opening story again, applied directly: Qi et al.'s ten-example, sub-$0.20 fine-tuning attack shows that alignment can be cheap to disturb. A separate, more cautious data point: a 2025 Cisco and University of Pennsylvania evaluation applied an automated jailbreak method to 50 random HarmBench prompts and reported a 100% attack-success rate against the tested DeepSeek-R1 endpoint, suggesting — but not proving — that reinforcement learning, chain-of-thought self-evaluation, and distillation may have contributed. That report didn't isolate distillation as the cause, didn't compare one artifact before and after distillation, and didn't establish a universal failure rate for the model family. Treat it as a real, sourced result for one evaluation configuration, not a causal verdict on compression generally — and treat "the teacher passed our red team" as evidence about the teacher, full stop.

The fix: any change to weights, precision, prompts, or tools is what I'd call an assurance-invalidating event. Re-run the safety suite against the descendant artifact, bound to its exact hash and configuration, rather than inheriting the teacher's report card.

One model cannot be your classifier, your summarizer, and your SQL generator

Forcing an 8B student to generalize across every task your product needs — classification, summarization, tool calls, structured generation — trades away the capacity each task actually needs. The industry's answer at two different layers is the same bet: specialize, then route. At the architecture level, mixture-of-experts models make this trade inside a single network — Mixtral gives each token access to 47B parameters but only activates 13B of them per token. At the fleet level, production systems make the same bet across separate models: OpenAI's GPT-5 ships "a real-time router that quickly decides which [model] to use based on conversation type, complexity, tool needs, and your explicit intent" OpenAI, "Introducing GPT-5", and Cursor's smaller Tab model "runs on every user action, handling over 400 million requests per day" Cursor, "Improving Cursor Tab with Online RL" — specifically because it doesn't need to be a generalist to do that one job well. The cost case for routing over one big generalist is well documented: FrugalGPT's cascades matched the best individual LLM's performance with up to 98% cost reduction, and RouteLLM reports over 2x cost reduction in certain cases without compromising response quality.

The fix: a lightweight router (or classifier) in front of several narrow, specialized students beats one model asked to be everything.

Teacher API calls are a budget line, not a background job

A generation job with no reservation, no rate limiting, and no circuit breaker is a cost incident waiting for a retry loop. This is an operational problem with operational solutions that already exist and are documented: OpenAI's Batch API processes jobs within 24 hours at a 50% cost discount compared to synchronous calls — a real lever for cutting generation cost if your pipeline doesn't need synchronous responses. Rate limit documentation spells out RPM/RPD/TPM/TPD ceilings you should be enforcing client-side, not discovering via a 429. And AWS's builders' library on idempotent retries is the standard reference for making a retried request safe instead of a duplicate-generating hazard.

The fix: batch where you can, cache idempotently, enforce a hard dollar ceiling before the job starts, and isolate failures into the same kind of quarantine the data pipeline uses — don't let a malformed prompt recurse into an unbounded API bill.

What this checklist can't catch by itself

Ten patterns, ten fixes, and a green checkmark next to each one is still not proof the model is safe to ship. A model can pass every item below and still fail on a rare, consequential case that none of these ten happen to probe — these are common failure shapes, not an exhaustive certification. Treat a fully checked list as "we ruled out the usual suspects," not "we proved the model is correct."

The field guide

Key takeaways

• Style and capability are different axes — a fluent student can still be factually unmoored, and the only way to know is to test reasoning, not tone.
• Every adaptation step (fine-tune, distill, quantize, route) invalidates the safety and quality evidence collected on the previous artifact. Re-test the thing you're actually shipping.
• Cost and specialization aren't in tension — FrugalGPT and RouteLLM both show routing to smaller models reduces cost while holding quality, which is the opposite of the "one big generalist" instinct.
• A ten-item checklist rules out the common failures. It doesn't replace a residual-risk review for your specific product.

What's next

If any of these ten started upstream, in the data, the previous post in this series builds the schema, deduplication, decontamination, and manifest gates that catch most of patterns one through four before training even starts.

This piece is adapted from Chapters 6, 9, and 11 of Distilled: The Engineering of Small, Fast, Cheap AI Models.

Sources and References

Safety and adaptation

• Qi et al., "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!"
• Cisco Security and University of Pennsylvania, "Evaluating Security Risk in DeepSeek and Other Frontier Reasoning Models"

Distillation technique

• Gudibande et al., "The False Promise of Imitating Proprietary LLMs"
• Bengio et al., "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks"
• Agarwal et al., "On-Policy Distillation of Language Models"
• Rafailov et al., "Direct Preference Optimization"
• Tunstall et al., "Zephyr: Direct Distillation of LM Alignment"
• DeepSeek-AI et al., "DeepSeek-R1"
• Sanh et al., "DistilBERT"

Evaluation and contamination

• Zhang et al., "A Careful Examination of Large Language Model Performance on Grade School Arithmetic" (GSM1k)
• Jain et al., "LiveCodeBench"
• Lee et al., "Deduplicating Training Data Makes Language Models Better"

Serving and compression

• Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention"
• NVIDIA TensorRT-LLM, "Multi-Head, Multi-Query, and Group-Query Attention"
• Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference"
• Hugging Face, "Transformers Quantization Overview"

Routing and cost

• Jiang et al., "Mixtral of Experts"
• Chen, Zaharia, and Zou, "FrugalGPT"
• Ong et al., "RouteLLM"
• OpenAI, "Introducing GPT-5"
• Cursor, "Improving Cursor Tab with Online RL"
• Character.AI, "Optimizing AI Inference at Character.AI"
• OpenAI, "Model Distillation in the API"

Operations

• OpenAI, "Batch API"
• OpenAI, "Rate Limits"
• AWS Builders' Library, "Making Retries Safe with Idempotent APIs"