Dark Knowledge: What a Teacher's Wrong Answers Are Actually Worth

Train a student on hard labels alone and every wrong answer looks equally wrong. A refund classifier that assigns 0.01 probability to escalate and one that assigns 0.30 to it both get scored as "correct: refund" against a one-hot label. But those are different models. One is confident and narrowly right. The other thinks this case might need a human, and just didn't say so out loud.

That difference — the shape of a model's uncertainty across every answer it didn't give — is what Geoffrey Hinton, Oriol Vinyals, and Jeff Dean named dark knowledge in their 2015 distillation paper, and it's the reason distillation can transfer more than a teacher's final answer.

A hard label discards the one signal a teacher spent its whole training run producing

A logit is an unnormalized score a model produces for each possible output, before softmax turns the vector of scores into a probability distribution. A hard label names exactly one accepted answer — a one-hot vector, 1.0 on the correct class and 0.0 everywhere else. A soft target is the full distribution instead: maybe 0.64 on refund, 0.24 on replace, 0.09 on escalate, and 0.03 on deny.

Hinton, Vinyals, and Dean's original recipe uses exactly these relative probabilities, because the paper's core claim is that "the relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize." A model that's 64% sure a ticket needs a refund and puts real weight on escalate over deny is telling you something about how it discriminates between edge cases — information a one-hot label physically cannot carry, because it only ever contains a single 1.0.

That extra signal is only useful if you trust it. A teacher can be right about its top answer and badly miscalibrated about everything else. Calibration asks whether a stated confidence level matches empirical correctness, and Guo et al. found that modern neural networks are frequently accurate while being poorly calibrated. Distilling from an overconfident teacher can transfer that overconfidence with impressive numerical precision — dark knowledge is a signal worth extracting, not an oracle worth trusting blindly.

Temperature is the dial that decides how much of that signal you can see

At temperature T = 1, a well-trained classifier's softmax output is often extremely sharp — the winning class north of 0.999, everything else vanishing toward zero. That sharpness is exactly what buries dark knowledge: the structure Hinton's paper cares about lives in the tail, and a sharp distribution has almost no visible tail.

Raising the temperature flattens the distribution and exposes it:

def softmax(logits: list[float], temperature: float) -> list[float]:
    scaled = [z / temperature for z in logits]
    max_z = max(scaled)
    exps = [pow(2.718281828, z - max_z) for z in scaled]  # ①
    total = sum(exps)
    return [e / total for e in exps]

① Subtracting the maximum before exponentiating is standard numerical-stability practice — it keeps the largest exponent at zero instead of overflowing on a large logit.

A worked run against one fixture — four logits over the tokens refund, replace, escalate, deny — makes the effect concrete. Running the same four logits at three temperatures and tracking how much probability mass survives in the top two entries produces this table:

The winning token never changes across all three temperatures. A top-token agreement metric would report zero difference between them. But the mass discarded by keeping only the top two entries grows from about 1.8% at T = 0.5 to nearly 27% at T = 2.0 — the same model, the same input, and a completely different amount of visible structure depending on the dial you chose. A metric that only checks "did the top answer match" is blind to that entire shift.

Renormalizing a stored top-k distribution makes the kept entries sum back to one, but it does not recover what the teacher discarded. If you truncate to the top two and then renormalize, you get a distribution that looks complete and isn't — record the lost mass explicitly, or keep an honest residual bucket, instead of quietly pretending the tail never existed.

The loss, in the same terms the fixture just computed

For teacher logits z^T, student logits z^S, and a positive temperature τ (tau), define softened distributions the same way the table above did:

p_i^T(τ) = exp(z_i^T / τ) / Σⱼ exp(z_j^T / τ)
p_i^S(τ) = exp(z_i^S / τ) / Σⱼ exp(z_j^S / τ)

The canonical mixed objective from Hinton's paper combines a hard-label term with a soft-target term:

L = (1 - α) · CE(y, p^S(1)) + α · τ² · KL(p^T(τ) ‖ p^S(τ))

α (alpha), between zero and one, weights the softened-teacher term against plain cross-entropy on the hard label. The τ² factor isn't decoration — because raising temperature flattens gradients along with the distribution, multiplying by τ² restores the gradient magnitude the classic recipe relies on to keep training stable across different temperature settings.

Kullback-Leibler (KL) divergence measures how much one probability distribution diverges from another, and the direction you compute it in changes what mistake it punishes. Forward KL, KL(p^T ‖ p^S), penalizes the student heavily wherever it assigns too little probability to something the teacher supports — it's often called mass-covering because the pressure spans every mode the teacher considers plausible. Reverse KL, KL(p^S ‖ p^T), penalizes the student for putting mass where the teacher assigns little — mode-seeking, because the student can concentrate on one high-probability region instead of covering all of them. MiniLLM replaces the standard forward-KL objective with reverse KL, combined with on-policy training, specifically for generative language-model distillation.

Neither direction is universally safer. Missing a minority escalate mode entirely and spreading probability mass thin across several weak answers are two different product failures — the right choice depends on which one your system can tolerate less.

Full logits, top-k, or no logits at all: a storage decision, not a modeling one

Storing dark knowledge has a cost, and the cost scales with how much of the vocabulary you keep per token. For N training examples, L token positions per example, vocabulary size V, and b bytes per stored value, an uncompressed full-logit artifact needs:

bytes = N × L × V × b

That's a planning bound, not a benchmark — metadata, indexing, compression, and replication all change the realized number, but the shape of the problem (storage scales with vocabulary size, not with how much of that vocabulary actually matters) is the point.

The right row can differ by product slice inside the same project — full logits might be worth the storage bill for refusal and escalation decisions specifically, while routine formatting output rides on cheap transcripts alone.

What could go wrong

• Truncation erases exactly the alternative that mattered. If escalate is usually ranked sixth, a top-five store gives it no explicit probability at all. An "other" bucket preserves the total mass but not which specific omitted token owned it — a faithful tail can't be reconstructed from an aggregate residual after the fact.
• Online scoring introduces teacher drift. If the endpoint changes between training epochs — a version bump, a routing change — two retries of the same record can silently produce two different targets. Version the teacher explicitly, cache request identities, and reject unrecorded substitutions, or a fixed student seed won't make the run reproducible.
• Agreement can improve while correctness falls. A teacher can be confidently wrong on a policy edge case, and a softened language-model distribution isn't automatically a trustworthy routing confidence score. Evaluate independent labels and calibration alongside KL — the teacher is a source of structure, not an oracle, a caveat worth repeating because it's the one most post-mortems skip.
• Tokenizer mismatch makes KL meaningless without ever throwing an error. Token-level KL assumes teacher probability p_i and student probability q_i describe the same event. If one tokenizer encodes a word as a single token and another splits it into two, "position seven" isn't the same random variable in both models — the loss can decrease numerically while the student learns nothing coherent, and nothing in the arithmetic will tell you that happened.

Key takeaways

• Dark knowledge is the structure inside a teacher's full output distribution — the relative weight on wrong answers — that a one-hot hard label discards entirely.
• Temperature controls how much of that structure is visible. The same fixture at T = 0.5, 1.0, and 2.0 keeps the same top answer while losing dramatically different amounts of tail probability.
• The mixed distillation loss needs the τ² correction to keep gradient magnitude comparable across temperatures — dropping it silently changes what the temperature knob actually does.
• Forward KL and reverse KL punish different mistakes (missing a mode versus spreading mass thin). Neither is a universally correct default.
• Storing dark knowledge is a cost decision with a real failure mode per option — full logits, top-k with a residual bucket, online scoring, or no logits at all.
• Matching a teacher's uncertainty exactly is only worth it where that uncertainty is trustworthy. Calibration and independent evaluation are not optional side checks.

What's next

The next post in this series moves from matching one teacher's output distribution to building the actual training data a student learns from — the four gates a synthetic data factory needs before it needs more prompts.

This post adapts material from Chapter 3, "Matching Minds," in Distilled: The Engineering of Small, Fast, Cheap AI Models. The rest of the series is indexed at the series page.

Sources and References

Distillation fundamentals

• Hinton, Vinyals, and Dean, "Distilling the Knowledge in a Neural Network"
• Sanh et al., "DistilBERT, a distilled version of BERT"

Calibration and divergence

• Guo et al., "On Calibration of Modern Neural Networks"
• Gu et al., "MiniLLM: On-Policy Distillation of Large Language Models"

Sequence and attention transfer

• Bengio et al., "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks"
• Jain and Wallace, "Attention is not Explanation"