The Distillation Pipeline Is an Artifact DAG, Not a Folder of Checkpoints

A first distillation project often looks like this: a script calls the teacher API in a loop, writes replies to outputs.jsonl, trains a small model on the file, and someone uploads the checkpoint to /s3/models/final_v2_new/. It works once, on a laptop, with nobody else watching.

Now suppose the teacher API times out after writing part of a 500-request batch. Someone reruns the script. Some requests appear twice in the training file, and nobody can say which teacher calls were billed or which rows are safe to reuse.

That's not really a distillation problem. It's a release-engineering problem, and it's the same one message queues and payment processors solved years ago: you need identity before expensive work happens, retries that can't duplicate side effects, and somewhere quarantined data goes that isn't "delete it and hope."

A production distillation pipeline is an artifact directed acyclic graph — contract, requests, results, quarantine, release, receipt — with identity, idempotency, reservation, evaluation, staged rollout, and dated economics built in from the first script. Skip any one of those and you don't get a faster pipeline. You get one that happens to work until a retry, a quantization pass, or a traffic spike proves it didn't.

What you'll learn

• Why a distillation pipeline needs an exact request identity before it needs a scheduler
• How an append-only reservation ledger stops concurrent workers from double-spending a token budget
• Why evaluation has to run as CI with named authority per gate, not one aggregate pass/fail score
• Why the artifact you evaluated is not automatically the artifact you serve, and what re-gating catches
• Why a break-even number is a dated scenario, not a promise

The pipeline is a graph of artifacts, not a sequence of jobs

A directed acyclic graph (DAG) expresses work as nodes connected by one-way dependencies with no path back to the start — the structure Apache Airflow uses to describe task dependencies. A distillation factory adds one more discipline on top: downstream work advances because a contracted artifact exists, not because a process once printed "done."

Distilled: The Engineering of Small, Fast, Cheap AI Models — the book this series adapts — builds its reference pipeline, SupportDistill, around six artifacts: a capability contract, generated teacher requests, terminal results, a quarantine store for excluded rows, a versioned release, and a signed receipt that binds the whole chain. Everything else — deterministic shards, an exact cache, an attempt journal, a reservation ledger — is control-plane machinery that produces and verifies those six nodes.

capability contract
        │
        ▼
  teacher requests ──▶ terminal results ──┬──▶ quarantine (excluded, with reason)
        │                                  └──▶ released rows
        │                                            │
        └───────────── reservation ledger ────────────┘
                                                     │
                                                     ▼
                                        dataset release receipt
                                       (PROMOTABLE | BLOCKED)

The receipt is the only node with promotion authority, and it doesn't have much: it hashes the upstream artifacts, checks that every submitted request has exactly one disposition, and verifies that reservations reconcile. A PROMOTABLE receipt means the checked-in factory checks passed. It does not mean anyone approved a deployment — more on that below.

Identity has to be exact before the teacher API gets expensive

The dataset layer already answers "what content may the student consume." The factory needs an earlier question: has this exact teacher intent already reached a terminal outcome? Conflate the two and you either repeat billable teacher calls or make dataset membership depend on transport accidents like which worker happened to pick up the request.

An exact request identity is a digest over every field that changes what the teacher is being asked to do. In the book's implementation, that's 15 fields: four bind the teacher-access boundary (adapter identity, capabilities, transfer contract, rights manifest), six describe the execution itself (input text, prompt version, teacher version, generation parameters, policy version, code version), and five preserve curriculum and governance intent (cell, intent, mode, rights, parent IDs). Deliberately excluded: the logical request ID, shard assignment, and split — because none of those change what the teacher actually does.

def request_identity(request, teacher) -> str:
    fields = [
        teacher.adapter_request_identity,   # ①
        teacher.capabilities_sha256,
        teacher.transfer_contract_sha256,
        teacher.rights_manifest_sha256,
        request.text,
        request.prompt_version,
        teacher.version,
        request.generation_params,
        request.policy_version,
        request.code_version,
        request.cell_id,                    # ②
        request.intent_id,
        request.mode,
        request.rights,
        request.parent_ids,
    ]
    # request_id, shard_id, split are excluded on purpose  # ③
    return sha256("\n".join(str(f) for f in fields))

① Four fields bind the teacher-access boundary. Change the rights manifest or the transfer contract and the digest changes too, so a policy change forces new teacher work instead of silently reusing an old answer under a rule that no longer applies.

② Curriculum and governance fields make two requests distinct even when the literal prompt text matches, because their downstream meaning differs — a demonstration and a counterexample built from the same text are not the same row.

③ Request ID, shard, and split are transport and scheduling details. Hashing them would turn an ordinary scheduler retry into a guaranteed cache miss, which defeats the entire point of exact identity.

Retries need a policy, and the ledger has to be append-only

Concurrency doesn't just make a loop faster — it creates independent worker, provider, and persistence clocks. A timeout can mean failure, success, or success-whose-evidence-hasn't-landed-locally-yet. Those are three different states, and a scheduler that collapses them into one is guessing.

The safe order is reserve first, submit second: check a projected total against budget, and only start a teacher call if every required reservation succeeds.

projected_requests = reserved_requests + len(shard.request_ids)
projected_tokens    = reserved_tokens + shard.reserved_tokens
projected_dollars   = reserved_dollars + shard.reserved_dollars
 
if projected_requests > request_budget:              # ①
    raise ValueError("request budget exhausted")
if projected_tokens > token_budget:
    raise ValueError("token budget exhausted")
if projected_dollars > dollar_budget:
    raise ValueError("dollar budget exhausted")
 
ledger.append_once(reserve_event)                    # ②
outcome = teacher_call(request, attempt)
if outcome.status == "recoverable" and attempts(request) < max_attempts:  # ③
    schedule_retry(request)

① Reserve before you spend. A worker checks its projected total against budget before a single teacher call happens, so backpressure is a decision made ahead of cost, not a bill discovered after the fact.

② append_once() refuses to add an event whose ID already exists. That's what makes a resumed run safe — a run that crashes mid-shard and restarts doesn't append a second reservation for the same shard. Worth being honest about the limit here: a file-backed append_once() is idempotent within one process. It is not a transactional multi-writer ledger. Two schedulers reading the same starting balance concurrently can each pass the check and both append — put reservation enforcement behind an atomic compare-and-set or single-writer service before you add real workers.

③ An identity gets a bounded number of attempts, and the outcome — recoverable, completed, quarantined — is a state the system actually emits, not a status invented after the fact to make a postmortem read cleanly.

For a closed run, reserved dollars have to equal actual spend plus released capacity. In the book's own labeled fixture: reserved $0.0440 = actual $0.0380 + released $0.0060. That's a toy number chosen to make the identity checkable by hand, not a claim about your bill — but the invariant it demonstrates (every reserved dollar ends as either settled spend or released capacity) is the one worth keeping.

The interactive workbench below injects a timeout, worker crash, schema change, or exhausted budget. Compare a directory-driven job with an artifact ledger and inspect which terminal rows can be reused safely.

Three ways to run teacher generation, and none of them is free

Where retries and partial completion live depends on which generation topology you pick, and the choice changes who owns recovery — not just throughput.

A team can pick a topology for its headline advantage and ignore its failure model. Batch pricing looks attractive until a product needs interactive latency; synchronous calls are simple until unknown outcomes accumulate; self-hosting looks predictable until upgrades and queue recovery become their own product. Run a partial-completion drill before committing production volume to any of the three.

PROMOTABLE means the receipt passed its checks — not that anyone approved a deployment

The released row schema is a compatible successor: it keeps every field the upstream dataset contract requires (content, lineage, rights, parents, split eligibility) and adds execution metadata on top, rather than silently dropping fields a downstream consumer expects. A receipt that hashes ten factory artifacts, verifies row-ID uniqueness, checks split policy, and reconciles spend earns exactly one of two states: PROMOTABLE or BLOCKED. Requests that never resolve go to a dead-letter store instead of vanishing.

PROMOTABLE is not PROMOTED. The receipt proves the checked-in factory checks passed. It doesn't prove student quality, doesn't authorize export, and doesn't authorize deployment. Those are separate evidence boundaries with separate authorities — conflating them is exactly how a passing pipeline ships the wrong artifact.

The design decision worth stealing here: make resumption an artifact query, not a memory of where a process was. A resumable run reconstructs expected requests from the request store, rebuilds reservations from ledger events, and reconciles membership from the results store — it doesn't ask which loop index the last process reached. That answer survives a crashed process because it comes from durable artifacts, not runtime state.

Evaluation has to run as CI, with named authority per gate

Continuous integration (CI), applied to evaluation, means a candidate crosses a repeatable gate, produces versioned evidence, and can't cross an approval boundary while any blocking check is false. A notebook that prints one aggregate score is not that — it can't tell you which exact artifact is authorized to cross which boundary.

Public benchmarks decay as evidence once a model ecosystem has had time to adapt around them. Zhang et al. built GSM1k, a fresh grade-school arithmetic set designed to mirror GSM8K without prior exposure, and reported accuracy drops of up to 8% for some evaluated models even though many frontier models generalized well. Freshness doesn't make an evaluation perfect — it reduces the candidate's opportunity to have already seen the answer key.

Different evaluation products answer different questions, and merging their authority is the fastest way to let a convenient score approve the wrong thing:

A judge model scales qualitative review, but it's an instrument, not an oracle — and instruments need calibration. In the book's own deliberately small fixture (eight comparison cases, chosen so the math stays checkable by hand): overall agreement between judge and human labels comes out to 0.75, a mean calibration error of 0.35, and a 95% Wilson interval of roughly 0.15 to 0.85 on two passes out of four high-risk cases. That interval is wide because four observations carry almost no information — which is exactly the point of the example. Four cases don't justify a release decision, and a real calibration program needs enough labeled cases per risk slice that the interval actually narrows.

The gates themselves need explicit authority classes, not a single pass/fail:

• Fail closed — identity, lineage, contamination, high-risk behavior, blocking calibration. No deadline overrides these.
• Advisory — early latency proxies, exploratory robustness checks. Recorded, doesn't alone block.
• Observe only — a new metric or sparse slice that isn't reliable enough for a decision yet, so it has no override because it has no authority to begin with.

The artifact you evaluated is not the artifact you serve

Quantization and export are semantic transformations, not packaging steps. AWQ and GPTQ both operate directly on weight precision, and a runtime that dequantizes on the fly or lacks an efficient kernel for the resulting format can be both slower and behaviorally different from the FP16 checkpoint that passed your evaluation gate. This is why re-gating on target hardware isn't optional polish — it's the only place the pipeline actually checks the thing it's about to serve.

The book's fixture separates two gates that ask genuinely different questions:

Two gates, two questions. served_fixture_behavior_gate asks whether deterministic runtime behavior stayed within limits. target_runtime_gate asks whether the exact served artifact has adequate evidence on target hardware. One gate cannot answer the other — a fixture can pass every behavior check while the hardware gate stays BLOCKED because the memory and latency numbers are still mocked, not measured.

Track named safety metrics separately from an aggregate score, because export can move exactly the metric that controls release while barely touching the average. In the book's fixture, unsafe_compliance and over_refusal both hold at 0.0 (limit 0.0) while correct_escalation and policy_consistency hold at 1.0 (limit 1.0) — four cases per slice, deliberately small, and explicitly not a production sample size. What matters is the pattern: name the metric, name the limit, name the direction, and don't average a safety blocker into a composite that hides it.

Canary rollout is a ladder, not a switch

Progressive delivery decides exposure; routing decides authority. The two are separate control surfaces, and a canary that gets both right looks like a ladder with an exit condition at every rung:

Those percentages are one fixture's choice, not a universal recommendation — exposure should reflect risk, label delay, traffic volume, and how fast you can detect a problem, not a template. What should generalize is the stop-trigger discipline: name each trigger's metric, evaluation window, minimum sample size, and action before rollout starts. The book's fixture trips on high_risk_safety_rate > 0.10, p95_latency_ms > 500, or artifact_identity_mismatch — and that latency threshold is deliberately tighter than the product's own 800 ms end-to-end p95, so the canary stops while there's still response budget left to work with.

Rollback starts before promotion. Retain and hash the rollback artifact, verify runtime compatibility, and rehearse the traffic switch before a canary ever receives live authority. In the book's drill, a tripped breaker opens, the action is disable_student_route, and restoration checks that the prior served package's hash still matches before anything gets restored — a plan that depends on rebuilding the prior model from scratch is a recovery project, not a rollback. Google's SRE book frames a postmortem the same way: it only counts if it changes the next gate, not just the next conversation.

What could go wrong

• Quantization changes what the model does, not just how fast it runs. AWQ and GPTQ both operate on weight precision directly — a passing FP16 evaluation gate tells you nothing about the INT4 artifact until you re-run the gate on it.
• A benchmark stops being evidence once the ecosystem adapts to it. GSM1k's up-to-8% accuracy drop versus GSM8K is the concrete version of "your holdout isn't hidden anymore."
• KV-cache memory, not parameter count, is what breaks serving under concurrency. The PagedAttention paper identifies dynamic cache growth as a serving bottleneck that a pre-export latency proxy running at concurrency one will never see.
• Recursive training on model-generated data degrades quality across generations even when each individual run looks fine. Shumailov et al., Nature 2024 is the direct argument for keeping quarantine and lineage checks live in the DAG instead of treating "the dataset got bigger" as an unambiguous improvement.
• Governance treated as a sign-off instead of a gate. NIST's AI Risk Management Framework frames governance as a function that runs across the whole system — which is exactly why a receipt hash and a release decision need to stay two separate artifacts, checked separately, instead of one score that quietly does both jobs.

The spreadsheet is a dated scenario, not a receipt

Every distillation economics argument eventually produces a "we'll save $X/month" line, and that line is almost always doing less work than it looks like. The book's own worked scenario — dated July 31, 2026, and explicitly labeled as fixture assumptions rather than a vendor quote — makes the sensitivity visible instead of hiding it behind one number:

Same 14,200/month fixed cost (build amortization, maintenance, refresh), same 0.008/call API alternative, same 0.0004/call student variable cost — and the break-even volume moves from 11,295,455 calls at 35% utilization down to 3,315,934 calls at 85% utilization. That's a 3.4× swing from one assumption. The self-host number in the base case beats the API number by exactly 323.08 a month, which is not a margin worth building a fixed pipeline around without first replacing the utilization assumption with a measurement.

The general form is simple, and worth keeping simpler than the spreadsheet that implements it:

F = build amortization + maintenance + refresh
serving_savings(u) = c_api - (c_student + redundancy * c_capacity / u + fallback_rate * c_fallback)
break_even_calls = F / serving_savings(u), only when serving_savings(u) > 0

u (utilization) is doing almost all the work in that formula, and it's also the number every team is most tempted to assume optimistically instead of measure. Replace it — and the fallback rate, and the redundancy factor — with reconciled local measurements before the break-even number gets to influence a go/no-go decision. Until then it's a scenario that tells you where to point instrumentation, not a receipt.

Key takeaways

• Identity comes before cost: hash the fields that change what the teacher is asked to do, and exclude the transport fields that don't.
• Reserve before you spend, and make the reservation ledger append-only — that's what makes a resumed run safe instead of a guess.
• PROMOTABLE is a receipt passing its own checks. It is not deployment approval, and treating it as one is how the wrong artifact ships.
• Evaluation authority has to be named per gate (fail closed, advisory, observe only) — an aggregate score can't carry that distinction.
• Re-gate the exact served artifact. Quantization and export change behavior, not just speed.
• A canary needs named stop triggers and a rehearsed rollback before it gets live traffic, not after.
• A break-even number is a snapshot of assumptions, most sensitively utilization — treat it as an instrumentation plan, not a conclusion.

What's next

This post adapts Chapters 7 through 9 of Distilled: The Engineering of Small, Fast, Cheap AI Models. The next post in the series picks up exactly where a BLOCKED governance decision leaves off — what changes when one student becomes a fleet of specialists, and why that turns "adaptive" into a routing and reliability problem before it's a modeling one.

Sources and References

Pipeline and DAG mechanics

• Apache Airflow: DAGs
• Distilled: The Engineering of Small, Fast, Cheap AI Models

Evaluation and contamination

• GSM1k: A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Quantization and serving

• AWQ: Activation-aware Weight Quantization
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Efficient Memory Management for Large Language Model Serving with PagedAttention

Governance and reliability

• NIST AI Risk Management Framework
• Google SRE Book: Postmortem Culture
• AI models collapse when trained on recursively generated data, Nature 2024