Why Small Models Win the Enterprise AI Cost Argument

Your prototype shipped in a week. A Python script, an API key, and a system prompt that turns a messy support ticket into {"intent": "refund_status", "urgency": "high"}. It worked in the demo. Stakeholders were impressed. Then it went to production, and the first monthly invoice landed with a number nobody had budgeted for.

That invoice isn't a pricing accident. It's the predictable output of a decision made at the prototype stage: routing every request, including the trivial ones, through the most capable model available.

A frontier model prices capability you didn't ask for

A frontier model is a model near the leading edge of capability and training investment at a given time — it names a moving category, not a fixed architecture. Epoch AI estimated that the amortized hardware and energy cost of frontier models' final training runs grew 2.4 times per year since 2016, with a 95% confidence interval of 2.0 to 3.1 times — and the researchers were explicit that these are modeled estimates, not invoices. Someone pays for that scale, and the provider recovers it by charging a marginal price per request.

That marginal price varies more than most teams assume, even within a single vendor. OpenAI's standard pricing, accessed August 4, 2026, lists one model at 5 per million input tokens and 30 per million output tokens, while a smaller model in the same family lists at 0.20 and 1.20 — a 25-fold spread inside one product line. That spread is evidence that a single provider already segments capability into different cost tiers. It isn't evidence that the cheapest tier meets your quality bar; you still have to measure that separately.

Extracting five structured fields from a support ticket doesn't require a model that can also reason about quantum mechanics or draft a sonnet. You're renting the whole warehouse to deliver one envelope, and the bill reflects the warehouse, not the envelope.

Prefill and decode explain where the latency and the bill actually come from

Strip away the chat interface and a language model is a conditional-probability service with two computational phases. During prefill, the runtime processes every input token in parallel and builds the state needed to start generating. During decode, it produces one new token at a time, appends it to the sequence, and repeats — a serial process even though each individual step does highly parallel arithmetic (Vaswani et al., "Attention Is All You Need").

Decode reads the full weight set, and the KV cache — stored attention keys and values for every token already in the sequence — for every single output token. The paper behind vLLM identifies that dynamic KV-cache growth as a serving bottleneck that can fragment memory and limit how many requests a runtime can batch together when it's managed inefficiently. This is why a long conversation or a high-concurrency workload degrades tail latency before it visibly moves the average: the tokens you're not looking at are still occupying memory bandwidth.

None of this is a reason bigger models are wrong. It's a reason "which model" and "which resource is actually constrained" are two different questions, and conflating them is how a prototype's architecture survives unexamined into a production incident.

Scaling laws tell you what a teacher can do, not what your product should own

For years, Kaplan et al.'s scaling laws and their 2022 correction — Hoffmann et al.'s Chinchilla work, which trained more than 400 models and found that many large models had been undertrained relative to their data budget — got cited as proof that bigger is better. Read them more carefully and they say something narrower: under a fixed compute budget, there's an empirically better allocation between parameter count and training tokens. Chinchilla, at seventy-billion parameters, used the same training compute as the 280-billion-parameter Gopher model, trained on four times the data, and beat it on the reported evaluation suite.

That's a resource-allocation lesson, not a service-level guarantee. A scaling curve describes validation loss on a broad web corpus. It says nothing about whether your billing-support classifier handles duplicate_charge correctly, whether p95 latency clears 800 ms, or whether a smaller model can be trusted with an escalation decision. Scaling evidence justifies experimentation. Only workload-specific measurement justifies an architecture decision — the same discipline knowledge distillation, the transfer of selected predictive behavior from a teacher model into a student model, depends on. The 2015 paper that introduced it was explicit that the goal is compressing an expensive model into "a single model which is much easier to deploy" (Hinton, Vinyals, and Dean) — not proving the student is smarter, only that it's sufficient and cheaper to run.

A capability contract turns "the demo worked" into a testable obligation

A demo answers "can the model handle this one ticket?" A product has to answer a harder question: can the system handle the distribution of tickets it will actually see, inside a defined cost and latency budget? A single polished transcript carries no information about rare intents, long prompts, traffic bursts, retries, or the fraction of cases that need a human.

That's the job of a capability contract: a document that names the target behavior, the traffic slices it has to hold up under, quality and safety gates, a latency objective, a cost ceiling, a privacy boundary, a fallback, and explicit non-goals. A scenario version of one might look like this:

quality_gate: "All deterministic contract fixtures must pass."
p95_latency_ms: 800
availability_percent: 99.5
cost_ceiling_per_request_usd: 0.35
privacy_boundary: "synthetic_data_only"
fallback: "human_billing_specialist"

These numbers are scenario targets you'd set for your own product, not universal defaults — the point of writing them down is that they can fail, and a failing number tells you something a demo never could.

A worked scenario shows why the escalation rate matters more than the model price

Here's where the temptation to invent a dramatic cost story is strongest, and where it's most important not to. Instead, consider the shape of a cost-envelope calculator built around exactly this kind of support-ticket workload, which turns traffic, token, retry, and human-escalation assumptions into three scenarios:

These are synthetic scenario outputs from dated fixture assumptions — token mixes, retry rates, escalation rates, and labor cost — not a vendor quote or a production measurement. What makes the exercise useful isn't the dollar figures; it's what the components reveal. In the expected scenario, the modeled teacher API charge is only 204.75. The modeled human-escalation cost is 12,500 — the term that actually drives the $12,704.75 total. Halving the model's per-token price would barely move that number. The lever that matters is understanding why five percent of cases escalate to a person, not shopping for a cheaper model.

The expected scenario's 0.2541 per request clears an assumed 0.35 ceiling; the high scenario's $0.5073 does not. That tells you the high scenario fails a cost threshold — it doesn't tell you anything about the other envelope constraints in that same scenario, and it doesn't mean distillation is the fix. It means escalation policy and routing are the first thing to investigate, before anyone commits to training a student.

That fixture includes human handling, while the calculator below isolates a narrower question: after product policy and escalation are fixed, can a student's serving path carry the traffic and repay the one-time distillation program?

Change the traffic, token mix, provider rates, measured student throughput, redundancy, and amortization period. The result only favors the student when both the cost crossing and the capacity constraint hold.

The distill-or-not gate: seven questions before you write a training script

Distillation is attractive precisely because it hands you a new artifact you get to shape. That's also why teams reach for it before checking whether a cheaper path already clears the bar. A product can be underperforming because the prompt is vague, because current facts are missing (which retrieval-augmented generation, or RAG — combining a model's parameters with an explicit external memory, solves without training anything new — Lewis et al.), because a deterministic rule got disguised as a language-understanding problem, or because traffic is too small to repay the fixed cost of a training program. None of those require a student model.

Ask, in order:

1. Is the target behavior named narrowly enough to test by traffic slice?
2. Does the current system fail a quality, latency, cost, privacy, availability, or control boundary?
3. Can prompting, retrieval, deterministic code, or an existing hosted model repair that failure with less fixed work?
4. Is the behavior repeated often enough that lower marginal cost could repay student creation and refresh?
5. Can unsafe or unsupported cases be detected and routed to a person or a router — a component that selects a destination for each request — or a cascade, which tries one path and escalates only when a gate says the result is insufficient?
6. Will the student be measured against independent product rules, not just agreement with the teacher?
7. What evidence would cancel the project before a single row of teacher data gets collected?

The seventh question is the most economical one on the list, because it's the one most teams skip. Sunk engineering time is not future savings, and an architecture decision record that never named an exit condition can't tell anyone when to stop.

Seven paths solve "repeated behavior," and only one of them is a student

Distillation competes with real alternatives, and the honest comparison names the failure mode each option is prone to, not just its headline benefit.

A hosted small model and a distilled student look similar on paper — both are "a smaller model" — but they answer different questions. The former may already satisfy your envelope with no custom training program at all. The latter is worth its fixed cost only when the specific capability contract your product needs doesn't exist anywhere off the shelf.

What could go wrong

• Presenting a modeled training estimate as an invoice. Epoch AI's own 2.4×-per-year figure ships with a confidence interval for a reason — stripping that away and quoting it as a hard number misrepresents the source.
• Comparing an API's marginal price against only the student's hardware bill. The honest comparison also includes data collection, training, evaluation, incident response, and refresh cost — the fixed side of the ledger that a "GPU is cheaper than API tokens" argument conveniently skips.
• Freezing a volatile price table into a permanent business case. Cursor's Tab completion model reportedly handles more than 400 million requests a day at a workload scale that justifies its own dedicated economics — a number that belongs to Cursor's traffic, not a transferable rate for yours. Record the access date on every price you cite, and set a review date on the whole comparison.
• Optimizing the model price while the escalation rate does all the damage. The worked scenario above makes this concrete: a cheaper model barely moves a bill dominated by human handling cost.

A real production counter-example is worth naming precisely because it's rare: Character.AI's engineering team reports cutting inference serving cost by at least 33 times since their 2022 launch, driven substantially by moving high-volume traffic onto smaller, purpose-built models. That result belongs to Character.AI's traffic profile and model family — it's evidence the trade-off can work at scale, not a rate you should expect to reproduce without your own measurement.

Key takeaways

• A frontier model's price reflects the whole warehouse of capability it carries, not the narrow task you're actually asking it to do.
• Prefill and decode are different computational phases with different bottlenecks — the KV cache, not parameter count, is often what actually limits your tail latency.
• Scaling laws and Chinchilla-style corrections describe resource allocation for training a teacher. They say nothing about whether your product's specific behavior transfers safely to a smaller model.
• A capability contract — quality gate, latency objective, cost ceiling, privacy boundary, fallback — turns "the demo worked" into something you can actually fail, which is the point.
• Run the seven-question gate before writing a training script. The question most teams skip — what evidence would cancel this project — is the one that saves the most money.
• Compare the full ledger: fixed data and training cost against marginal serving savings, not just today's API line item against a GPU quote.

What's next

The next post in this series opens the mechanism distillation actually depends on: dark knowledge, the structure hidden inside a teacher's full probability distribution over possible answers, and why matching it — carefully, and only where it earns its cost — teaches a student more than copying its final answer ever could.

This post adapts material from Chapter 1, "The Genius in the Warehouse," in Distilled: The Engineering of Small, Fast, Cheap AI Models. The rest of the series is indexed at the series page.

Sources and References

Training cost and scaling

• Cottier et al., "How much does it cost to train frontier AI models?" — Epoch AI
• Brown et al., "Language Models are Few-Shot Learners" (GPT-3)
• Kaplan et al., "Scaling Laws for Neural Language Models"
• Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla)

Serving mechanics

• Vaswani et al., "Attention Is All You Need"
• Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention"

Distillation and alternatives

• Hinton, Vinyals, and Dean, "Distilling the Knowledge in a Neural Network"
• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"

Production case studies

• OpenAI, "Pricing" (accessed 2026-07-31)
• Cursor, "Improving Cursor Tab with Online RL"
• Character.AI, "Optimizing AI Inference at Character.AI"