LLM Compression Decision Matrix: Let the Bottleneck Pick the Technique

You inherited a fine-tuned 8B-parameter model that scores well on every offline eval. Then someone asks when it ships to the edge fleet, or into the p95-under-200ms tier of the API, or onto a GPU budget that just got cut in half. You start reading about quantization, pruning, distillation, LoRA, and mixture-of-experts, and the honest answer to "which one should I use" turns out to be "it depends which resource is actually choking your service" — a question most comparison posts skip straight past.

That's the mistake worth naming up front: these five techniques aren't five contestants for the same title. They change different resource terms, and a change that helps one term can leave another untouched, or make it worse.

Find the bottleneck before you pick a technique

A model has more than one size. Artifact bytes are what you store or download — the checkpoint on disk. Resident memory is what the serving process actually occupies once it's running: weights, adapters, runtime buffers, and the KV cache, short for key-value cache, which stores attention keys and values for every token in every active sequence and grows with context length, layer count, precision, and concurrency. A checkpoint can shrink on disk and still blow your memory budget if the KV cache is what's binding.

Speed splits the same way. Memory bandwidth is how fast bytes move from HBM to the compute units; on autoregressive decode you're bandwidth-bound most of the time, because you re-read the whole weight set for every new token. Active compute is the arithmetic for one request — the term that dominates long-prompt prefill, the pass that processes the input before generation starts. Communication is data crossing devices or experts. And latency isn't one number: tail latency (p95, p99) describes your slow requests, and it's usually those tail requests that page someone.

Training has its own footprint — peak training memory, accelerator-hours, teacher-generation tokens — and none of it automatically converts into a serving win. There's a fifth column that's easy to forget entirely: control-plane cost. Routers, adapter registries, kernel pinning, and rollback automation cost engineering time even after the model itself gets cheaper.

The first useful move isn't picking a technique. It's writing the constraint down as a measurable inequality: which of these terms is actually over budget, on your hardware, under your workload, right now? A minimal version of that check looks like this:

components = {
    "weights_mib": to_mib(weight_bytes),       # ①
    "kv_cache_mib": to_mib(kv_cache_bytes),    # ②
    "adapters_mib": to_mib(adapter_bytes),
    "runtime_mib": to_mib(runtime_overhead),   # ③
}
binding = max(components, key=components.__getitem__)

① Weight bytes come straight from parameter count and precision — the term everyone reaches for first. ② KV-cache bytes depend on sequence length, layer count, head count, precision, and concurrent requests — on a chatty production workload this is very often the term that actually binds, not the weights. ③ Runtime overhead — workspaces, allocator reserve, communication buffers — is the term most back-of-envelope estimates forget entirely.

That last line, max(components, ...), is the whole point of this post. It isn't a benchmark; it's a habit. Pick the largest measured component, not the one you assumed, and let that pick your technique. It's the same discipline behind Meta's own account of building the lightweight Llama 3.2 models: the team applied single-shot structured pruning to a Llama 3.1 8B model, then used logits from the 8B and 70B models to recover performance through distillation — pruning first because the binding constraint was architecture size, distillation second to repair what pruning cost, rather than reaching for a generic "compression pipeline" (Meta's Llama 3.2 announcement). NVIDIA's Minitron work reports a similar pattern — pruning by depth or width followed by distillation and alignment to recover compact models — as one more existence proof, not a universal ordering law (Minitron).

The five techniques move five different resource terms

Three traps in that table matter more than the rest. A tiny LoRA adapter file says nothing about resident memory, because the base stays loaded. An MoE's active-parameter count says nothing about tail latency, because dispatch and imbalance sit outside the FLOP count. And pruning's stored-weight reduction says nothing about speed until you've confirmed the runtime actually has a kernel for the resulting shape.

A worked ladder shows why a smaller file isn't a faster service

Here's a fixture-level example that puts numbers on those traps. It's a deterministic test case, not a benchmark you should quote in a launch doc — but the shape of the result generalizes past the fixture:

Two things happen at once here, and both are the point. prune30 cuts the artifact by roughly 30%, and resident memory and latency don't move at all — the fixture runtime has no sparse kernel, so it still reads and multiplies the dense layout, zeros included. adapter-r8 produces a file that's smaller by three orders of magnitude, and resident memory goes up, not down, because the base model is still fully loaded underneath it. Only int4 actually moves the binding resource: −75% on artifact size, −35% on p95 latency, and a −2 percentage-point quality cost that lands exactly at the fixture's 0.90 floor.

If you only measured artifact bytes, all three non-baseline profiles would look like wins. Measure resident memory and latency on the target runtime, and two of them are illusions.

Composition beats any single technique — with one real caveat

Llama 3.2's prune-then-distill sequence and QLoRA's quantize-then-adapt sequence are both compositions, and both work because each stage has a clear, separately measurable job. QLoRA specifically keeps a frozen four-bit base resident during training and backpropagates through it into LoRA adapters, which is how it fits a 65B-parameter fine-tune onto a single 48GB GPU while preserving full 16-bit fine-tuning performance in the paper's reported setting (QLoRA). In code, that recipe looks like this:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, TaskType, get_peft_model
 
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,               # ①
    bnb_4bit_quant_type="nf4",       # ②
    bnb_4bit_compute_dtype="bfloat16",
)
 
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quant_config,
    device_map="auto",
)
 
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],   # ③
    lora_dropout=0.05,
    task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)

① Loading in four-bit is what makes the 65B-on-one-GPU claim possible — it's the memory move, not the adapter. ② nf4, short for NormalFloat4, is the QLoRA paper's information-optimal data type for weights that are roughly normally distributed, which is why it beats a naive four-bit integer format at the same bit width. ③ Targeting only the attention projections keeps the adapter small; widen target_modules only if the narrower set underperforms on your eval.

The caveat: composition only works if every stage gets its own before/after gate. The moment you change pruning, quantization, and adapters in one release, you lose the ability to say which change caused which regression — and a recovery run can pass every gate you wrote while a slice you didn't check quietly gets worse.

What could go wrong

• The quantization cliff is real below roughly four bits. AWQ's fix is to identify and protect the small fraction of salient weight channels — on the order of 1% — using activation magnitude rather than weight magnitude, precisely because naive rounding at that width degrades unevenly across channels (AWQ).
• Unstructured pruning without a matching kernel buys you nothing. SparseGPT can push one-shot pruning to high sparsity ratios with minimal reported accuracy loss, but the serving gain still depends entirely on whether the runtime realizes that sparsity pattern instead of walking a dense layout (SparseGPT).
• MoE routing imbalance shows up in the tail, not the average. The Switch Transformer paper names routing complexity, communication cost, and training instability as central adoption problems for sparse expert models — a single overloaded expert can dominate p99 while average active compute looks fine (Switch Transformer).
• Distilling repeatedly on model-generated data risks compounding degenerate patterns. Shumailov et al.'s "model collapse" study demonstrates this failure mode at a broader scale than any single distillation pipeline, and it's the reason a distillation curriculum needs real-data anchoring, not just successive rounds of synthetic generation (Nature, 2024).
• A shared base model updated after adapters were trained can silently invalidate every adapter trained against it. This one isn't from a paper — it's an operational risk worth naming plainly: version-pin the base a LoRA adapter targets, and treat a base update as a reason to re-validate every adapter, not just re-deploy them.

Pick fast, verify slower

Match the binding resource to the shortlist, then verify on target hardware before you believe any of it:

1. Resident memory or bandwidth binding → quantization first. It's the only technique here that reliably moves the term without an architecture change.
2. Domain accuracy binding, model too generic for the task → distillation from a stronger teacher.
3. Fine-tuning memory binding, not serving memory → LoRA or QLoRA.
4. Dense compute binding, and the runtime has a matching sparse or shape-aware kernel → structured pruning. If it doesn't have that kernel, skip pruning entirely rather than banking on a future one.
5. Large aggregate capacity needed against a tight active-compute budget, and you can absorb dispatch complexity → MoE.

None of these five sits at the end of the list because it's worse in the abstract. They sit where they do because their mechanism only pays off against a specific bottleneck — and the bottleneck is something you measure, not something you guess.

Key takeaways

• Compression techniques change different resource terms; they aren't ranked contestants on one leaderboard.
• Name the binding resource — artifact bytes, resident memory, bandwidth, compute, latency, training footprint, or control-plane cost — before touching a technique.
• A technique's headline numbers are boundary conditions, not universal guarantees: DistilBERT's 97%-at-40%-smaller and LoRA's 10,000×-fewer-trainable-parameters figures are tied to specific models and tasks, not laws of compression.
• Composition (prune-then-distill, quantize-then-adapt) usually beats any single technique, but only if every stage gets its own before/after gate.
• Re-run quality and safety checks after every artifact-changing step — an aggregate score can hide a regression on exactly the slice that matters.

What's next

The next post in this series picks up right where a real compression pipeline's behavioral stage sits — the step between a stable distilled student and the quantization pass covered above — and shows how to turn that student's own mistakes into an on-policy DPO training signal, verified against the original objective and Hugging Face's TRL trainer.

This post adapts material from Chapter 6, "The Compression Toolbox," in Distilled: The Engineering of Small, Fast, Cheap AI Models. The rest of the series is indexed at the series page.

Sources

Papers and primary research

• Meta's Llama 3.2 announcement
• Minitron: Compact Language Models via Pruning and Knowledge Distillation
• Hinton et al., Distilling the Knowledge in a Neural Network
• DistilBERT
• GPTQ
• AWQ
• SparseGPT
• Wanda: A Simple and Effective Pruning Approach for Large Language Models
• LoRA
• QLoRA
• Mixtral of Experts
• Switch Transformers
• FrugalGPT
• PagedAttention (vLLM)
• Shumailov et al., AI models collapse when trained on recursively generated data (Nature, 2024)

Documentation

• Hugging Face Transformers quantization overview