LLM Long Context: Storage, KV Cache, and Retrieval

A model or application programming interface (API) raises its context limit from 128K to one million tokens. Product requirements change before anybody measures the workload:

• load the whole repository;
• retain every tool result;
• stop retrieving documents;
• replace summaries with raw history.

Think of context as a storage system with four layers: the logical record a request may address, the text selected for this call, the model state derived from that text, and the physical memory holding the state. A larger token limit widens the first boundary. It does not automatically enlarge the other three.

The larger number answers an interface question. It does not tell you how many requests fit, which prefix, an exact starting token sequence, survives eviction, removal to free cache space, whether retrieval finds the right passage, what billing does on a cache miss, or whether the model uses a fact buried among distractors.

A context window says how much input and reserved output a request may address. Attention or recurrent state, a fixed-size record carried forward, represents that context in memory. Serving runtimes allocate and cache the state; retrieval selects external passages, while compaction rewrites or removes history before it reaches the model.

The interactive lab below models representation bytes, ideal exact-prefix sharing, and page rounding. It does not simulate a runtime's scheduler, eviction trace, kernels, or answer quality.

Start with the defaults, then change one input at a time. Keep the widget's four outputs separate: logical bytes per token (or total fixed state at the selected concurrency), total physical modeled state, prefix bytes avoided, and page-rounding waste. They use different denominators. If one number changes, explain which layer moved before you treat it as a capacity improvement.

Six context numbers answer six different questions

Use separate names:

Configured context is not useful context. The Lost in the Middle study measured position effects in long inputs. RULER tested retrieval, aggregation, and multi-hop tasks and showed why nominal length alone is not a quality guarantee.

Write useful context as a measured function:

useful_context(task, input_distribution, quality_threshold)

There is no honest universal percentage of the advertised window.

Stored attention state grows linearly with retained tokens

During autoregressive generation, producing one token at a time, a standard transformer keeps a key/value (KV) cache, the prior keys and values needed so each new token does not recompute the full prefix.

For:

• T: cached tokens;
• L: cache-bearing layers;
• H_kv: key/value heads;
• d_k, d_v: key and value dimensions;
• s_k, s_v: bytes per key and value element;

the logical payload is:

KV bytes =
T × L × H_kv × (d_k × s_k + d_v × s_v)

When key and value dimensions and data types (dtypes) match:

KV bytes/token =
L × H_kv × (d_k + d_v) × s

Take the defaults used by the lab:

L     = 32
H_kv  = 8
d_k   = 128
d_v   = 128
s     = 2 bytes
 
bytes/token
= 32 × 8 × (128 + 128) × 2
= 131,072 bytes
= 128 KiB

At 131,072 cached tokens:

131,072 tokens × 131,072 bytes/token
= 17,179,869,184 bytes
= 16 GiB per request

Eight such requests need 128 GiB of logical KV before page rounding, runtime metadata, workspaces, or fragmentation.

Here is the deployment decision hidden inside that arithmetic. In a clearly hypothetical 80 GiB graphics processing unit (GPU), 48 GiB of weights and 8 GiB of runtime reserve leave 24 GiB for KV. One 16 GiB request fits; two do not. “Supports 128K context” is not the same claim as “serves two 128K requests concurrently.”

KiB and GiB are binary units: 1 KiB is 1,024 bytes, and 1 GiB is 1,073,741,824 bytes.

The Transformer paper defines full attention. Its pairwise attention-score work grows roughly with the square of prompt length, while retained KV payload grows linearly. Cache bytes and prefill compute are separate capacity problems.

Shared KV heads reduce storage, not every serving cost

Multi-Head Attention (MHA) stores separate key/value heads for its query heads.

Multi-Query Attention (MQA) shares one key/value head across query heads. The MQA paper targets incremental-decoding memory bandwidth.

Grouped-Query Attention (GQA) uses several shared KV groups. The GQA paper describes the intermediate design between MHA and MQA.

For 32 query heads:

The ratios assume equal layer count, dimensions, and cache dtype. They are storage calculations, not quality or throughput predictions.

Changing a trained MHA model into MQA is not a page-allocation flag. It changes the architecture or requires a conversion and training procedure.

Optional senior depth: if you are sizing a conventional MHA, GQA, or MQA service, skip the next two sections and resume at paging. Read them when the model stores a compressed latent record or carries a fixed-size recurrent state.

Latent attention helps only when the runtime keeps it latent

Multi-Head Latent Attention (MLA) stores a lower-rank latent representation plus positional key state. The DeepSeek-V2 paper describes the design.

For latent width d_c, positional-key width d_R, and L_mla layers:

MLA bytes/token =
L_mla × (d_c × s_c + d_R × s_R)

Against same-dtype MHA:

MLA/MHA ratio =
(d_c + d_R) / (H_q × (d_k + d_v))

That is an architecture-level logical record. A runtime may retain the latent form, materialize expanded values, quantize the cache, or add alignment and metadata. Pin the model and runtime before turning the formula into resident bytes.

Kimi K3 is a useful hybrid example. Its pinned config lists 69 Kimi Delta Attention (KDA) layers and 24 full-attention layers. The SGLang launch engineering report describes fixed-size KDA state beside token-growing MLA state. One request can therefore pressure two allocators with different units.

Recurrent state trades random access for a fixed-size record

Kernelized linear attention can carry forward a model-defined state:

S_t = S_(t-1) + phi(k_t) v_t^T
z_t = z_(t-1) + phi(k_t)
 
output_t =
phi(q_t)^T S_t
/ phi(q_t)^T z_t

The Transformers are RNNs paper develops this recurrence.

For feature width r, value width d_v, and equal state precision:

state bytes =
concurrency
× layers
× heads
× (r × d_v + r)
× bytes/element

The state does not grow with sequence length. It still grows with concurrent requests.

Fixed state is not lossless storage of every prior token. It is the state that this recurrence carries into the next step. Interference, forgetting, and recoverability become model and task questions.

Paging changes allocation, not logical bytes per token

The junior path resumes here. A page is a fixed-size block of token state, similar to a fixed-size bin in a warehouse. The final bin may be partly empty, but the allocator still reserves the whole bin.

The PagedAttention paper maps logical KV blocks to non-contiguous physical blocks. This avoids reserving one maximum-size contiguous region per request and enables block-level sharing.

With page width P:

allocated_tokens/request =
ceil(actual_tokens / P) × P

For 1,001 retained tokens and 16-token pages:

ceil(1,001 / 16) × 16
= 63 × 16
= 1,008 allocated token slots

The final page wastes seven slots. Waste stays below one page for each independently rounded sequence or cache group.

Paging can provide:

• non-contiguous allocation;
• block-level sharing;
• copy-on-write, sharing a block until one request changes it;
• lower maximum-length reservation waste;
• block-granular eviction.

It does not shrink the logical representation of one retained token.

Page size is a runtime and workload decision. Larger pages reduce block-table metadata and may suit kernels; they increase tail-page waste and reduce partial-prefix granularity.

Prefix caching is a prefill cache with exact identity rules

An exact-prefix cache reuses derived state for matching prefix blocks. It can reduce repeated prefill work and time to first token (TTFT). It does not accelerate the generation of new output tokens.

The distinction is explicit in vLLM's automatic prefix caching snapshot: Automatic prefix caching (APC) skips computation for matching prompt blocks but does not reduce decode time.

Change an early token, tokenizer version, chat template, adapter, media input, or other hash input, and later blocks no longer identify the same prefix.

One runtime snapshot shows why version pinning matters

At vLLM commit 6c7e679 from 2026-07-28, the prefix-cache design documents:

These are vLLM snapshot facts, not universal cache laws. Another runtime or release may use radix trees, different block sizes, different eviction, or a different security boundary.

Measure reused tokens, not only hit requests. A one-token match and a 100K-token match should not count as equivalent wins.

Eviction, compaction, retrieval, and recurrence lose different things

Calling all four “memory management” hides the decision.

Retrieval-augmented generation (RAG) combines parametric generation with external document memory. The original RAG paper separates those stores.

Retrieval can fail before generation:

• the index is stale;
• chunking splits jointly needed evidence;
• metadata filters remove the right document;
• ranking misses it;
• too many passages add distractors;
• conflicting versions are not resolved.

Track retrieval and answer metrics separately. High recall with a wrong answer is not success. A correct answer without required evidence may also fail the product contract.

Choose the policy from the failure you can tolerate

Long context and retrieval are not opponents. A retrieved passage still consumes context, and a large context can still benefit from retrieval that selects the right version and preserves citations.

Token economics includes misses and recomputation

Hosted APIs use different and changeable price categories. Do not hard-code a universal “cached token discount.” Record the dated contract and calculate:

hosted_request_cost =
(
  uncached_input_tokens × input_price
  + cache_write_tokens × cache_write_price
  + cache_read_tokens × cache_read_price
  + output_tokens × output_price
) / price_unit

If a provider has no separate write category, fold that term into its documented input category. price_unit is the provider's quoted token unit, often 1,000,000 tokens. Keep currency, region, service tier, and date beside the numbers.

For self-hosting, count both GPU time and central processing unit (CPU) time:

cost_per_success =
(
  GPU_seconds × blended_GPU_rate
  + CPU_seconds × CPU_rate
  + network_bytes × network_rate
  + storage_byte_seconds × storage_rate
  + reserved_idle_cost
) / successful_requests

Cache eviction increases prefill compute. Offload increases transfer work. Failed and cancelled requests still consume resources. Cost per accepted token can look good while cost per successful task gets worse.

Offload creates a hierarchy with slower failure modes

Moving KV blocks or recurrent checkpoints from high-bandwidth memory (HBM), the GPU's local memory, to host memory, remote memory, or storage increases apparent capacity. It also adds:

• transfer and registration latency;
• bandwidth and queue pressure;
• placement metadata;
• stale or partial transfer handling;
• reclamation and admission policy;
• another timeout and recovery boundary.

Capacity did not become free. It moved to a slower tier.

The operating-system analogy is useful: a large virtual address space does not make every page resident or equally fast.

Prefill and decode are different workloads

Prefill processes prompt tokens and creates model state. Decode produces new tokens while repeatedly reading weights and prior state. Inter-token latency is the delay between those generated tokens.

Separating them can isolate queues and permit phase-specific hardware or parallelism. It also introduces KV transfer, duplicate weight pools, layout compatibility, backpressure, and a new failure boundary.

At vLLM commit 6c7e679, the disaggregated-prefill documentation marks the feature experimental and states that it does not improve throughput. Its intended benefit is separate TTFT and inter-token-latency control.

The latency path becomes:

prefill queue
+ prefill compute
+ KV transfer
+ decode queue
+ first decode step

Disaggregation moves interference. It does not remove work.

Concurrency is a trace, not an average

For homogeneous paged KV:

bytes/request =
ceil(tokens / page_tokens)
× page_tokens
× bytes_per_token

Then a rough idealized ceiling is:

concurrent_requests <=
cache_pool_bytes / bytes_per_request

This estimate is useful for a first rejection and dangerous for capacity approval.

Production traces include:

• long-tailed prompt and output lengths;
• shared-prefix groups;
• cache retention after request completion;
• eviction and recomputation bursts;
• fixed recurrent slots beside token-growing KV;
• cancellation;
• priority classes;
• prefill/decode transfer queues.

Replay a timestamped trace against an explicit admission and eviction policy. Report p50, p95, and p99 percentiles, the values below which 50%, 95%, and 99% of observations fall, rather than only a mean request.

Observability must preserve the layer boundaries

The vLLM metrics snapshot also warns through its deprecation policy that metric names change across versions. Pin dashboards and alerts to the runtime release.

The workshop config is an experiment contract

Before running the workshop, predict which metric should move in each condition. For example, a warm exact prefix should reduce TTFT and recomputed tokens, but it should not reduce decode time for newly generated tokens.

The download specifies architecture fields, page size, cache pool, context lengths, concurrency, prefix reuse, retrieval k (the number of selected passages), and required metrics. It does not include a model, corpus, retrieval index, serving runtime, or 100 gold-labelled questions.

Supply those fixtures, then compare:

Required demonstrations:

1. change one early token and record the prefix miss;
2. compare TTFT and inter-token latency after a warm prefix;
3. change page size and record rounding and reuse granularity;
4. force eviction with interleaved prefixes;
5. compact a history and list facts that cannot be reconstructed;
6. vary retrieval depth and compare recall, citations, and answer quality;
7. compare many short recurrent requests with fewer long KV-heavy requests.

Keep serving, retrieval, task, and cost results in separate tables. A gain in one is not evidence of a gain in the others.

After the run, explain every gain twice: first in storage terms—what was kept, moved, or recomputed—and then in product terms—what happened to latency, cost, citations, and answer quality. If you cannot make both explanations, the lab has not established a production decision.

What could go wrong

• Configured context is reported as useful context.
• KV payload and prefill compute are treated as one complexity.
• Paging receives credit for shrinking logical bytes per token.
• Hit requests replace reused-token counts.
• LRU retention is called semantic memory.
• Cross-tenant cache reuse lacks an isolation key and threat model.
• Summary quality is measured only by token reduction.
• Retrieval recall is reported without answer and citation quality.
• MQA, GQA, or MLA storage ratios become quality claims.
• Recurrent state is described as lossless history.
• Offload capacity ignores queueing and transfer failure.
• Average context length hides p99 concurrency collapse.
• Disaggregated prefill ignores duplicate weights and KV transfer.
• A pricing calculation omits cache misses, failed requests, or idle reserve.

Long context is not one feature. It is a stack of representation, allocation, retention, selection, transfer, and model-use decisions.

Ask which state is stored, where, for how long, at what cost, and with what measured task value. “How many tokens?” comes after that.

Workshop four places weights and context state on physical devices, then maps device-to-device communication onto local GPU links, CPU locality, and the network. The course roadmap keeps the logical state model separate from that physical placement problem.

Sources and references

Representation and useful context

• Transformer
• Multi-Query Attention
• Grouped-Query Attention
• DeepSeek-V2 and MLA
• Transformers are RNNs
• Lost in the Middle
• RULER

Allocation, retrieval, and runtime behavior

• PagedAttention
• Retrieval-Augmented Generation
• vLLM automatic prefix caching snapshot
• vLLM prefix-cache design snapshot
• vLLM disaggregated prefill snapshot
• vLLM metrics snapshot
• Kimi K3 pinned config
• SGLang Kimi K3 serving report