vLLM, from the inside out — and what happens when attention goes linear.
vLLM is, at its heart, an operating system for tokens: it took virtual-memory paging from OS design and applied it to the KV cache. Every major piece of the engine — the block tables, the scheduler, prefix caching, preemption, even speculative decoding — leans on one quiet assumption: a request's memory grows one token at a time, append-only, forever readable. Linear attention breaks that assumption on purpose. This page builds vLLM up piece by piece, then shows exactly where linear attention collides with it, and exactly how the engine was rebuilt so the two could coexist.
The shape of the problem vLLM was built to solve
You can't understand any of vLLM's design decisions until you understand the two-phase structure of LLM inference, and why the KV cache — not compute — is what actually limits a serving system.
Prefill and decode are two different workloads wearing one trench coat
Generating from a decoder-only transformer has two phases with opposite performance characters. Prefill processes the entire prompt in one pass. Every layer runs big matrix multiplications over thousands of tokens at once, so the GPU's tensor cores are saturated — prefill is compute-bound, and its cost grows quadratically with prompt length inside the attention layers (every token attends to every earlier token). Decode then produces the response one token at a time. Each step is a forward pass for a single token per sequence: the arithmetic is tiny, but the GPU must still stream all model weights plus the entire accumulated KV cache from HBM to compute units. Decode is therefore memory-bandwidth-bound — the GPU spends most of its time waiting on memory, with compute units mostly idle.
This asymmetry drives everything. If decode is bandwidth-bound and one sequence can't saturate the GPU, the obvious fix is batching: run many sequences' decode steps in the same forward pass, reusing each weight-load across the whole batch. Throughput rises almost linearly with batch size — until you run out of the one resource batching consumes: memory for KV caches.
What the KV cache actually is
In causal attention, token t computes attention over the keys and values of every token ≤ t. Those keys and values don't change once computed, so recomputing them every step would waste a quadratic amount of work. Instead, each layer stores each token's K and V vectors after computing them once. That store is the KV cache. Its size per token is fixed by architecture:
Plug in real models: Llama-3-8B (32 layers, 8 KV heads via GQA, head dim 128, fp16) needs 2×32×8×128×2 = 128 KiB per token — about 1 GiB per sequence at an 8K context. The vLLM paper's example, OPT-13B (40 layers, 40 full heads, head dim 128), needs ~800 KiB per token: a single 2K-token sequence eats 1.6 GiB. And critically, the cache grows during generation and its final size is unknown in advance — you don't know how long the model will talk.
Why this is a systems problem, not a model problem
Put the pieces together. Throughput requires large batches; large batches require holding many KV caches simultaneously; each cache grows unpredictably; and GPU memory left over after weights is finite. The KV cache is the scarce, dynamically-sized, per-request resource — exactly the profile of a problem operating systems solved fifty years ago for process memory. vLLM's founding observation (Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023) was that pre-2023 serving systems were managing this resource the way computers managed memory before virtual memory existed — and wasting most of it.
The world before vLLM: contiguous chunks and static batches
Two pre-vLLM defaults — contiguous KV allocation and request-level batching — each silently threw away most of the GPU. Quantifying that waste explains why PagedAttention was worth inventing.
Failure #1 — contiguous allocation and its three kinds of waste
Attention kernels of the era (FasterTransformer and friends) expected each sequence's K and V tensors to live in one contiguous slab of GPU memory. Since output length is unknown, systems pre-allocated each request a slab sized for the maximum possible length (say 2,048 tokens). Three distinct pathologies follow:
Reserved-but-unused
A request that ends at 80 tokens held a 2,048-token slab the entire time. The unused 1,968 slots were locked away from every other request for the request's whole lifetime.
Internal fragmentation
Even the used region over-provisions: memory reserved ahead of the current position is dead until (unless) generation reaches it. Waste scales with max_len − actual_len.
External fragmentation
Differently-sized contiguous slabs leave gaps between them — free memory that exists but is too scattered to fit a new request's slab. Classic malloc pain, on an 80 GB heap.
The SOSP paper measured the combined effect: in existing systems, only 20.4%–38.2% of KV cache memory held actual token state. The majority of the most precious resource on the GPU stored nothing. On top of that, contiguous slabs make sharing impossible: parallel sampling (n answers from one prompt) and beam search duplicated the entire prompt's KV per branch, even though the prompt portion is byte-identical.
Failure #2 — request-level (static) batching
The second default: form a batch of N requests, run them in lockstep until every one finishes, then admit the next batch. Sequences finish at wildly different lengths, so a request that generated 20 tokens sits idle — occupying its batch slot and its memory slab — while a neighbor generates 900. GPU lanes go dark one by one. New requests queue outside no matter how much capacity has effectively freed up. You'll see this animated (against its fix) in section 0x04.
Serving throughput was limited not by FLOPs, and not even by raw memory capacity, but by memory management policy. Fix allocation granularity and fix scheduling granularity, and the same GPU serves several times more traffic. Those two fixes are, respectively, PagedAttention (0x03) and continuous batching (0x04) — the twin pillars of vLLM.
PagedAttention: virtual memory for the KV cache
One idea, borrowed whole from operating systems: stop storing a sequence's cache contiguously. Chop it into fixed-size blocks, scatter them anywhere in a shared pool, and keep a per-request page table that maps logical order to physical location.
The mechanism, precisely
- Blocks. GPU memory reserved for KV is carved into equal physical blocks, each holding the K and V vectors of a fixed number of tokens —
block_size, default 16. A block is the unit of allocation, the "page". - Logical blocks. Each request's cache is a sequence of logical blocks: block 0 holds tokens 0–15, block 1 holds 16–31, and so on. Logical order is just the token order.
- Block table. Per request, a small array mapping logical index → physical block ID (plus a fill count for the last block). This is exactly a per-process page table. Physical blocks backing one request can be scattered arbitrarily across the pool.
- On-demand allocation. A new physical block is grabbed from a free list only when generation crosses a 16-token boundary. Nothing is reserved for a future that may not happen.
- The kernel side. Attention math doesn't change — but the kernel can no longer assume contiguous K/V. vLLM ships custom kernels (
paged_attention_v1/v2, and today paged variants inside FlashAttention and FlashInfer) that take the block table as an argument and gather keys/values block-by-block while computing online softmax. The cost of the indirection is small and is the price of everything below.
What the paging buys, item by item
Fragmentation collapses to a rounding error. Internal waste is bounded by less than one block per sequence — at most 15 unfilled slots in the final block, a few hundred KB instead of gigabytes. External fragmentation is zero by construction: every allocation is the same size, so any free block satisfies any request. Reserved-but-unused waste disappears because nothing is reserved. Measured KV memory utilization went from ~20–40% to near 96%+, which directly converts into batch size, which converts into throughput — the paper reported 2–4× throughput over FasterTransformer and Orca at equal latency, with bigger gains for longer sequences.
Sharing becomes a pointer operation. Because the map from logical to physical is per-request, two requests may map to the same physical block. Each physical block carries a reference count. This unlocks:
- Parallel sampling — n samples from one prompt map their prompt blocks to the same physical blocks (refcount = n). Only divergent generated tokens allocate new memory.
- Copy-on-write — when a sequence must append into a block with refcount > 1 (a shared, partially-filled last prompt block), vLLM copies that one block, decrements the original's count, and writes into the private copy. Block-granular COW, exactly like fork().
- Beam search — beams fork and die constantly; with paging, a surviving beam shares all ancestor blocks and abandoned beams just decrement refcounts. Memory cost of beam search drops massively versus slab-copying.
- Prefix caching — shared system prompts across different requests can be deduplicated the same way. This grows into a full feature in section 0x06.
Everything above works because KV entries are append-only and immutable: written once when their token is processed, then only ever read. Immutability is what makes sharing safe, refcounts sufficient, copy-on-write meaningful, and any prefix of the cache forever reusable. Hold that thought.
Continuous batching and the scheduler
Paging fixed where token state lives. The scheduler fixes when work happens: decisions are made every single model step, not once per batch.
Iteration-level scheduling
The idea (introduced by Orca, OSDI 2022, and adopted then extended by vLLM) is called continuous batching: the "batch" is re-formed at every forward pass. The moment a sequence emits its end-of-sequence token, it leaves the batch and its blocks return to the pool on that very step; a waiting request is admitted into the freed capacity immediately. No GPU lane ever idles waiting for a slow neighbor. Combined with paging (which makes admission possible whenever any blocks are free, not whenever a contiguous slab is free), this is where vLLM's throughput comes from.
Inside the V1 scheduler: one loop, one currency
vLLM's original (V0) scheduler kept prefill and decode as separate request states with separate scheduling paths. The V1 engine rewrite (rolled out through 2025, now the only engine) deleted that distinction. The scheduler thinks in a single currency — tokens to process this step — and emits a plain mapping like {reqA: 512, reqB: 1, reqC: 1}: reqA is mid-prefill (chunked), B and C are decoding. There is no "prefill phase" or "decode phase" at the batch level anymore; every step is just a token budget (max_num_batched_tokens, alongside a concurrency cap max_num_seqs) allocated across requests by policy (FCFS or priority).
- Chunked prefill (default in V1). A 100K-token prompt is not processed in one monolithic pass — that would stall every decoding request behind it for seconds, spiking inter-token latency. Instead the prompt is fed in budget-sized chunks, and each step mixes prefill chunks with ongoing decodes in one batch. Compute-heavy chunks and bandwidth-heavy decodes overlap beautifully on the hardware; latency becomes smooth and controllable.
- Admission control. A request is scheduled only if the KV cache manager can supply the blocks its next chunk needs. Memory, not slot count, is the true admission gate.
- Preemption. When the pool runs dry mid-flight (many long generations), the scheduler evicts the lowest-priority running request. Two mechanisms exist: recomputation — free its blocks entirely and, later, re-run its prefill (cheap to trigger, wastes compute; the V1 default) — and swapping — copy its blocks to pinned CPU RAM over PCIe and bring them back later (saves compute, costs transfer time and host memory). Either way the request resumes correctly because its state is reconstructible: for recompute, from its token IDs; for swap, byte-for-byte. Note this reconstructibility — it's another thing Part II will stress-test.
- Speculative & multi-token steps. The V1 scheduler natively handles steps that consume/emit more than one token per request (speculative decoding verification, jump-decoding in structured output), because "n tokens for request X" is already its native vocabulary.
Notice what the scheduler doesn't know: anything about attention math. It reasons about token budgets and block availability. That abstraction boundary is precisely why continuous batching survives the jump to linear attention untouched — while the memory manager underneath it very much does not.
The full machine: a request's life through vLLM V1
Zooming out from the two big ideas to the whole pipeline. This is what actually happens between an HTTP POST and streamed tokens — every component named, in order.
Architecturally, V1 splits vLLM into two processes. Process 0 hosts the API layer and everything CPU-flavored around it (tokenization, detokenization, streaming); Process 1 hosts the EngineCore — the scheduler + model execution busy-loop — kept deliberately lean so the GPU never waits on Python bookkeeping. They talk over ZeroMQ with asynchronous message passing, so detokenizing step n overlaps with computing step n+1. This isolation of the hot loop is one of the quiet reasons V1 is fast.
Entry: API server / LLM class
Online serving enters through an OpenAI-compatible FastAPI server (vllm serve); offline batch jobs use the LLM Python class. Both funnel into AsyncLLM, the process-0 client of the engine.
Chat templates are applied, multimodal inputs preprocessed, sampling params validated here.
Tokenize & enqueue
The prompt becomes token IDs and the request is shipped over ZMQ to the EngineCore, landing in the scheduler's waiting queue with its sampling parameters and any structured-output grammar attached.
Schedule
Each iteration, the scheduler assembles the step: continuing every running request, admitting waiting ones while the token budget and block supply allow, chunking prefills, preempting if cornered. Output: the token-count map from 0x04.
KV cache manager: allocate
For every scheduled request, the KVCacheManager hands out physical blocks from the pool and updates block tables. With prefix caching on (V1 default), it first checks whether needed blocks already exist — hash hit means tokens are skipped, not recomputed (details in 0x06).
Pool size is set at startup: vLLM profiles a worst-case forward pass, sees what's left of gpu_memory_utilization (default 0.9), and carves it all into blocks.
Execute: workers & model runner
The Executor broadcasts the step to one Worker per GPU (tensor/pipeline parallel ranks). Each worker's ModelRunner maintains a persistent batch — input tensors kept resident on-GPU and diff-updated each step rather than rebuilt, a signature V1 optimization — and launches the forward pass.
Forward pass: attention backend
Inside each attention layer, new K/V vectors are written into their block-table slots and the attention backend — FlashAttention-2/3, FlashInfer, Triton, or FlexAttention, chosen per hardware — computes paged attention over scattered blocks. Everything else (MLPs, norms, MoE routing) is ordinary tensor math, compiled and graph-captured (0x06).
Sample
Final-layer logits flow through the sampler: repetition/presence penalties, temperature, top-k/top-p, seeds, logprobs. Structured output applies grammar masks (XGrammar/Guidance) so invalid tokens are literally unpickable. One token (or more, under speculation) emerges per sequence.
Stream back & loop
Token IDs return over ZMQ; process 0 detokenizes incrementally and streams SSE chunks to the client while the EngineCore is already computing the next step. On EOS/stop/max-tokens, the scheduler frees the request's blocks — instantly reusable, possibly retained as cached prefix.
The optimization stack layered on top
Each of these features quietly assumes the paged, append-only KV world. Reading them with that lens now makes Part II's collision obvious later.
Automatic prefix caching (APC)
Every full physical block gets a content hash: hash(parent block's hash, this block's 16 token IDs) — a hash chain, so a block's identity encodes its entire prefix, not just its own tokens. Finished requests' blocks aren't zeroed; they linger in the pool under their hashes (evicted LRU only when space is needed). A new request's prompt is hashed block-by-block; every leading hit means those 16 tokens' KV is mapped, not computed. Multi-turn chat, shared system prompts, and agent loops routinely skip most of their prefill this way. Cost when it misses: a hash lookup — effectively free, hence on by default in V1. Two structural facts to remember: reuse works at 16-token granularity, for any prefix-aligned subset, because each block's KV is independently addressable and immutable.
CUDA graphs & torch.compile
Decode steps are so small that launching their kernels from Python can cost more than running them. vLLM's answer is layered: torch.compile fuses and specializes the model graph, and CUDA Graphs record entire step's kernel sequences for replay at near-zero CPU cost. V1 defaults to piecewise capture — graph the compiled tensor regions, leave attention (whose shapes shift with batch composition) eager — with full-graph modes for uniform decode batches. File away: graph capture dislikes shape-shifting and is sensitive to in-place mutation patterns — both return in Part II.
The rest of the arsenal, compactly
Quantization
Weights: GPTQ, AWQ, FP8, INT8, bitsandbytes shrink the static footprint, freeing pool space. The KV cache itself can be stored FP8, nearly doubling cacheable tokens at slight fidelity cost.
Parallelism
Tensor parallel splits every layer (and each KV head-set) across GPUs; pipeline parallel splits by layer depth; expert parallel spreads MoE experts; data parallel replicates engines. KV blocks shard right along with the attention heads they belong to.
Speculative decoding
A cheap drafter (small model, n-gram, EAGLE, Medusa, or the model's own MTP heads) proposes k tokens; the target model verifies them in one parallel pass. Rejected tail? Just truncate the KV cache back — append-only makes rollback a pointer move. Remember that.
Disaggregation & KV transfer
Prefill/decode disaggregation runs the two phases on separate GPU pools sized to their different natures, shipping KV blocks between them via KV-connectors (NIXL, LMCache…). Works because KV pages are well-defined, serializable byte ranges.
Multi-LoRA
Many LoRA adapters served concurrently over one base model; punica-style batched kernels apply a different adapter per sequence within a single batch.
Structured output
Grammar-constrained decoding compiled to token-level masks (XGrammar), applied in the sampler with near-zero overhead; V1 added jump-ahead decoding for forced token spans.
vLLM = a paged memory system for an append-only, per-token, immutable cache + a scheduler that re-decides the batch every step + a lean two-process executor with compiled, graph-captured kernels + a feature stack (prefix caching, COW sharing, speculative rollback, KV transfer) that all exploit the same property: any token's cached state is an addressable, immutable 16-token-block citizen. Now we break that property.
Linear attention, from scratch
Before the collision, the other party. "Linear attention" names a family — kernelized attention, gated variants, delta-rule variants, and state space models like Mamba — that all share one property: a fixed-size recurrent state instead of a growing KV cache.
Step 1 — remove the softmax
Softmax attention for query t computes similarity exp(qₜ·kᵢ) against every past key — the exponential couples q and k, so nothing can be precomputed independent of the query. Katharopoulos et al. (2020) asked: what if similarity were a plain dot product of feature maps, φ(qₜ)·φ(kᵢ)? Then associativity works magic:
linear: oₜ = φ(qₜ)ᵀ Sₜ / (φ(qₜ)ᵀzₜ), where Sₜ = Sₜ₋₁ + φ(kₜ)vₜᵀ — O(1) per step, O(1) memory
The sum over history collapses into a running matrix S of shape d_key × d_value per head (plus a normalizer vector z, dropped in most modern variants). All of history now lives inside a fixed-size, lossy, in-place-updated summary. That single sentence contains the entire systems consequence of Part II.
Step 2 — make the state forget and edit: the modern family
Pure additive state (above) just accumulates and eventually smears. Modern variants make the update selective, and they're not academic curiosities — each row below is running in production models that vLLM serves:
| Mechanism | State update rule | Intuition | Shipped in |
|---|---|---|---|
| Vanilla linear attention | S ← S + kvᵀ | Pure accumulation, no forgetting | (foundation for all below) |
| Lightning Attention | decayed S ← λS + kvᵀ, tiled I/O-aware kernel | Exponential forgetting, hardware-shaped | MiniMax-Text-01 / M1 |
| Gated Linear Attn (GLA) | S ← Diag(αₜ)S + kvᵀ | Learned, per-channel, data-dependent forgetting | GLA models; ancestor of GDN |
| DeltaNet | S ← S(I − βₜkₜkₜᵀ) + βₜkₜvₜᵀ | Delta rule: erase old value at key k, write new — associative-memory editing | DeltaNet line |
| Gated DeltaNet (GDN) | gating × delta rule combined | Forget globally and edit precisely | Qwen3-Next, Qwen3.5/3.6 hybrids |
| Mamba-1 (selective SSM) | hₜ = Āₜhₜ₋₁ + B̄ₜxₜ, A/B/C input-dependent | Control-theory lineage; selectivity fixed S4's copying failures | Mamba, Falcon-Mamba, Codestral Mamba |
| Mamba-2 (SSD) | scalar-decay structured SSM ≡ a linear attention | The duality result: SSMs and linear attention are one family; matmul-friendly → tensor cores | Granite 4.0, Nemotron-H/Nano-2, Falcon-H1, Bamba, Zamba2, Jamba(M1) |
| Kimi Delta Attention | refined gated delta variant | Finer-grained channel-wise gating | Kimi Linear |
Step 3 — the three computation modes (this is the systems-relevant part)
Every member of the family admits three equivalent ways to compute the same function, and a serving engine uses all three:
- Recurrent mode — decode. One tiny state update + readout per token. O(1) compute, O(1) memory traffic per step, no dependence on context length whatsoever. This is the superpower.
- Parallel mode — theory. Materialize the full T×T (masked, unnormalized) interaction — quadratic, defeats the purpose, never used at scale.
- Chunkwise-parallel mode — prefill. The workhorse: split the prompt into chunks (64–256 tokens); within a chunk, compute with dense matmuls (tensor cores go brrr); between chunks, carry the recurrent state forward. Linear in T with matmul-class hardware efficiency. Mamba-2's SSD paper made this the standard recipe, and every serious kernel — the FLA (flash-linear-attention) Triton kernels vLLM uses for GLA/GDN, Mamba's fused scan kernels, Lightning Attention's tiled kernels — is a variation on it.
Chunkwise prefill naturally passes through the exact recurrent state at every chunk boundary. If you wanted to snapshot states at regular token intervals — say, block boundaries — the algorithm is already handing them to you. This becomes the key that unlocks prefix caching in 0x09.
Step 4 — the memory arithmetic that motivates everything
Concrete numbers, from the vLLM/IBM team's write-up on NVIDIA-Nemotron-Nano-12B-v2: one 16-token attention KV block ≈ 64 KiB; one sequence's whole Mamba state ≈ 2.57 MiB, total, forever (a chunky object: the SSM state matrices plus a small causal-conv1d rolling window, typically kept fp32 for numerical stability). So for short prompts the recurrent model actually spends more memory per sequence. But KV grows and state doesn't: at 128K context the KV cache is roughly 200× larger than the Mamba state. Decode bandwidth tells the same story — softmax streams an ever-growing cache per token; linear streams a constant few MiB. That, plus linear-time prefill (no quadratic TTFT blowup), is the entire pitch for RAG, agents, and long reasoning traces.
Why hybrids, though?
Pure recurrence compresses all history into a fixed budget, and lossy compression has a price: exact long-range recall (find the needle, quote it verbatim) degrades, and in-context retrieval is provably bounded by state capacity. The field's pragmatic answer is the hybrid: mostly linear/Mamba layers for scale, a sprinkling of full-attention layers (often ~1 in 4 to 1 in 8, sometimes sliding-window) as random-access memory. Qwen3-Next & Qwen3.5/3.6 (GDN + attention), Nemotron-Nano-2, Granite 4.0, Falcon-H1, Jamba, Zamba2, MiniMax-Text-01 all follow this template — which means a serving engine can't pick one memory model. It must run both, in the same forward pass, from the same memory pool. That is the actual engineering problem vLLM had to solve.
The collision: four load-bearing assumptions, tested
Here is the honest answer to "can vLLM work with linear attention?" — decomposed into the four properties Part I kept flagging. Two survive. Two shatter.
Memory grows one token-slot at a time
Paging exists to tame unpredictable per-token growth: allocate a 16-token block on demand, never reserve the future.
Memory never grows at all
The state is allocated once, at admission, at full size (~MiBs), and stays that size until the request dies. There is nothing to page in the growth sense — but a big fixed object still needs a slot in a shared pool.
Cached state is append-only & immutable
Written once, read forever. This is what makes refcounted sharing, copy-on-write, hash-chained prefix caching, and truncate-to-rollback all sound.
The state is destructively overwritten every token
S ← f(S, token). The state at time t replaces the state at t−1. Yesterday's state is gone unless you explicitly snapshotted it.
Every token's cache entry is independently addressable
Want tokens 0–4095's KV? Take those 256 blocks. Drop tokens outside a sliding window? Free those blocks. Reuse half a prefix? Take half the blocks.
Per-token state does not exist
The state is a holistic, lossy fold of the whole prefix. You cannot extract "tokens 0–4095's contribution", cannot evict token spans, cannot combine state(A) with tokens(B). A snapshot is valid for one exact prefix, in full, or not at all.
Rollback = truncation
Speculative decoding verifies k draft tokens, keeps j, discards the rest by pointing the length counter back. Preempted requests recompute from raw token IDs. Cheap, exact, trivial.
The update is a one-way door
After absorbing k speculative tokens, S cannot be un-updated (inverting it is numerically hopeless even where algebraically defined). Going "back 3 tokens" means restoring a snapshot from before, then re-advancing.
And the parts that never cared
Just as important is what doesn't collide. Continuous batching is attention-agnostic — a token budget per request schedules identically (admission control even gets easier: memory need per request is a known constant, no growth forecasting). Chunked prefill maps one-to-one onto chunkwise-parallel kernels. The two-process engine, persistent batch, sampler, structured output, API layer: untouched. Even the decode-side economics improve — batches are capped by compute, not by a ballooning cache. vLLM's skeleton fits linear attention beautifully; it's the memory manager and every feature built on cache immutability that had to be reinvented. Which brings us to how they actually did it.
How vLLM made it work — the actual engineering
The story in three acts: a fragile V0 hack, a unified V1 allocator with page-size alignment, and the slow reconquest of every advanced feature — prefix caching, CUDA graphs, speculation — under the new rules.
Act I · The V0 hack: two memory systems in a trench coat
First support (Jamba-era, 2024) bolted a separate MambaCacheManager next to the paged pool: each Mamba layer got a plain tensor holding one state slot per possible concurrent sequence, sized by the user-set max_num_seqs. It worked, and it was miserable. The two allocators couldn't see each other, so the user had to guess the memory split: set max_num_seqs too high → OOM crash; too low → idle GPU. Being outside the paged world, the state was invisible to prefix caching, KV transfer, and disaggregation — permanently. The vLLM/IBM team's own retrospective calls it "a pragmatic but fragile hack." V0's whole hybrid path has since been deleted.
Act II · The V1 unified allocator: one pool, many page dialects
V1 rebuilt memory management around a generalization: layers declare a KVCacheSpec describing what they cache, and layers with identical specs form a KVCacheGroup. A hybrid model has (at least) two groups — full-attention layers and Mamba/linear layers — and a coordinator (HybridKVCacheCoordinator) runs a per-group manager over one shared pool of physical pages, backed by shared KVCacheTensors. (The same machinery is what serves sliding-window hybrids like Gemma-3 and gpt-oss, where a window layer only keeps blocks for the last W tokens — linear attention is the extreme point of the same generalization.) For this to be simple and fragmentation-free, one invariant is enforced: every group's page must be the same number of bytes. And there's the rub — an attention page was 16 tokens ≈ 64 KiB; a Mamba "page" is one whole state ≈ 2.57 MiB.
The fix is delightfully blunt: make the attention block bigger until the byte sizes meet. vLLM auto-raises the attention block_size so one attention page ≥ one Mamba state, then pads the Mamba page slightly so the two are byte-identical. Real values: 672 tokens/block for Nemotron-Nano-12B-v2, 528 for Qwen3.5 (you'll see it in the logs: "Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size"). Now any free page serves either group; allocation stays trivially uniform. Two wrinkles, both solved:
- Giant blocks vs kernels. Some attention kernels (e.g. FlashInfer's TRT-LLM path on Blackwell) only accept small block sizes. vLLM decoupled the manager's block size from the kernel's view: memory is bookkept in 672-token pages, while the kernel sees them re-sliced into its preferred granularity. Measured cost of the odd block sizes themselves: negligible — in these models attention is the minority of runtime anyway.
- The striding bug. Groups share the same underlying tensors through different views. Attention backends store K/V interleaved block-by-block; Mamba's default layout stored all conv-states then all SSM-states — so a Mamba write could land inside a different request's attention block. Silent cross-request corruption. Fix: restride the Mamba state tensors (and later, the FlashAttention view too) so both views tile the shared memory identically. A great reminder that "unified memory" is a layout contract, not just an allocator.
Act III · Winning the features back, one by one
Prefix caching → state checkpoints at block boundaries
Recall the collision: no per-token addressability, so reuse must mean exact-prefix snapshots. And recall the gift from 0x07: chunked prefill already surfaces the recurrent state at chunk boundaries. vLLM combines them — during prefill, snapshot the Mamba/linear state at attention-block-aligned boundaries (every 528/672 tokens) into pool pages that are hashed and retained exactly like KV blocks. A new request that exactly matches a cached prefix up to some boundary loads the snapshot and skips that much prefill; attention layers reuse their KV blocks as usual over the same span. Shipped first for Mamba-2 hybrids (experimental, late 2025), then extended across the family — Mamba-1, GDN, short-conv and linear-attention layers — with two flavors: an align mode (snapshot only at aligned boundaries) and an all mode (denser snapshots for more hit chances), plus Marconi-style admission policies deciding which prefixes are worth the MiB-sized snapshot rent, and fused GPU-side post-processing kernels to keep snapshotting off the CPU critical path.
Granularity hurts short prompts: caching happens per full page, and pages are now 528–672 tokens. A 500-token shared system prompt → 0% hit rate; 979 tokens → ~1 page cached, ~450 recomputed. Pure transformers with 16-token blocks would cache nearly all of it. Also, each snapshot spends MiBs where softmax spends KiBs, so cache capacity (in reusable prefixes) is thinner. Fundamental? No. The natural consequence of R3? Exactly.
CUDA graphs → FULL_AND_PIECEWISE
The linear/Mamba kernels are mostly Triton — superb for iterating on exotic recurrences, portable across vendors, but saddled with heavy CPU launch overhead, which is deadly precisely where these models shine (small-batch, low-latency decode: Granite-4.0-h-tiny activates ~1B params — the GPU finishes before Python does). vLLM staged its way out: eager → piecewise graphs → full graphs for decode-only batches → the now-default FULL_AND_PIECEWISE: fully-captured graphs for uniform decode steps, piecewise for mixed prefill+decode steps. That final stage is what let V1 beat V0 and delete it: on H100s, +2–18% throughput for Nemotron-Nano-12B-v2 and up to +91% for granite-4.0-h-tiny, with better TTFT and ITL across the sweep.
Speculative decoding → checkpoint, verify, re-advance
Qwen3-Next ships MTP heads, so speculation had to work over GDN layers. The recipe follows A4's verdict: hold the pre-speculation state, verify the k drafted tokens in one chunked pass, and commit by advancing the state only through the j accepted tokens (the kernels expose the intermediate states the verification pass produces, so "re-advance" is a select, not a recompute). Rejected tokens simply never touch the canonical state. Same contract as truncation — achieved with snapshots instead of pointers.
Scheduling & preemption → mostly free, one asymmetry
As promised, the scheduler barely noticed: a linear layer's "block need" is constant, chunked prefill drives the chunkwise kernels directly, admission control is simpler than ever. Preemption inverted its economics: swapping a preempted request is now cheap (2.57 MiB beats gigabytes of KV over PCIe), while recompute-style preemption is pricier per unit of progress lost — without a checkpoint, a 100K-token prefix must be fully re-folded, since no partial state survives. Snapshots from prefix caching double as recovery points here, softening exactly this cost. KV-transfer / disaggregation followed the same trick as everything else: once the state is an opaque, fixed-size page, connectors like LMCache ship it between nodes with zero model-specific code.
Scoreboard: what works, what's partial, what's just different
The state of play in current vLLM (V1 engine, 2026), feature by feature — plus the model landscape this machinery now serves.
| Capability | Status | How / why |
|---|---|---|
| Continuous batching | full | Token-budget scheduling is attention-agnostic; admission is even simpler (constant per-request footprint). |
| Chunked prefill | full | Maps directly onto chunkwise-parallel kernels (FLA/Triton, Mamba fused scans, Lightning tiles). |
| Unified paged memory | full | State = one padded page; page-size alignment (528/672-token attention blocks); shared pool, shared tensors, fixed strides. |
| CUDA graphs / low-latency decode | full | FULL_AND_PIECEWISE default; tames Triton launch overhead; up to +91% throughput vs V0 on small-activation hybrids. |
| Tensor / expert parallelism | full | State shards along the head/group dimension like everything else; MoE hybrids (Qwen3-Next, Granite 4.0) run EP as usual. |
| Automatic prefix caching | partial | Exact-prefix state checkpoints at page boundaries (align/all modes). Coarse granularity: prompts shorter than one ~528-token page get zero reuse; snapshots cost MiBs each. |
| Speculative decoding / MTP | partial | Checkpoint-verify-re-advance for GDN (Qwen3-Next MTP). Correct; costs state bookkeeping per round instead of free truncation. |
| Preemption | partial | Swap: cheaper than ever. Recompute: all-or-nothing without a checkpoint — no partial-prefix credit exists to salvage. |
| Copy-on-write forking (n>1, beams) | gone by nature | In-place mutation forbids lazy sharing; forks eagerly copy the full state. Cheap-ish in absolute MiBs, but the elegant refcount trick doesn't translate. |
| Per-token cache surgery | gone by nature | No trimming to a window, no partial-prefix stitching, no mid-sequence dedup — the state has no per-token anatomy. Mathematical, not fixable. |
The models this serves today
SSM-only lines
Mamba & Mamba-2 checkpoints, Falcon-Mamba, Codestral Mamba (7B code model), Bamba, PLaMo-2, Zamba-family cores — the maximal "no KV cache anywhere" bet; strongest at raw long-context throughput, weakest at exact recall.
Attention + SSD
Jamba (the original hybrid at scale), IBM Granite 4.0 (MoE hybrids to enterprise), NVIDIA Nemotron-H / Nemotron-Nano-2, Falcon-H1, Zamba2 — a minority of attention layers as random-access memory over an SSD backbone.
Attention + GDN/Lightning/KDA
Qwen3-Next 80B-A3B and the Qwen3.5/3.6 series (Gated DeltaNet), MiniMax-Text-01/M1 (Lightning Attention at 456B), Kimi Linear (Kimi Delta Attention) — the current frontier of the recipe, all V1-only citizens.
Sliding-window hybrids
Gemma-3, Llama-4, gpt-oss mix full + windowed attention — served by the same hybrid coordinator. Proof that the V1 allocator is a general theory of heterogeneous caches, of which linear attention is one (extreme) dialect.
Takeaways
1 · vLLM is an OS, and that's why it survived
Its deepest ideas — pages, page tables, iteration-level scheduling — are resource-management ideas, not transformer ideas. The parts that assumed nothing about attention (scheduler, executor, API) crossed over to linear attention untouched.
2 · The real dependency was immutability
"Can vLLM do linear attention?" resolves to "which features depend on an append-only, per-token, immutable cache?" Sharing, hashing, COW, truncation-rollback did — and each had to be re-derived for a mutable, holistic, fixed-size state.
3 · The reconciliation was conceptual, not brute force
One move unlocked everything: promote the recurrent state to a first-class page. Align page bytes, fix strides, checkpoint at boundaries — and prefix caching, disaggregation, KV transfer all come back as corollaries.
4 · The trade-offs that remain are mathematics
Coarse cache granularity, eager fork copies, exact-match-only reuse aren't bugs in vLLM — they're the systems-level shadow of lossy compression. Hybrids exist precisely to buy those properties back where they matter.
Yes, vLLM works with linear attention — because its scheduler never cared about attention math, and its memory manager was generalized (V1) until a fixed-size mutable state could masquerade as one big immutable-ish page; what cannot be recovered — per-token addressability and lazy sharing — is lost to the math of lossy recurrent compression, not to any limitation of vLLM, and hybrid architectures are the industry's standing compromise with exactly that fact.