# vLLM Deep Dive: How PagedAttention, Continuous Batching and the V1 Engine Serve LLMs in Production

A production LLM platform has many layers: a gateway for tenants and budgets, model-aware routing, a lifecycle controller such as KServe, an inference-aware endpoint picker such as llm-d, and a GPU scheduler underneath. All of them exist to feed one component, the engine that actually generates tokens. In most open-source stacks that engine is vLLM.

vLLM is where most of your GPU money goes, and nearly every signal the upper layers use (queue depth, KV-cache usage, prefix-cache hits, time to first token) is produced inside it. You cannot tune the router, the autoscaler or the GPU scheduler well without knowing what those numbers mean.

This post opens the box. By the end you should be able to answer:

- Why LLM serving is a **memory problem** before it is a compute problem, and how to calculate the KV-cache size for a model by hand.
- How **PagedAttention** and the **block manager** turn GPU memory into a pool that hundreds of requests share.
- What the **V1 scheduler** actually does on every step: token budgets, chunked prefill, preemption and admission control.
- How the engine is split across **processes** (API server, EngineCore, GPU workers) and why that matters for CPU sizing and latency.
- How **prefix caching, speculative decoding, parallelism and disaggregated prefill/decode** work, and when each one pays off.
- Which **flags and metrics** matter in production, and how to size a deployment for a target number of users.

> **Scope and accuracy note:** This is an architecture deep dive written from the vLLM source, design docs, papers, blog posts and release notes as available on 9 November 2026. It was first written against v0.32.0 and is updated for **vLLM v0.33.0** (2 November 2026); the section "What changed in v0.33.0" near the end lists what moved. vLLM ships a release roughly every two weeks and internal names move fast, so treat code-level names as a map, not a contract. The sizing numbers below are worked arithmetic, not benchmarks I have measured. Pin a version and validate against it.

## The physics of LLM inference

An LLM serving engine is a memory manager that happens to run matrix multiplications. Everything vLLM does follows from two facts: generation has two phases with opposite bottlenecks, and the state each request carries (the KV cache) grows with every token.

### Two phases, two bottlenecks

![Prefill fills the KV cache once; every decode step rereads it](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/c02b8816-c084-43e6-8045-50bae264d133.svg align="center")

A request goes through **prefill** once and **decode** many times.

- **Prefill** processes all prompt tokens in a single forward pass. A 2,000-token prompt becomes large matrix multiplications that keep the tensor cores busy, so prefill is usually **compute-bound**. Its cost shows up as **time to first token (TTFT)**.
- **Decode** produces one token per request per step. To do that, the GPU must stream every weight of the model from HBM, plus that request's whole KV cache, to compute a handful of numbers. Decode is **memory-bandwidth-bound**. Its cost shows up as **inter-token latency (ITL)**, also called time per output token (TPOT).

A back-of-the-envelope number makes this concrete. Llama-3.1-8B in BF16 is about 16 GB of weights. An H100 SXM has roughly 3.35 TB/s of HBM bandwidth. Reading the weights once takes about 16 / 3,350 s ≈ **4.8 ms**, so one decode step can never be faster than that. At batch size 1 that caps a single stream near **200 tokens/s**, with the tensor cores mostly idle.

The escape is **batching**. If 64 requests decode in the same step, the weights are still read once, but 64 tokens come out. Below a saturation point, adding requests to the batch costs almost nothing per step. That is why every serving engine is built to keep as many requests in flight as memory allows, and why memory, not FLOPs, usually decides how many users one GPU can serve.

> The trade-off never goes away: bigger batches raise throughput but lengthen each step, so ITL rises. Every scheduling knob discussed later moves you along that curve.

### The KV cache: the state that limits concurrency

Attention needs the key (K) and value (V) vectors of every previous token. Recomputing them each step would make generation quadratic, so engines cache them. That cache lives in GPU memory next to the weights and grows by one entry per token per layer.

The size per token is fixed by the model architecture:

$$\text{KV bytes per token} = 2 \times n_{\text{layers}} \times n_{\text{kv\heads}} \times d{\text{head}} \times \text{bytes per element}$$

The 2 is for K and V. Grouped-query attention (GQA) shrinks `n_kv_heads` well below the number of query heads, which is the main reason modern models are cheaper to serve than Llama-2-era models.

| Model | Layers | KV heads × head dim | KV per token (BF16) | One 8,192-token sequence |
| --- | --- | --- | --- | --- |
| Llama-2-7B (no GQA) | 32 | 32 × 128 | 512 KiB | 4 GiB |
| Llama-3.1-8B | 32 | 8 × 128 | 128 KiB | 1 GiB |
| Qwen3-8B | 36 | 8 × 128 | 144 KiB | 1.125 GiB |
| Llama-3.1-70B | 80 | 8 × 128 | 320 KiB | 2.5 GiB |

Read the last column carefully. A single 8k-token conversation on Llama-3.1-70B needs 2.5 GiB of KV cache, before tensor parallelism splits it across GPUs. Long context is not free: it is paid for in concurrent users.

### Worked example: how many tokens fit on one GPU

Take Llama-3.1-8B in BF16 on one 80 GB H100 with vLLM's default `--gpu-memory-utilization 0.9`:

1. vLLM may use 0.9 × 80 GB = **72 GB**.
2. Weights take about **16 GB**.
3. Activations, CUDA graph pools and workspace take a few GB more; vLLM measures this with a profiling run at startup. Assume about **2 GB** here.
4. That leaves roughly **54 GB** for KV cache.
5. 54 GB ÷ 128 KiB per token ≈ **410,000 tokens**, or about **25,700 blocks** of 16 tokens.

That pool is the real capacity of the replica. It could hold about 50 conversations at a full 8k context, or about 200 at an average of 2k tokens. Those numbers are approximate; vLLM prints the exact figure at startup as the GPU KV cache size and the maximum concurrency for your `--max-model-len`. Check that log line first whenever you change models, GPUs or quantization.

The next question is how to share that pool between hundreds of requests whose lengths nobody knows in advance. That is what PagedAttention solves.

## PagedAttention: virtual memory for the KV cache

PagedAttention stores each request's KV cache in small fixed-size blocks that can live anywhere in GPU memory, so memory is allocated as tokens arrive instead of reserved up front. This one idea is what lets vLLM pack hundreds of requests onto a GPU.

### The problem it solved

Before vLLM, serving systems typically allocated one contiguous KV buffer per request, sized for the maximum possible length. That wastes memory three ways:

- **Reservation:** a request that may generate 4,096 tokens holds space for all of them from the first step, even if it stops after 100.
- **Internal fragmentation:** the unused tail of each buffer cannot be used by anyone else.
- **External fragmentation:** buffers of different sizes leave gaps between them that no new request fits into.

The original vLLM authors measured that [existing systems wasted 60% to 80% of KV memory to fragmentation and over-reservation](https://vllm.ai/blog/vllm). Wasted KV memory means a smaller batch, and a smaller batch means the bandwidth math from the previous section works against you.

### Blocks, block tables and slot mapping

PagedAttention borrows the operating-system idea of paging.

![Block tables let requests share and scatter KV blocks freely](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/6f73c359-1874-4b3a-b249-517975fd2901.svg align="center")

- The KV cache pool is carved into **physical blocks**. Each block holds K and V for `block_size` tokens (16 by default) for one layer group. All blocks are pre-allocated as one large tensor at startup, after the profiling run.
- Each request sees a sequence of **logical blocks**: tokens 0–15 are logical block 0, tokens 16–31 are logical block 1, and so on.
- A per-request **block table** maps logical block numbers to physical block IDs. The physical blocks do not need to be contiguous or in order.
- Before each forward pass the model runner builds a **slot mapping**: for every new token, the exact physical slot (block ID × block size + offset) its K and V will be written to. The attention kernel reads the block table to gather the past K and V.

Blocks are allocated **on demand**. A new block is taken from the free pool only when the request's last block fills up. The only waste is the unfilled part of each request's final block, at most 15 tokens. The vLLM team reported that [this brings waste under 4% in practice](https://vllm.ai/blog/vllm).

The free pool itself is a doubly linked list of free block IDs. [The Anatomy of vLLM post](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm) describes it as often holding hundreds of thousands of blocks, and gives the per-block size for a standard (non-MLA) layer as 2 × `block_size` × `num_kv_heads` × `head_size` × bytes per element.

### Sharing and copy-on-write

Because a block table is just a list of IDs, two tables can point at the same physical block. Each block carries a **reference count**. In the figure, requests A and B share block #7, so its count is 2. This enables:

- **Prefix sharing:** requests with the same system prompt reuse the same physical blocks (automatic prefix caching, covered below).
- **Parallel sampling (**`n > 1`**):** several completions of one prompt share the prompt's blocks and diverge only where their tokens differ.
- A block is returned to the free pool only when its reference count drops to zero.

### What this means for operators

- `--block-size` rarely needs changing. Larger blocks reduce bookkeeping but waste more in each request's last block, and prefix-cache hits only count in whole blocks. Some attention backends and hybrid models dictate their own block sizes.
- **Hybrid models** (sliding-window layers, Mamba or linear-attention layers mixed with full attention) use vLLM's hybrid KV-cache manager, which groups layers with different memory needs. Capacity math for these models is not the simple per-token formula above; trust the startup log.
- The metric `vllm:kv_cache_usage_perc` is the fraction of this block pool in use. Near 1.0 means new prefills cannot get blocks and running requests may be preempted.

## Continuous batching and the V1 scheduler

The scheduler rebuilds the batch on every forward pass: running requests decode first, waiting requests fill whatever token budget and KV blocks remain, and finished requests leave immediately. That per-step decision is where most latency trade-offs are made.

### From static batches to iteration-level scheduling

A naive server collects N requests, runs them together until the longest one finishes, then starts the next batch. Short requests sit idle in a batch that is still running, and new arrivals wait for the whole batch.

**Continuous batching** (iteration-level scheduling, introduced by the Orca paper) schedules at the granularity of one model step instead. After every step, finished requests exit and new ones can join. The GPU stays full as long as there is work and memory.

vLLM's V1 engine takes this one step further. Internally there is no separate "prefill phase" or "decode phase". Each request has a count of tokens already computed and a count it still needs. The scheduler simply decides how many tokens each request advances this step. A decode is a request that needs one more token; a prefill is a request that needs many. Both go into the same flattened batch, and custom attention kernels keep each sequence attending only to its own tokens, so no padding is required.

### What one scheduling step does

![Each scheduler step: decodes first, then prefills until the token budget or KV blocks run out](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/9a674273-ccc2-4191-95e5-e7b99540a64a.svg align="center")

Two budgets bound every step:

- `max_num_batched_tokens` is the total number of tokens one forward pass may process across all requests.
- `max_num_seqs` caps how many requests can be in the running set at once.

Within those limits the V1 scheduler, as the [vLLM optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/) describes, batches all pending decodes before scheduling any prefill, and chunks a prefill that does not fit the remaining budget. Chunked prefill is enabled by default in V1 whenever possible.

A worked step with `max_num_batched_tokens = 8192`:

| Request | State | Tokens needed | Scheduled this step | Note |
| --- | --- | --- | --- | --- |
| 60 chat sessions | Running, decoding | 60 | 60 | Always first |
| C | Waiting, 3,000-token prompt, 1,024 cached | 1,976 | 1,976 | Prefix-cache hit skips 64 blocks |
| D | Waiting, 7,000-token prompt | 7,000 | 6,156 | Chunked: the remaining 844 run next step |
| E | Waiting | 1,200 | 0 | Budget exhausted, waits |

The 60 decoding users each get their next token in this step even though D's prompt is huge. Without chunking, D's 7,000-token prefill would occupy a whole step on its own and every streaming user would see a stall.

### Tuning the token budget

The official guidance captures the trade-off well:

- A **smaller** `max_num_batched_tokens` (for example 2048) gives better ITL, because fewer prefill tokens share each step with decodes.
- A **larger** value gives better TTFT, because more prompt tokens are processed per step.
- For throughput, the guide recommends values above 8192, especially for smaller models on large GPUs.

In practice: chat and coding assistants with streaming users want moderate values; batch and RAG pipelines with long prompts and no human watching tokens can go large.

### Preemption: what happens when blocks run out

Blocks are allocated on demand, so the scheduler is optimistic. If a running request needs a new block and none is free, the scheduler preempts another running request: it frees that request's blocks and moves it back to the waiting queue. In V1 the default mode is **recompute** rather than swapping to CPU, because recomputation has lower overhead in this architecture. The preempted request later re-runs its prefill (cheaper if its prefix is still in the cache).

Occasional preemption is the system working as designed. Sustained preemption means you are oversubscribed: tokens get computed twice and tail latency suffers. The guide's remedies are to raise `gpu_memory_utilization`, lower `max_num_seqs` or `max_num_batched_tokens`, or add tensor or pipeline parallelism to free memory per GPU. Watch `vllm:num_preemptions` (a counter) in Prometheus.

### Newer scheduling controls

Recent releases added levers worth knowing about. Check that your pinned version has them:

- `--scheduling-policy priority` lets clients pass a `priority` field; lower values run first and are preempted last. The default is FCFS.
- **Admission control**: [release notes](https://github.com/vllm-project/vllm/releases) mention `--max-num-queued-reqs` and `--max-num-queued-tokens` to cap the waiting queue, so an overloaded replica rejects work quickly instead of queuing it forever. v0.31.0 added `--max-num-active-seqs`, which caps running admission independently of `max_num_seqs`.
- **Waiting-queue rework** in v0.31.0 schedules requests that already hold KV blocks first, and `--long-prefill-token-threshold` now adapts to the number of waiting prefills.
- **Async scheduling** overlaps scheduling of step N+1 with GPU execution of step N, hiding CPU overhead between steps.

## Inside the V1 engine: processes, threads and the busy loop

A running `vllm serve` is several cooperating processes: an API server that speaks HTTP and handles text, an EngineCore that schedules, and one worker per GPU that runs the model. The split exists so that Python work on text never stalls the GPU.

![A vLLM server is a pipeline of processes, not one Python loop](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/cfae73de-6f07-4ba1-9f3c-78f369c2c501.svg align="center")

### The three kinds of process

The [architecture overview](https://docs.vllm.ai/en/latest/design/arch_overview/) describes the V1 layout:

- **API server process.** Handles HTTP, applies the chat template, tokenizes, loads multimodal inputs, and streams results back. It talks to engine cores over ZMQ sockets. There is one by default; with data parallelism the count scales to match, and `--api-server-count` sets it explicitly. Each API server connects to every engine core.
- **EngineCore process.** Runs the scheduler and KV cache manager and drives the GPU workers in a busy loop. There is one per data-parallel rank. Inside, an input thread decodes requests from the socket, the main thread runs schedule → execute → update, and an output thread ships results back. Only the main thread touches scheduling state.
- **GPU worker processes.** One per GPU (TP × PP per engine core). Each worker owns a model runner, loads its weight shard, holds its slice of the KV cache and executes forward passes. The engine core broadcasts each step's scheduler output to all ranks through a shared-memory message queue; tensor-parallel ranks synchronise with NCCL all-reduces inside the model.
- **DP coordinator** (only when `--data-parallel-size` > 1). Collects load from each engine and helps the API servers balance across them.

| Deployment | API servers | Engine cores | GPU workers | Coordinator | Total processes |
| --- | --- | --- | --- | --- | --- |
| `vllm serve -tp 4` | 1 | 1 | 4 | 0 | 6 |
| `vllm serve -tp 2 -dp 4` | 4 | 4 | 8 | 1 | 17 |

### Why this matters for your pod spec

The busy loop is the heartbeat of the server, and it is CPU work. The [optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/#cpu-resources-for-gpu-deployments) is direct about this: plan for at least **2 + N physical cores** for N GPUs (more with DP or extra API servers), remember that one vCPU is usually half a physical core with hyperthreading, and treat low GPU utilisation as a possible sign of CPU starvation.

In Kubernetes terms: a 4-GPU pod with `cpu: "4"` is under-provisioned before it serves a single request. Give it at least 12 vCPUs, more if you serve images or very short requests, and avoid CPU limits that throttle the engine core. If tokenization is your bottleneck (very short prompts at high QPS, or heavy multimodal preprocessing), add API servers with `--api-server-count`; since v0.32.0, vLLM sends each new connection to the least-loaded API worker.

### What one engine step looks like

The EngineCore main loop repeats the same three stages, as the [Anatomy of vLLM post](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm) walks through in detail:

1. **Schedule.** Pick requests and token counts for this step (previous section).
2. **Execute.** The model runner copies the step's inputs to the GPU, builds the slot mapping and attention metadata, runs the forward pass (usually by replaying a captured CUDA graph), gathers the last hidden state of each sequence, and samples.
3. **Update.** Append sampled tokens to each request, check stop conditions, free the blocks of finished requests, and push outputs to the output thread.

With **async scheduling**, the scheduler prepares step N+1 while the GPU is still running step N, so the CPU gap between kernels shrinks.

### Model Runner V2

vLLM is midway through replacing its model runner, the per-worker component that turns scheduler decisions into GPU inputs. The [Model Runner V2 design doc](https://docs.vllm.ai/en/latest/design/model_runner_v2/) is candid that the original V1 runner accumulated design mistakes and technical debt. MRV2 is async-first: it assumes the execution loop is a CUDA stream with no CPU synchronisation points, prepares input metadata with Triton kernels on the GPU, uses a Triton-native sampler, and manages CUDA graphs explicitly.

The release notes show MRV2 absorbing features release by release (speculative decoding, pipeline parallelism, Elastic EP), and v0.32.0 made it the default on ROCm. You do not configure it day to day, but it explains why release notes so often mention "MRV2", and it is the reason some features land on one runner before the other.

## Automatic prefix caching

Prefix caching lets a new request reuse KV blocks that an earlier request already computed for the same leading tokens, so only the new part of the prompt is prefilled. It is on by default, it does not change outputs, and for chat, agents and RAG it is often the single biggest TTFT win available.

### How a block is identified

vLLM uses a hash-based design. As the [prefix caching design doc](https://docs.vllm.ai/en/latest/design/prefix_caching/) explains, each full block is keyed by a hash of three things: the **parent block's hash**, the **token IDs in this block**, and **extra keys** such as the LoRA adapter, multimodal input hashes and an optional cache salt.

![A prefix hit needs every earlier block to match, not just this one](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/d0ec6a5c-425e-4817-98fa-3d0b4811ddd2.svg align="center")

The chaining is what makes this a *prefix* cache. Block 3's hash depends on blocks 1 and 2, so a block can only be reused if everything before it matched too. Two consequences follow:

- **Only full blocks are cached.** With a block size of 16, a 1,000-token shared system prompt yields 62 cacheable blocks; the last 8 tokens are recomputed.
- **Order matters.** If anything near the start of the prompt changes per request (a timestamp, a user name, a request ID), every block after it misses.

Since v0.11 the default hash is SHA-256, chosen to remove collision risk. `--prefix-caching-hash-algo` also offers reproducible CBOR-based variants and faster xxHash variants; the docs warn that non-cryptographic hashes slightly raise collision risk, which matters in multi-tenant deployments.

### Lookup, sharing and eviction

When a request is scheduled, the KV cache manager hashes its prompt blocks and walks the hash table to find the longest chain of hits. Hit blocks are "touched": their reference count goes up and they are pulled out of the free queue so nobody evicts them while in use.

Finished requests do not erase their blocks. Blocks whose reference count reaches zero go back to the **free queue but keep their hash**, so they remain reusable until the space is needed. The free queue is ordered least recently used first, and a request's blocks are returned in reverse order so its tail blocks, which carry the most request-specific tokens, are evicted first. A cached block is only invalidated at the moment it is popped for reuse.

The effect is that the whole idle part of the KV pool works as a cache. A replica at 40% KV usage is not wasting 60% of its memory; it is holding recently used prefixes.

### Multi-tenant isolation

Prefix caching has a side channel: a request that hits the cache returns its first token faster, which can reveal that someone else recently sent the same prefix. vLLM supports a per-request `cache_salt` field that is mixed into the first block's hash, so only requests with the same salt can share blocks. On a shared platform, set the salt per tenant at the gateway (an API gateway such as LiteLLM or Envoy AI Gateway is the natural place).

### Making it work in production

- **Design prompts for reuse.** Put static content first: system prompt, tool definitions, few-shot examples, then retrieved documents, then the user turn. Move volatile values to the end.
- **Route for locality.** The cache lives inside one replica. With round-robin routing across 8 replicas, a conversation's next turn has a 1-in-8 chance of landing where its history is cached. This is exactly why llm-d's endpoint picker scores prefix-cache locality.
- **Extend it beyond GPU memory.** KV offloading connectors (vLLM's native CPU offloading, LMCache, Mooncake and others listed in the [docs](https://docs.vllm.ai/en/latest/features/kv_offloading_usage/)) spill evicted blocks to CPU RAM or storage and bring them back on a hit, which is much cheaper than recomputing a long prefix.
- **Measure it.** `vllm:prefix_cache_hits` and `vllm:prefix_cache_queries` are token counters; their ratio is your hit rate:

```promql
# Prefix-cache hit rate per model over 5 minutes
sum by (model_name) (rate(vllm:prefix_cache_hits_total[5m]))
/
sum by (model_name) (rate(vllm:prefix_cache_queries_total[5m]))
```

Prometheus clients append `_total` to counters, so check the exact exported names in your `/metrics` output before building dashboards.

## Speculative decoding: more tokens per weight read

Speculative decoding cuts inter-token latency by letting a cheap drafter guess several tokens and having the real model check them all in one forward pass. It works because decode is bandwidth-bound: verifying 4 tokens costs almost the same as generating 1.

![One target pass can verify several drafted tokens at once](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/725363a0-b732-45a0-8ac7-1db6e679c2ac.svg align="center")

### How it works

1. **Draft.** A cheap proposer suggests `k` next tokens.
2. **Verify.** The target model runs once over the context plus all `k` drafts. Because it computes a distribution at every position, one pass yields `k + 1` candidate positions.
3. **Accept or reject.** Left to right, each draft is accepted with a probability based on the target and draft distributions. At the first rejection, the target samples a replacement from a corrected distribution and the step ends. If every draft passes, the extra position yields a bonus token.

The rejection sampler is what makes this safe. The [vLLM speculative decoding docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/) describe it as theoretically lossless up to floating-point precision, validated by tests that check greedy outputs are identical with and without speculation. The lossy "FLy" verifier is the explicit exception.

In the scheduler, each running request asks for up to `1 + k` token slots per step instead of 1, which is why the earlier scheduler diagram says "one token per decode (more with spec decode)". Rejected draft slots are simply overwritten next step. v0.33.0 relaxed this for the block-diffusion drafters DFlash and DSpark, which no longer reserve draft slots in the token budget; the [release notes](https://github.com/vllm-project/vllm/releases) report that this alone took DeepSeek-V4.1-Flash with DSpark on GB300 from 1,373 to 3,390 output tokens per second in the PR's benchmark.

### Choosing a method

| Method | Where drafts come from | Best for | Cost |
| --- | --- | --- | --- |
| N-gram | Matching n-grams already in the prompt or output | Code editing, summarisation, RAG answers that copy from context | No extra model; lightweight |
| Suffix decoding | A suffix tree over prompts and past outputs | Repetitive agent loops and structured outputs | No extra model; dynamic depth |
| EAGLE / EAGLE-3 | A small head trained on the target's hidden states | General chat and reasoning | Needs a trained head for your model |
| MTP | The model's own multi-token-prediction layers | Models shipped with MTP heads (DeepSeek, GLM, Qwen families and others) | Built into the checkpoint |
| Draft model | A separate small model from the same family | When a good small sibling exists | Extra weights and KV memory |

The docs' own guidance is that model-based methods (EAGLE, MTP, draft models) give the largest latency reduction, while n-gram and suffix decoding give modest speedups without adding work at peak traffic. A newer family of parallel drafters (P-EAGLE, DFlash and DSpark) proposes several tokens in a single drafter pass instead of one at a time. Medusa and the MLP speculator were removed in v0.33.0, so older configurations that use them must move to one of the methods above. Configuration is a JSON object:

```bash
vllm serve Qwen/Qwen3-8B \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_min": 2, "prompt_lookup_max": 5}'
```

### When it helps and when it hurts

Speculation spends extra compute to save memory reads. That trade is excellent at **low to medium load**, when the GPU is bandwidth-bound and idle compute is available. At **high load** the batch already saturates compute, verification of rejected drafts becomes wasted FLOPs, and throughput can drop.

So treat it as a latency feature with a load-dependent payoff:

- Interactive products with a strict ITL target and moderate concurrency usually win.
- Saturated batch pipelines often lose; benchmark before enabling.
- Watch the acceptance rate. vLLM exports speculative-decoding counters for drafted and accepted tokens, and newer releases add [per-request acceptance metrics](https://docs.vllm.ai/en/latest/features/speculative_decoding/acceptance_metrics/). A low acceptance rate means you pay for drafts you throw away.
- Newer features such as dynamic speculative decoding and adaptive verification adjust speculation depth with load or drafter confidence, which softens the high-QPS penalty. Check that they support your model and runner.

## Scaling out: tensor, pipeline, data and expert parallelism

When one GPU is not enough, vLLM can split the model's work across GPUs in four ways that combine freely. The right mix depends on whether you are short of memory, short of throughput, or serving a mixture-of-experts model.

![Four ways to spread a model over GPUs: tensor, pipeline, data and expert parallelism](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/977c9bcd-5c50-4121-9d99-015d93ab95ff.svg align="center")

| Strategy | Flag | What is split | Communication | Typical use |
| --- | --- | --- | --- | --- |
| Tensor parallel (TP) | `--tensor-parallel-size N` | Every weight matrix | All-reduce after attention and MLP in every layer | Model does not fit on one GPU; stay within one NVLink domain |
| Pipeline parallel (PP) | `--pipeline-parallel-size N` | Groups of layers | Activations passed between stages | Model does not fit on one node; slower interconnect between nodes |
| Data parallel (DP) | `--data-parallel-size N` | Nothing: full copies | None for dense models; synchronised steps for MoE | More throughput inside one server process |
| Expert parallel (EP) | `--enable-expert-parallel` | MoE experts | All-to-all dispatch and combine per MoE layer | Large MoE models (DeepSeek, Qwen MoE, Kimi, GLM) |

The [optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/) summarises the intent: TP is the most common strategy inside a node, PP extends a model across nodes once TP is exhausted, DP replicates the model for throughput, and EP replaces TP for MoE layers using the same degree of parallelism.

### Tensor parallelism and the KV cache

TP is not only a way to fit big models. It also splits the KV cache: each rank stores the KV heads assigned to it. Two GPUs at TP=2 therefore have far more KV capacity per replica than one GPU, because the weights are paid for once across both. The cost is an all-reduce in every layer on every step, which adds latency and depends heavily on interconnect. Keep TP inside NVLink, and do not set TP higher than the number of KV heads.

### A worked choice: Llama-3.1-70B on 8 × H100

In FP8, the 70B weights are about 70 GB. Consider three layouts for one 8-GPU node:

| Layout | Replicas | Weights per GPU | Approximate KV room per replica | Character |
| --- | --- | --- | --- | --- |
| TP=8 | 1 | ~9 GB | ~490 GB | Lowest ITL, huge single KV pool, all eggs in one basket |
| TP=4 × 2 replicas | 2 | ~18 GB | ~210 GB each | Balanced; a pod failure halves capacity instead of zeroing it |
| TP=2 × 4 replicas | 4 | ~35 GB | ~70 GB each | Most replicas, least all-reduce traffic, smallest KV pool each |

The KV figures assume `--gpu-memory-utilization 0.9` and a few GB of overhead per GPU; they are arithmetic, not measurements. Which is best depends on context length: if users send 32k-token contexts, a 70 GB pool holds only about 215,000 tokens, or six or seven such conversations per replica in BF16 KV. Long-context workloads push towards higher TP, short chat pushes towards more replicas.

For dense models on Kubernetes I usually prefer **independent replicas behind a smart router** over vLLM's built-in DP. Each replica is its own pod, so the router (llm-d EPP) can do cache-aware placement, and failures and rollouts are per pod.

### MoE models: DP attention plus expert parallelism

Mixture-of-experts models change the picture. Attention layers are small, while expert weights are huge but each token only touches a few experts. The common large-scale layout runs **attention as data parallel** (each DP rank owns its own requests and KV cache) and **experts as expert parallel** across all ranks, with all-to-all communication routing tokens to the GPUs that own their experts. Because every MoE layer needs all ranks, DP ranks must step in lockstep, which is why idle ranks run dummy forward passes.

This "wide EP" pattern is where much of vLLM's recent engineering has gone: expert-parallel load balancing (EPLB) to even out hot experts, several all-to-all backends, dual-batch overlap to hide communication, and Elastic EP. A recent vLLM blog post describes [near-zero-downtime Elastic EP scaling](https://vllm.ai/blog/2026-10-19-elastic-ep-near-zero-downtime). If you run DeepSeek-class models, this is the area to study next.

### Going multi-node

Across nodes, vLLM can run with Ray or with its own multiprocessing launcher, where secondary nodes start in `--headless` mode and join the primary's data-parallel group. On Kubernetes the usual wrapper is a LeaderWorkerSet (LWS), which KServe and llm-d both use for multi-node replicas. The node topology matters as much as the flags: TP across nodes over Ethernet is usually a mistake, whereas PP or DP across nodes is fine.

## Disaggregated prefill and decode

Disaggregation runs prefill and decode on different vLLM instances, often on different GPUs or nodes, and moves the KV cache between them. It removes interference between the two phases and lets you size and tune each pool for its own bottleneck, at the cost of a fast network and more moving parts.

### The problem it solves

Chunked prefill already mixes the phases politely, but they still share one GPU and one step. A burst of long prompts raises every streaming user's ITL; a large decode batch delays new users' first tokens. You also cannot tune one engine for both: prefill wants a large token budget, decode wants many concurrent sequences, CUDA graphs and steady small steps.

Splitting them means TTFT is governed by the prefill pool and ITL by the decode pool, and each scales on its own signal. Frameworks such as NVIDIA Dynamo and llm-d are built around this idea; in both, vLLM is the engine doing the actual work on each side.

![Prefill and decode run on separate GPUs; only the KV cache crosses between them](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/c5c92485-806c-4f47-aac7-2d383af41253.svg align="center")

### How it works inside vLLM

The mechanism is the **KV connector** API. A connector has a scheduler-side half and a worker-side half, as the [Anatomy of vLLM post](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm) traces:

- When the scheduler considers a waiting request, it asks the connector how many tokens are already available externally (after checking the local prefix cache). On the decode side, that is the whole prompt computed by the prefill instance.
- The scheduler allocates local blocks for those tokens and tells the connector which requests need loading or saving.
- Around the forward pass, the worker-side connector starts asynchronous loads into paged KV memory and waits for saves to finish. Transfers can run layer by layer to overlap with compute.

With the [NixlConnector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage/), prefill instances run with `kv_role: kv_producer` and decode instances with `kv_role: kv_consumer`; the old `kv_both` role is deprecated for NIXL. A router sends the request to a prefill instance first, which returns `kv_transfer_params` (block handles). The router then sends the request plus those handles to a decode instance, which **pulls** the blocks directly from the prefill GPU's memory using RDMA. The prefiller holds the blocks under a lease (`kv_lease_duration`, default 30 seconds, renewed by heartbeats) so a slow decoder does not read freed memory.

Pull is the default, but it is no longer the only mode. With `NixlPushConnector` on both sides, the decode instance registers its destination blocks while prefill is still running, and the prefill instance **writes** the KV into them from a background thread as soon as both are ready. That takes the router relay and decode-side transfer setup off the path to the first token. In the [push connector write-up](https://vllm.ai/blog/2026-10-14-nixl-push-connector), a 1P1D Qwen3-32B deployment on AWS P5en instances saw up to 3x lower TTFT and up to 30% lower TPOT than pull at high request rates. The trade-off is routing flexibility: push must pick the decode instance before prefill starts, whereas pull can choose after it, and pull remains the mode for bidirectional transfer.

![Push mode overlaps decode-side setup with prefill, so the first token arrives sooner](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/96e08b35-9a11-456a-aed5-afc7729637f8.svg align="center")

```bash
# Prefill instance
vllm serve <model> \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}'

# Decode instance
vllm serve <model> \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}'
```

Two operational details from the docs deserve attention. `kv_load_failure_policy: fail` (the default) fails a request whose KV transfer fails; `recompute` makes the decode instance redo the prefill, which hurts every other stream on that decoder. And NIXL uses UCX by default, so network tuning happens through `UCX_*` variables, not `NCCL_*`. A newer bidirectional mode even lets the prefiller pull earlier turns' KV back from the decoder for multi-turn chat. From v0.33.0 the NIXL connector requires NIXL 1.5 or newer, and a new EFA image variant (the v0.33.0-efa tag) bundles the AWS EFA libraries for KV transfer on EC2.

### Other connectors

NIXL is one of several. LMCache and Mooncake provide shared KV stores (useful for both P/D and cross-instance prefix reuse), native CPU offloading extends a single instance's cache, and others target specific hardware. All plug into the same scheduler hooks, which is why llm-d and Dynamo can build on vLLM without forking it.

### When it is worth it

- **Worth trying:** large models on multi-node clusters with RDMA (InfiniBand or RoCE), long prompts, strict separate TTFT and ITL targets, and enough traffic to keep two pools busy. vLLM's own [GLM-5.2 on 24 B300 GPUs write-up](https://vllm.ai/blog/2026-07-23-glm-5.2-nvfp4-b300-pd) combined P/D disaggregation, MTP and Model Runner V2 to cut mean TPOT from 40 ms to 17 ms.
- **Probably not:** small and mid-size models that fit on one node, short prompts, low traffic, or clusters without a fast fabric. A well-tuned aggregated deployment with chunked prefill and prefix-aware routing is simpler and often close in cost.
- **Watch:** the prefill-to-decode ratio (xPyD) must follow the workload's input/output length mix; NIXL exports metrics such as `vllm:nixl_xfer_time_seconds` and `vllm:nixl_num_kv_expired_reqs` for transfer health.

## Performance internals: graphs, compilation, kernels and quantization

Below the scheduler, vLLM's speed comes from removing CPU overhead (CUDA graphs and torch.compile), from specialised attention kernels that read the paged KV layout, and from quantization that shrinks both weights and KV cache. Most of this is automatic; knowing it helps you read startup logs and make sense of cold-start time.

### CUDA graphs and torch.compile

A decode step for a small batch is thousands of tiny kernels. Launching them one by one from Python can take longer than running them. A **CUDA graph** records the whole sequence once and replays it with a single launch. vLLM captures graphs for a set of batch sizes at startup and pads each step up to the nearest captured size.

vLLM also runs the model through **torch.compile** to fuse operations (for example normalisation with quantization) and to split the graph around attention so the rest can be captured. The [optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/) exposes this as optimization levels:

| Level | What you get | Use it for |
| --- | --- | --- |
| `-O0` | No compilation or graphs | Fastest startup, lowest performance |
| `-O1` | Simple compilation, fast fusions, piecewise CUDA graphs | Faster boots with most of the benefit |
| `-O2` (default) | More fusions, full and piecewise CUDA graphs | Production |
| `-O3` | Currently equal to `-O2`; reserved for heavier future optimisations | Experiments |

`--enforce-eager` disables both compilation and graphs. It is useful for debugging and for measuring how much of your boot time is compile and capture, but it costs steady-state decode speed.

### Attention backends

PagedAttention is a memory layout; the kernels that read it are pluggable. vLLM picks a backend from your GPU generation, model type and features: FlashAttention and FlashInfer on most NVIDIA GPUs, FlashMLA and related kernels for MLA models such as DeepSeek, Triton kernels as a portable path, and vendor kernels on ROCm and other accelerators. The [attention backend feature matrix](https://docs.vllm.ai/en/latest/design/attention_backends/) lists what each supports. Leave the default unless you are chasing a specific regression; the startup log tells you which one was selected.

### Quantization: weights and KV cache

Quantization buys capacity in two places. Smaller weights leave more memory for KV cache and reduce bytes read per decode step; a smaller KV cache directly multiplies the number of tokens in the pool.

| Technique | What it shrinks | Typical hardware | Notes |
| --- | --- | --- | --- |
| FP8 W8A8 | Weights and activations to 8 bits | Ada, Hopper, Blackwell | Common default for production checkpoints |
| NVFP4 / MXFP4 | Weights to 4 bits | Blackwell (and selected paths elsewhere) | Large models on fewer GPUs |
| INT4 W4A16 (AWQ, GPTQ) | Weights to 4 bits, compute in 16 bits | Ampere and newer | Good for memory-bound small batches |
| FP8 KV cache (`--kv-cache-dtype fp8`) | KV cache to 8 bits | Hopper, Blackwell and others | Roughly doubles the token pool |
| NVFP4 KV cache | KV cache to 4 bits | Expanding in v0.32 | Newest; validate quality |

Two cautions. First, evaluate quality on your own tasks: an FP8 KV cache halves KV memory, but the v0.32.0 notes record that turning it on by default for GLM-5.x gave 2.3–5.5% more throughput at a 7–11% TTFT cost in the PR's benchmark, so even the trade-offs are workload-specific. Second, quantization flags change between releases (v0.31.0 replaced online `quantization="fp8"` with an `fp8_per_tensor` shorthand), so prefer pre-quantized checkpoints, which carry their own config.

### Cold start: the hidden SLO

From an autoscaler's point of view, the most important performance number is not tokens per second but **time from pod start to ready**. Every phase in the diagram below costs time, and large models can take many minutes end to end.

![Cold start is five phases, and each has a shortcut](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/86009eab-a911-4ee8-b931-747e1e1b64ea.svg align="center")

Recent releases attack each phase:

- **Weights:** keep them on a node-local cache or a pre-populated volume, or stream them with the Run:ai model streamer or similar loaders.
- **Load:** v0.30–0.31 added a weight-cache daemon (`vllm preload`) that keeps post-quantized, sharded weights resident in GPU memory so a restarting engine maps them over CUDA IPC with `--load-format ipc_cache` instead of reloading from disk. Experimental `vllm snapshot` restores a fully initialised engine with CRIU.
- **Memory profiling:** vLLM logs the exact `--kv-cache-memory` value for the current setup; passing it back skips profiling on the next boot on identical hardware.
- **Compilation:** the torch.compile cache under `VLLM_CACHE_ROOT` can be baked into the image or a shared volume.
- **Kernels:** v0.32.0 added `vllm download-kernels` for FlashInfer's precompiled kernels and an autotune table that survives restarts. v0.33.0 starts tensor-parallel workers concurrently (about 50 seconds saved at TP8) and shares one Triton JIT compile pool across processes (9–28% faster cold warmup), both according to the release notes.

Treat these as optimisations to adopt after you have measured your cold start; several are new and marked experimental.

## Production tuning: the flags that matter

Most vLLM tuning comes down to one budget: how GPU memory is split between weights and KV cache, and how the scheduler spends that KV cache per step. A handful of flags control it; everything else is model-specific.

![On an 80 GB H100, about two thirds of memory becomes KV cache](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/6e8a3cb6-3f73-4970-b137-a758eee27e4e.svg align="center")

### The core flags

| Flag | Default | What it controls | How to choose |
| --- | --- | --- | --- |
| `--max-model-len` | Model's maximum | Longest prompt plus output per request | Set to what your product needs, not the model's 128k. It bounds per-request KV and drives startup checks. |
| `--gpu-memory-utilization` | 0.9 | Fraction of GPU memory vLLM may claim | Raise to 0.92–0.95 on dedicated GPUs; lower if anything else shares the GPU. Not an isolation boundary. |
| `--max-num-seqs` | Hardware-dependent | Requests in the running batch | Higher means more throughput and higher ITL; lower it if you see sustained preemption. |
| `--max-num-batched-tokens` | Hardware-dependent | Tokens per step | Lower for ITL, higher for TTFT and throughput (see the scheduler section). |
| `--kv-cache-dtype` | `auto` | KV precision | `fp8` roughly doubles the token pool; validate quality. |
| `--tensor-parallel-size` | 1 | GPUs per replica | Smallest TP that fits the weights plus the KV your context lengths need. |
| `--enable-prefix-caching` | On | Automatic prefix caching | Leave on; disable only for benchmarks that must not hit the cache. |
| `--scheduling-policy` | `fcfs` | FCFS or priority | Use `priority` when interactive and batch traffic share a replica. |
| `--api-server-count` | 1 (DP otherwise) | Frontend processes | Raise when tokenization or multimodal preprocessing is the bottleneck. |
| `--served-model-name` | Model path | Name clients send | Match your gateway and LiteLLM aliases. |
| `--generation-config` | `auto` | Default sampling parameters | `auto` applies the model repo's `generation_config.json`; set `vllm` if you want vLLM's neutral defaults. |

Defaults for `max_num_seqs` and `max_num_batched_tokens` depend on hardware and usage context and have changed across releases, so read the values from the startup log rather than assuming them.

### Context length is a capacity decision

Requests do not reserve their full `max_model_len` of KV up front; blocks are allocated on demand. So a high `max-model-len` does not directly waste memory. It matters in two other ways: it raises the worst case a single request can consume (one 128k request on Llama-3.1-8B needs 16 GiB of KV), and vLLM refuses to start if even one maximum-length request cannot fit in the pool. Keep `max-model-len`, your gateway's input/output limits and LiteLLM's `max_input_tokens` consistent. Note that v0.32.0 changed the behaviour of `max_tokens`, which is no longer reserved from the context window, so recheck limits when upgrading.

### Symptom to knob

| Symptom | Likely cause | First knobs to try |
| --- | --- | --- |
| p95 TTFT high, queue growing | Not enough capacity, or prefill starved | Add replicas; raise `max-num-batched-tokens`; check prefix-cache hit rate and routing |
| ITL spikes while long prompts arrive | Prefill chunks crowding decodes | Lower `max-num-batched-tokens`; consider P/D disaggregation for very long prompts |
| `vllm:num_preemptions` rising | KV pool oversubscribed | Lower `max-num-seqs`; FP8 KV cache; more TP; shorter `max-model-len` |
| Startup fails with insufficient KV memory | One max-length request does not fit | Lower `max-model-len`, raise `gpu-memory-utilization`, add TP, or quantize |
| GPU utilisation low, latency high | CPU starvation or tokenizer bottleneck | More physical cores; more API servers; check CPU limits |
| Throughput plateaus well below expectations | Batch too small or graphs not used | Raise `max-num-seqs`; confirm CUDA graphs are captured (not `--enforce-eager`) |

### Tune with a benchmark, not by feel

`vllm bench serve` replays a realistic arrival pattern against a running server and reports TTFT, ITL, end-to-end latency and throughput. Newer releases add `vllm bench sweep` to run parameter sweeps and plot the Pareto frontier. Use your own prompt and output length distribution; a benchmark with 128-token prompts tells you nothing about a RAG workload with 6,000-token prompts.

```bash
vllm bench serve \
  --model Qwen/Qwen3-8B \
  --dataset-name random --random-input-len 2000 --random-output-len 300 \
  --request-rate 8 --max-concurrency 64 --num-prompts 1000
```

## Observability: the metrics that explain user experience

vLLM exposes Prometheus metrics on `/metrics` that map directly onto the mechanisms above: queue depth and KV usage describe pressure, preemptions describe oversubscription, and the latency histograms describe what users feel. Build SLOs on TTFT and ITL, and use the rest to explain why they move.

![Each vLLM latency metric measures a different slice of one request](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/f31a3284-0f7d-4095-b790-3df64f349ae0.svg align="center")

### The metrics worth a dashboard

All names below come from the [production metrics reference](https://docs.vllm.ai/en/latest/usage/metrics/); counters gain a `_total` suffix in Prometheus.

| Question | Metric | Type |
| --- | --- | --- |
| Are users waiting for the first token? | `vllm:time_to_first_token_seconds` | Histogram |
| Is streaming smooth? | `vllm:inter_token_latency_seconds` | Histogram |
| Where does the time go? | `vllm:request_queue_time_seconds`, `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` | Histograms |
| Is the replica saturated? | `vllm:num_requests_waiting`, `vllm:num_requests_running` | Gauges |
| Why are requests waiting? | `vllm:num_requests_waiting_by_reason` | Gauge |
| Is the KV pool full? | `vllm:kv_cache_usage_perc` (1.0 = 100%) | Gauge |
| Are we recomputing work? | `vllm:num_preemptions` | Counter |
| Is the prefix cache paying off? | `vllm:prefix_cache_hits`, `vllm:prefix_cache_queries` | Counters |
| Are we shedding load? | `vllm:admission_rejections` | Counter |
| How much work is flowing? | `vllm:prompt_tokens`, `vllm:generation_tokens` | Counters |

The distinction between `request_queue_time` and `request_prefill_time` is the most useful one when TTFT degrades. High queue time means you lack capacity (scale out); high prefill time with low queue time means prompts are long or the token budget is small (tune or disaggregate).

### Queries to start with

```promql
# p95 TTFT and p95 ITL per model
histogram_quantile(0.95, sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le, model_name) (rate(vllm:inter_token_latency_seconds_bucket[5m])))

# Is TTFT queueing or computing? p95 of each component
histogram_quantile(0.95, sum by (le) (rate(vllm:request_queue_time_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le) (rate(vllm:request_prefill_time_seconds_bucket[5m])))

# Saturation per pod
max by (pod) (vllm:kv_cache_usage_perc)
sum by (pod) (vllm:num_requests_waiting)

# Preemptions per second
sum by (pod) (rate(vllm:num_preemptions_total[5m]))

# Generated tokens per second across the fleet
sum(rate(vllm:generation_tokens_total[1m]))
```

Histogram buckets matter for SLOs. The request-phase histograms start at 0.3 s by default; if your SLO is tighter, set `--custom-histogram-buckets` (for example a `time_to_first_token` family with sub-100 ms buckets) so the quantiles are not interpolated across one wide bucket.

### Alerts that are actually actionable

- **TTFT SLO burn:** p95 TTFT above target for 10 minutes. Page.
- **Sustained queue:** `num_requests_waiting` above zero on every replica for longer than your scale-out time. Scale or investigate the autoscaler.
- **KV pressure with preemption:** `kv_cache_usage_perc` above 0.95 *and* preemption rate above zero. Capacity or configuration problem.
- **Admission rejections:** any sustained rate means users are being turned away by queue limits.
- **Prefix hit rate drop** after a deploy: often a prompt template change that put a volatile field first, or a routing change.

Pair these with GPU telemetry from DCGM Exporter and with gateway metrics (route latency, upstream errors, token counts), so a latency regression can be traced to a replica, a GPU or a route. vLLM also supports OpenTelemetry tracing for per-request spans when aggregate histograms are not enough. Since v0.33.0, a watchdog also dumps stack traces from engine or worker processes that stop making progress, which turns a silent hang into a log entry you can alert on.

## Capacity planning: from users to replicas

The number of replicas you need is the larger of two bounds: enough KV memory to hold every in-flight conversation, and enough decode throughput to serve them at your ITL target. Both start from the same number, in-flight requests, which you get from Little's law.

![Replica count is the larger of a memory bound and a throughput bound](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/fd1b7a52-571e-4d26-8f64-99430cedb104.svg align="center")

### Step 1: turn users into in-flight requests

Users are not requests. What the engine sees is the number of requests in flight at once, which Little's law gives from the arrival rate and the time each request spends in the system:

$$L = \lambda \times W$$

The worked example assumes a coding assistant at peak:

| Input | Value | Where it comes from |
| --- | --- | --- |
| Active users at peak | 5,000 | Product analytics |
| Think time between requests | 60 s | Product analytics |
| Arrival rate λ | 5,000 ÷ 60 ≈ 83 req/s | Derived |
| Prompt length (with history) | 1,500 tokens | Gateway token metrics |
| Output length | 300 tokens | Gateway token metrics |
| Target TTFT, target ITL | 0.5 s, 25 ms | SLO |
| Time in system W | 0.5 + 300 × 0.025 ≈ 8 s | Derived |
| In-flight requests L | 83 × 8 ≈ 667 | Little's law |

These inputs are illustrative. Replace every one with your own measurements; output length especially varies by orders of magnitude between chat, coding agents and reasoning models.

### Step 2: the memory bound

Each in-flight request holds its prompt plus, on average, half its output in KV: about 1,650 tokens. On Llama-3.1-8B at 128 KiB per token, that is about 206 MiB per request, so 667 requests need about 134 GiB of KV. With roughly 50 GiB of KV pool per H100 (the earlier worked example), the memory bound is about **2.7 replicas**.

### Step 3: the throughput bound

Memory says how many sequences fit; it does not say they will decode fast enough. For that you need one measurement: how many concurrent streams a single replica sustains while p95 ITL stays under 25 ms on your prompt mix. Suppose `vllm bench serve` shows **200 streams**. Then 667 ÷ 200 gives a throughput bound of about **3.3 replicas**. Note that 200 streams × 206 MiB is about 41 GiB, which fits in the 50 GiB pool, so the two measurements agree.

### Step 4: round up and add headroom

Take the larger bound, round up to 4, and add at least one spare replica so a node failure or a rolling update does not breach the SLO: **5 replicas**, which is 5 H100s for this model. The autoscaler's `minReplicas` should cover the steady peak, and `maxReplicas` the burst you are willing to pay for.

### What changes the answer

- **Burstiness.** Little's law uses averages. Size for the peak of a short window (for example the busiest 5 minutes), not the daily mean.
- **Prefix caching.** Shared system prompts and conversation history reduce prefill work and TTFT, and shared blocks are stored once, which lowers effective KV per request.
- **Long tails.** A few 32k-token requests consume as much KV as dozens of normal ones. Cap `max-model-len` deliberately or route long contexts to a separate pool.
- **Reasoning models.** Thousands of output tokens per request raise W, and therefore L, several-fold. Re-run the arithmetic for every new model.
- **FP8 KV cache or a bigger GPU** shifts the memory bound; speculative decoding or a faster GPU shifts the throughput bound. Improve whichever bound is binding.

## Running vLLM on Kubernetes

On Kubernetes, a vLLM replica is a pod with whole GPUs, generous CPU, shared memory, a warm model cache and probes that tolerate a slow start. Its metrics then feed two loops: the router decides which pod gets each request, and the autoscaler decides how many pods exist.

![vLLM's metrics drive both routing and scaling on Kubernetes](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/7f05f26c-6da5-4921-a1fb-80463f34a755.svg align="center")

In practice vLLM usually runs under a controller such as KServe's `LLMInferenceService` or llm-d, with a GPU-aware scheduler such as KAI Scheduler placing the pods. Whatever wraps it, the pod underneath needs the same things. A plain Deployment makes them explicit:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3-8b
  namespace: models
spec:
  replicas: 2
  selector:
    matchLabels: {app: qwen3-8b}
  template:
    metadata:
      labels: {app: qwen3-8b}
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.33.0   # pin; never :latest
          args:
            - --model=Qwen/Qwen3-8B
            - --served-model-name=qwen3-8b
            - --max-model-len=32768
            - --gpu-memory-utilization=0.92
            - --max-num-seqs=256
            - --kv-cache-dtype=fp8
          env:
            - name: HF_HUB_OFFLINE          # weights come from the cache volume
              value: "1"
            - name: HF_HOME
              value: /models
          ports:
            - {name: http, containerPort: 8000}
          resources:
            requests: {nvidia.com/gpu: "1", cpu: "8", memory: 64Gi}
            limits:   {nvidia.com/gpu: "1", memory: 64Gi}   # no CPU limit
          startupProbe:                   # generous: load + compile + graphs
            httpGet: {path: /health, port: http}
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 5
          volumeMounts:
            - {name: models, mountPath: /models}
            - {name: dshm, mountPath: /dev/shm}
      volumes:
        - name: models
          persistentVolumeClaim: {claimName: model-cache}
        - name: dshm
          emptyDir: {medium: Memory, sizeLimit: 16Gi}
```

Why each piece is there:

- **Pinned image.** vLLM ships every two weeks with breaking changes in flags, defaults and metrics. Upgrade deliberately and re-benchmark. v0.33.0 publishes CUDA 13.0 images only (the CUDA 12.9 variants are gone), so GPU nodes need an R580 or newer driver before you upgrade. Every image is now also published under the shorter vllm/vllm name.
- **CPU request without a CPU limit.** The engine core is a busy loop; CFS throttling shows up as latency spikes. Follow the 2 + N physical cores floor and add more for multimodal or high-QPS short requests.
- `/dev/shm` **as memory.** Worker processes and NCCL communicate through shared memory; the container default is far too small for tensor parallelism.
- **A warm model cache.** `HF_HUB_OFFLINE=1` plus a pre-populated volume means a restart never depends on the public Hub.
- **A startup probe with a long budget.** Loading, compiling and graph capture can take minutes. Without a startup probe, the liveness or readiness logic kills or flaps the pod mid-boot.
- **A grace period.** Long streaming responses need time to finish when a pod is drained during a rollout.

The model choice drives the KV math. Qwen3-8B at 144 KiB per token in BF16 drops to roughly half with the FP8 KV cache, so the 32k `max-model-len` above needs about 2.3 GiB per maximum-length request.

### Routing: why round-robin is not enough

Each replica has its own KV pool and prefix cache. A plain Kubernetes Service balances connections, not work: it cannot see that one pod has 40 requests queued and another has none, or that the pod holding a conversation's history is the cheap place to send its next turn. The llm-d endpoint picker reads exactly the signals described in this post (waiting requests, KV usage, prefix-cache locality via KV events) and picks a pod per request. Newer control planes such as Kinference, announced in November 2026, combine the same cache-aware balancing with tenant rate limits, usage tracking and routing policies.

### Autoscaling: which signal

Scale on what saturates. GPU utilisation is a poor signal: decode can show high utilisation while still having room, or low utilisation while blocked on CPU. Better candidates:

- `vllm:num_requests_waiting` **per replica:** direct evidence that requests cannot be admitted.
- `vllm:kv_cache_usage_perc`**:** a leading indicator of preemption.
- **p95 TTFT:** closest to the user, but lagging and noisy at low traffic.

A KEDA trigger on queue depth is a reasonable start for a Deployment you manage directly:

```yaml
triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      query: sum(vllm:num_requests_waiting{namespace="models", model_name="qwen3-8b"})
      threshold: "4"   # target waiting requests per replica
```

If KServe owns the workload, use its workload-variant autoscaler instead, and never let two controllers fight over `replicas`. Whatever the trigger, scale-out takes as long as your cold start, so set thresholds that fire early enough to cover it.

## What changed in v0.33.0

vLLM v0.33.0 shipped on 2 November 2026 with 927 commits. Most of it is model support and kernels; these are the changes that touch what this post covers.

| Change | Why it matters here |
| --- | --- |
| Wheels and images are CUDA 13.0 only; CUDA 12.9 builds are gone | Roll out R580+ GPU drivers on your nodes before upgrading the image. |
| PyTorch 2.14, and the CUDA wheel is now stable against the libtorch ABI | Per the [Stable ABI write-up](https://vllm.ai/blog/2026-11-06-stable-abi-cuda), compatible PyTorch upgrades no longer require rebuilding vLLM. |
| Concurrent TP worker startup and a shared Triton JIT compile pool | Shorter cold start, which shortens your autoscaling reaction time. |
| Engine and worker watchdog | Stuck processes now dump stack traces instead of hanging silently. |
| DFlash and DSpark drafts no longer reserve token budget; Medusa and the MLP speculator removed | Re-tune speculative decoding, and migrate any Medusa configuration. |
| Lookahead block hashes on by default | EAGLE and MTP prefix-cache hits keep their last block. |
| NIXL connector requires NIXL 1.5+; new EFA image | Check P/D deployments before upgrading. |
| Requests sent while generation is paused return HTTP 503 | Make sure your gateway treats 503 from vLLM as retryable. |
| Rust frontend fails on unsupported flags and request fields | Catch these in staging rather than in production. |

The [full release notes](https://github.com/vllm-project/vllm/releases) list everything else, including new models and hardware-specific kernels.

## Key takeaways

- **Decode is bandwidth-bound, so batching is everything.** One decode step reads every weight; batching many requests into that step is how a GPU earns its keep.
- **The KV cache is your real capacity.** Compute KV bytes per token for every model you serve, and read the KV pool size from the startup log after every change.
- **PagedAttention turns memory into a shared pool.** On-demand blocks, block tables and reference counts are what make continuous batching, prefix caching and parallel sampling cheap.
- **The scheduler decides your latency trade-offs.** `max-num-batched-tokens` trades TTFT against ITL; `max-num-seqs` trades throughput against preemption.
- **Give the engine CPU.** The V1 engine is several processes with a busy loop; at least 2 + N physical cores for N GPUs, and no CPU limits.
- **Prefix caching needs both prompt design and routing.** Static content first, volatile content last, and a cache-aware router such as llm-d.
- **Speculative decoding and disaggregation are workload-specific.** Both pay off in the right conditions and cost you in the wrong ones; benchmark before enabling.
- **Size from Little's law.** Replicas are the larger of a memory bound and a measured throughput bound, plus headroom for failures and rollouts.
- **Scale on queue and KV signals, not CPU or GPU utilisation,** and remember that scale-out is only as fast as your cold start.

## Further reading

- [vLLM v0.33.0 release notes and earlier releases](https://github.com/vllm-project/vllm/releases)
- [vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention](https://vllm.ai/blog/vllm) (original announcement)
- [Inside vLLM: Anatomy of a High-Throughput LLM Inference System](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm) by Aleksa Gordić
- [vLLM architecture overview](https://docs.vllm.ai/en/latest/design/arch_overview/)
- [Automatic prefix caching design](https://docs.vllm.ai/en/latest/design/prefix_caching/)
- [Model Runner V2 design document](https://docs.vllm.ai/en/latest/design/model_runner_v2/)
- [Optimization and tuning guide](https://docs.vllm.ai/en/latest/configuration/optimization/)
- [Speculative decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/)
- [NixlConnector usage guide](https://docs.vllm.ai/en/latest/features/nixl_connector_usage/)
- [Taking vLLM Apart: A Practical Guide to Disaggregated Serving](https://vllm.ai/blog/2026-09-29-disaggregated-serving-guide)
- [Push-Based KV Transfer for Disaggregated Inference](https://vllm.ai/blog/2026-10-14-nixl-push-connector)
- [Tiered KV Cache Offloading in vLLM](https://vllm.ai/blog/2026-09-10-tiered-kv-offloading)
- [Production metrics reference](https://docs.vllm.ai/en/latest/usage/metrics/)
- [GLM-5.2 on 24 NVIDIA B300 GPUs with vLLM](https://vllm.ai/blog/2026-07-23-glm-5.2-nvfp4-b300-pd)
- [Near-zero-downtime Elastic Expert Parallelism](https://vllm.ai/blog/2026-10-19-elastic-ep-near-zero-downtime)
- [Introducing Kinference: an inference control plane for vLLM and Kubernetes](https://vllm.ai/blog/2026-11-04-introducing-kinference)