NVIDIA Dynamo vs KServe: Which Layer Should Own Your LLM Inference?
In my previous article I split a production inference platform into layers: LiteLLM for tenants and budgets, Envoy AI Gateway for model-aware routing, KServe LLMInferenceService for lifecycle, llm-d for picking the replica, and vLLM for generating tokens.
The question I kept getting afterwards was some version of "why not just use NVIDIA Dynamo?" It's a fair question, but it's slightly the wrong one. Dynamo and KServe overlap, and they aren't substitutes. Dynamo is a distributed inference framework: it coordinates KV-aware routing, prefill/decode disaggregation, KV transfer and latency-aware scaling. KServe is a Kubernetes model-serving platform whose GenAI path composes llm-d, vLLM and the Gateway API Inference Extension. So the useful question is narrower: for each model pool, which component owns endpoint selection, KV state and replica count?
Version baseline (checked 19 October 2026): Dynamo v1.5.1 (6 Oct), KServe v0.21.0 (25 Sep), llm-d v0.10.0 (29 Sep, bundling llm-d-router v0.11.0). Upstream vLLM is at 0.31.0. Dynamo has started publishing v1.6.0 model-specific dev builds, but those are previews and not QA-gated. This is an architecture comparison, not a benchmark. All three projects shipped breaking changes in the last month, so pin versions as a set.
The short answer
Start with Dynamo when distributed LLM serving is the hard problem: large models, long shared prompts, separate prefill and decode pools, and a scaler that should reason in TTFT and inter-token latency rather than CPU.
Start with KServe + llm-d when you want one Kubernetes-native serving API across many models (predictive and generative), and you prefer to compose routing, autoscaling and telemetry from replaceable parts.
Keep governance outside both. Neither an endpoint picker nor an inference router is an identity, quota or spend system. NVIDIA's own secure deployment guidance puts Dynamo behind a gateway that handles auth, TLS and rate limits.
Don't disaggregate by default. Measure an aggregated baseline first.
What each control plane actually creates
Both stacks are controllers that turn one custom resource into ordinary Kubernetes objects, and the shape of that tree tells you who owns what. On the KServe side, LLMInferenceService (composed with reusable LLMInferenceServiceConfig presets) renders the vLLM workload, an optional prefill workload, a Gateway API HTTPRoute, a GAIE InferencePool, the llm-d EPP, and since 0.21 a KEDA ScaledObject. The InferencePool is the hinge: the route points at it, it selects the model-server pods, and it names the EPP that picks among them.
On the Dynamo side, you can start one level higher. A DynamoGraphDeploymentRequest states intent (model, SLA, workload), a profiling job turns it into a DynamoGraphDeployment, and the operator renders one DynamoComponentDeployment per component on Deployments, Grove PodCliques or LeaderWorkerSets. The Planner doesn't touch Deployments directly; it sets replicas through a scaling adapter.
The structural difference is ownership. KServe deliberately hands inference scheduling to llm-d and keeps the lifecycle API. Dynamo pulls the scheduling loop, the KV transfer path and the scaling model into one system designed together.
Where a request actually goes
On Path A, Envoy never decides on its own. For each request it calls the EPP over gRPC ext_proc (port 9002 by default in KServe), the EPP runs flow control, filters, scorers and a picker, and Envoy forwards to whichever pod it returns. On Path B, Dynamo's Frontend tokenizes the request itself, the KV Router runs in the same process, and the request goes to the chosen worker over Dynamo's own request plane (TCP, with opt-in TLS or mTLS since 1.5, or NATS). Workers register through Kubernetes or etcd discovery and publish KV events on the event plane. If you want Dynamo behind a Gateway API front door instead, its Rust EPP slots into Path A.
What changed in September 2026
If you compared these projects in the summer, three releases moved the baseline. These are the changes that matter for this comparison.
Dynamo 1.5 deprecated the KV Block Manager. KVBM was one of Dynamo's most-cited differentiators. It is now deprecated, removal is targeted for v1.6.0, and NVIDIA's migration advice is to use the engine's native KV offloading for host and disk tiers (v1.5.0 notes). Cross-node cache sharing moves to a separate, early-stage project called KV Cache Runner. Meanwhile cache-tier awareness moves into the Router: it consumes Mooncake shared-cache events for SGLang HiCache and routes vLLM storage KV events to a disk tier. If an evaluation plan assumes KVBM, rewrite it.
Dynamo 1.5 also made worker selection pluggable (custom scorer, picker and filter policies), added a conditional disaggregation bypass that sends short requests straight to decode, replaced the Go EPP with a Rust EPP shipped inside the Frontend image, promoted CRD storage to nvidia.com/v1beta1, and added opt-in TLS and mTLS on its TCP and NATS transports. AIConfigurator was renamed AISimulate. The v1.5.1 patch (6 Oct) fixed Router overload recovery and Anthropic Messages API requests from clients like Claude Code, and carries two breaking changes of its own, so read its notes before a "patch" upgrade (v1.5.1 notes).
KServe 0.21 tightened the LLM path. LLMInferenceService gained direct KEDA scaling with idleReplicaCount=0 for real scale-to-zero, a rolloutStrategy field, and LocalModelCache support for LoRA adapters. Docs samples moved from v1alpha1 to v1alpha2. One fix deserves attention: P/D engines now receive a NixlConnector so KV state actually transfers (v0.21.0 notes). If you tested disaggregation on 0.20 and it underwhelmed, retest.
llm-d 0.10 made KEDA + EPP the autoscaling path. The Workload Variant Autoscaler guides are deprecated, the repo was renamed llm-d-autoscaling, and the KV-cache library moved into llm-d-router. The filesystem offload connector was merged upstream into vLLM's OffloadingConnector, so it no longer needs a separate install. The release also moved to upstream vLLM 0.30 images and LWS DisaggregatedSet as the base for wide-EP P/D (v0.10.0 notes).
The router inside it, llm-d-router v0.11.0, is where most of the behaviour changed (v0.11.0 notes). Flow control is now stage-aware for P/D, gating admission on whichever of prefill or decode is busier. A new latency-observation-scorer ranks endpoints by TTFT measured from real responses. And the upgrade is not gentle: plugins deprecated since v0.7 and v0.9 (including pd-profile-handler and precise-prefix-cache-scorer) are removed and fail at startup, and the legacy inference_extension_* and inference-pool metric families are gone in favour of llm_d_epp_*. That last one bites autoscaling. llm-d's own async processor at v0.10 still queried the removed metrics, fell back to a zero budget against the new router, and never dispatched queued requests until v0.11 fixed it. Check every KEDA query and alert before upgrading.
One practical consequence: Dynamo 1.5 pins vLLM 0.28.0, llm-d 0.10 ships vLLM 0.30.0, and upstream is already at 0.31.0. A head-to-head that doesn't match engine versions is partly a vLLM benchmark.
How cache-aware routing works
Both routers work from the same raw material. The engine splits the KV cache into fixed-size token blocks, hashes each block as a chain, and publishes BlockStored and BlockRemoved events as blocks enter and leave each tier. The router hashes the incoming prompt the same way and asks its index which pod already holds the longest matching prefix.
They differ in what happens next. llm-d's precise-prefix-cache-producer maintains the index and hands per-pod prefix matches to a set of weighted scorers (prefix cache with per-tier weights, load, and since router 0.11 observed TTFT); a max-score picker rotates ties so one pod doesn't absorb a whole burst. Dynamo keeps a radix tree across device, disk and Mooncake tiers, applies a cost policy that trades overlap against active load, then runs worker filters and session affinity. In both, cache hits only help if the scorer is allowed to prefer a slightly busier pod, so tune the weights on your real prompt mix.
How a disaggregated request flows
Prefill turns the prompt into KV blocks and is compute-heavy. Decode reads those blocks to generate tokens and is memory-bandwidth-bound. Putting them on separate pools stops long prompts from stalling everyone else's token stream, and lets each pool use different parallelism.
In the llm-d flow, the request lands on the decode pod, where a sidecar runs the prefill leg on a prefill pod with max_tokens=1. The prefill side returns kv_transfer_params, and the decode engine pulls the blocks with a NIXL read over UCX (InfiniBand or RoCE) or libfabric (EFA). Dynamo moves the KV the same way, but its Router orchestrates the two legs itself instead of a sidecar. The router decides where a request runs; the transfer layer moves state. They fail differently, so monitor them separately.
The cost is real. You keep two pools warm, you depend on the RDMA path (a fallback to plain TCP isn't the same deployment), and short requests pay a transfer they didn't need. Both stacks now address that last point. Dynamo's conditional bypass sends short requests straight to decode, and in llm-d the prefix-based-pd-decider plugin decides per request whether remote prefill is worth it, based on how many uncached tokens remain. Note one llm-d behaviour change: when a required prefill stage has no endpoint, the request now fails instead of quietly running prefill on the decode pod. Below sustained concurrency, an aggregated worker is usually the better choice.
Scaling is two loops, not one
The KServe path is an assembled loop: vLLM and EPP metrics go to Prometheus, a KEDA ScaledObject turns a PromQL query into an HPA target, and the HPA scales the workload. It's flexible, and llm-d 0.10 made it the recommended path, but the policy lives in your queries and thresholds. Dynamo's Planner reads forward-pass metrics and router load directly and can target throughput, latency, load or sla. In SLA mode, a throughput-based floor from a performance model combines with load-based bursts above it, all clamped by guardrails such as minimum endpoints and GPU power caps (Planner docs). llm-d's router can now route on observed TTFT, but turning a TTFT target into a replica count is still a loop you build on the KServe side.
Neither stack owns the second loop. A new replica that doesn't fit stays Pending until a node autoscaler provisions a GPU node, the GPU Operator (or DRA driver) makes it schedulable, and the weights load. That chain usually dominates scale-out time, so measure it before tuning either replica loop.
Who owns what
The two stacks look most alike at the edges (gateway, GPU capacity) and differ most in the middle: who keeps the KV index, who drives the P/D handoff, and whether replica count comes from a generic autoscaler or an inference-specific planner.
Where they meet
These ecosystems now plug into each other more than the "vs" framing suggests:
Dynamo's EPP discovers workers from a Gateway API Inference Extension
InferencePool, so Dynamo can sit behind the same Gateway API front door as llm-d.Dynamo 1.4 added a vLLM-compatible
/inference/v1/generateendpoint so vLLM and llm-d clients can target Dynamo's token path without reconfiguration, and 1.5 ships an llm-d Batch Gateway example.Both use NIXL for transfer, and both can work with LMCache and Mooncake.
That makes one rule more important, not less. Pick one authoritative endpoint picker per model pool. LiteLLM, Envoy AI Gateway, llm-d EPP and the Dynamo Router can all "route". If two of them make independent replica decisions for the same pool, cache affinity breaks and every incident becomes a question of which router lied. Outer gateways route by model name and policy; exactly one inner component picks the replica.
How I'd choose
Dynamo first if inference is the main workload, you run large or MoE models across nodes, prompts share long prefixes (agents, RAG, coding assistants), and you want TTFT/ITL-targeted scaling without writing it yourself. Budget time for its graph model and its release pace: 44 breaking changes in 1.5 alone.
KServe + llm-d first if you serve many model types, already run Gateway API, KEDA, Prometheus and GitOps, and want each layer replaceable. Inference-aware routing and P/D are available here too; you assemble more of the control loop. The pace isn't slower on this side either: the last llm-d router release removed a batch of plugins and metric families in one go.
Neither yet if traffic is low or bursty, prefixes rarely repeat, or you have no baseline numbers. One vLLM or SGLang deployment behind a gateway is a legitimate production architecture.
How I'd evaluate
Baseline first. Same model, quantization, GPU shape and engine version on both paths. Record P50/P95/P99 TTFT, inter-token latency, tokens/s, queue depth and GPU memory.
Use real prompts. Replay traffic with your actual prefix reuse and cache misses. Check whether cache-aware routing reduces prefill work or just moves load around.
Test P/D separately. Measure transfer latency on the production network, and compare against the aggregated baseline at your real concurrency.
Burst and break it. Time scale-up end to end (decision, GPU scheduling, weight load, ready). Kill a decode pod mid-stream, drain a node, roll back a release.
Rehearse the upgrade. Both stacks rename metrics and remove config between minor versions. Run your dashboards, alerts and KEDA queries against the new release before production does.
Hold governance constant. Same TLS, auth, quotas and logging in front of both. Benchmarking a raw internal endpoint against a governed public one tells you nothing.
Where I landed
For a platform already built on KServe, Envoy AI Gateway, LiteLLM and vLLM, I'd keep that baseline and adopt llm-d's routing and the new direct KEDA scaling incrementally. In parallel I'd run a Dynamo proof of concept on the workloads most likely to benefit: long shared prompts, sustained high concurrency, and strict latency targets. Same gateway in front, same load test, matched engine versions.
The architecture worth keeping is the one whose routing, cache behaviour and failure modes you can explain under real traffic. Feature lists change every month; that test doesn't.
Sources
NVIDIA Dynamo
KServe and llm-d
Previous article
