# NVIDIA Dynamo vs KServe: Which Layer Should Own Your LLM Inference?

In my [previous article](https://blog.shubhamtatvamasi.com/running-llms-on-kubernetes-in-production-kserve-vllm-llm-d-envoy-ai-gateway-and-litellm) I split a production inference platform into layers: LiteLLM for tenants and budgets, Envoy AI Gateway for model-aware routing, KServe `LLMInferenceService` for lifecycle, llm-d for picking the replica, and vLLM for generating tokens.

The question I kept getting afterwards was some version of "why not just use NVIDIA Dynamo?" It's a fair question, but it's slightly the wrong one. Dynamo and KServe overlap, and they aren't substitutes. Dynamo is a distributed inference framework: it coordinates KV-aware routing, prefill/decode disaggregation, KV transfer and latency-aware scaling. KServe is a Kubernetes model-serving platform whose GenAI path composes llm-d, vLLM and the Gateway API Inference Extension. So the useful question is narrower: **for each model pool, which component owns endpoint selection, KV state and replica count?**

> **Version baseline (checked 19 October 2026):** Dynamo v1.5.1 (6 Oct), KServe v0.21.0 (25 Sep), llm-d v0.10.0 (29 Sep, bundling llm-d-router v0.11.0). Upstream vLLM is at 0.31.0. Dynamo has started publishing v1.6.0 model-specific dev builds, but those are previews and not QA-gated. This is an architecture comparison, not a benchmark. All three projects shipped breaking changes in the last month, so pin versions as a set.

## The short answer

*   **Start with Dynamo** when distributed LLM serving is the hard problem: large models, long shared prompts, separate prefill and decode pools, and a scaler that should reason in TTFT and inter-token latency rather than CPU.
    
*   **Start with KServe + llm-d** when you want one Kubernetes-native serving API across many models (predictive and generative), and you prefer to compose routing, autoscaling and telemetry from replaceable parts.
    
*   **Keep governance outside both.** Neither an endpoint picker nor an inference router is an identity, quota or spend system. NVIDIA's own [secure deployment guidance](https://docs.nvidia.com/dynamo/dev/security/secure-deployment-guidelines) puts Dynamo behind a gateway that handles auth, TLS and rate limits.
    
*   **Don't disaggregate by default.** Measure an aggregated baseline first.
    

## What each control plane actually creates

![Kubernetes objects reconciled by KServe LLMInferenceService and by Dynamo DynamoGraphDeployment](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/3691b2a6-1159-4b12-9b5f-0672eedc93b1.svg align="center")

Both stacks are controllers that turn one custom resource into ordinary Kubernetes objects, and the shape of that tree tells you who owns what. On the KServe side, `LLMInferenceService` (composed with reusable `LLMInferenceServiceConfig` presets) renders the vLLM workload, an optional prefill workload, a Gateway API `HTTPRoute`, a GAIE `InferencePool`, the llm-d EPP, and since 0.21 a KEDA `ScaledObject`. The `InferencePool` is the hinge: the route points at it, it selects the model-server pods, and it names the EPP that picks among them.

On the Dynamo side, you can start one level higher. A `DynamoGraphDeploymentRequest` states intent (model, SLA, workload), a profiling job turns it into a `DynamoGraphDeployment`, and the operator renders one `DynamoComponentDeployment` per component on Deployments, Grove PodCliques or LeaderWorkerSets. The Planner doesn't touch Deployments directly; it sets replicas through a scaling adapter.

The structural difference is ownership. KServe deliberately hands inference scheduling to llm-d and keeps the lifecycle API. Dynamo pulls the scheduling loop, the KV transfer path and the scaling model into one system designed together.

### Where a request actually goes

![Request data path with protocols for the Gateway API + llm-d EPP path and the Dynamo Frontend path](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/d61a01b0-9667-4c71-ad6f-2308e6537d42.svg align="center")

On Path A, Envoy never decides on its own. For each request it calls the EPP over gRPC `ext_proc` (port 9002 by default in KServe), the EPP runs flow control, filters, scorers and a picker, and Envoy forwards to whichever pod it returns. On Path B, Dynamo's Frontend tokenizes the request itself, the KV Router runs in the same process, and the request goes to the chosen worker over Dynamo's own request plane (TCP, with opt-in TLS or mTLS since 1.5, or NATS). Workers register through Kubernetes or etcd discovery and publish KV events on the event plane. If you want Dynamo behind a Gateway API front door instead, its Rust EPP slots into Path A.

## What changed in September 2026

If you compared these projects in the summer, three releases moved the baseline. These are the changes that matter for this comparison.

**Dynamo 1.5 deprecated the KV Block Manager.** KVBM was one of Dynamo's most-cited differentiators. It is now deprecated, removal is targeted for v1.6.0, and NVIDIA's migration advice is to use the engine's native KV offloading for host and disk tiers ([v1.5.0 notes](https://github.com/ai-dynamo/dynamo/releases/tag/v1.5.0)). Cross-node cache sharing moves to a separate, early-stage project called KV Cache Runner. Meanwhile cache-tier awareness moves into the Router: it consumes Mooncake shared-cache events for SGLang HiCache and routes vLLM storage KV events to a disk tier. If an evaluation plan assumes KVBM, rewrite it.

Dynamo 1.5 also made worker selection pluggable (custom scorer, picker and filter policies), added a **conditional disaggregation bypass** that sends short requests straight to decode, replaced the Go EPP with a Rust EPP shipped inside the Frontend image, promoted CRD storage to `nvidia.com/v1beta1`, and added opt-in TLS and mTLS on its TCP and NATS transports. AIConfigurator was renamed AISimulate. The v1.5.1 patch (6 Oct) fixed Router overload recovery and Anthropic Messages API requests from clients like Claude Code, and carries two breaking changes of its own, so read its notes before a "patch" upgrade ([v1.5.1 notes](https://github.com/ai-dynamo/dynamo/releases/tag/v1.5.1)).

**KServe 0.21 tightened the LLM path.** `LLMInferenceService` gained direct KEDA scaling with `idleReplicaCount=0` for real scale-to-zero, a `rolloutStrategy` field, and LocalModelCache support for LoRA adapters. Docs samples moved from `v1alpha1` to `v1alpha2`. One fix deserves attention: P/D engines now receive a `NixlConnector` so KV state actually transfers ([v0.21.0 notes](https://github.com/kserve/kserve/releases/tag/v0.21.0)). If you tested disaggregation on 0.20 and it underwhelmed, retest.

**llm-d 0.10 made KEDA + EPP the autoscaling path.** The Workload Variant Autoscaler guides are deprecated, the repo was renamed `llm-d-autoscaling`, and the KV-cache library moved into `llm-d-router`. The filesystem offload connector was merged upstream into vLLM's `OffloadingConnector`, so it no longer needs a separate install. The release also moved to upstream vLLM 0.30 images and LWS DisaggregatedSet as the base for wide-EP P/D ([v0.10.0 notes](https://github.com/llm-d/llm-d/releases/tag/v0.10.0)).

The router inside it, llm-d-router v0.11.0, is where most of the behaviour changed ([v0.11.0 notes](https://github.com/llm-d/llm-d-router/releases/tag/v0.11.0)). Flow control is now stage-aware for P/D, gating admission on whichever of prefill or decode is busier. A new `latency-observation-scorer` ranks endpoints by TTFT measured from real responses. And the upgrade is not gentle: plugins deprecated since v0.7 and v0.9 (including `pd-profile-handler` and `precise-prefix-cache-scorer`) are removed and fail at startup, and the legacy `inference_extension_*` and inference-pool metric families are gone in favour of `llm_d_epp_*`. That last one bites autoscaling. llm-d's own async processor at v0.10 still queried the removed metrics, fell back to a zero budget against the new router, and never dispatched queued requests until v0.11 fixed it. Check every KEDA query and alert before upgrading.

One practical consequence: Dynamo 1.5 pins vLLM 0.28.0, llm-d 0.10 ships vLLM 0.30.0, and upstream is already at 0.31.0. A head-to-head that doesn't match engine versions is partly a vLLM benchmark.

## How cache-aware routing works

![KV event plane: engines publish block events, llm-d EPP and Dynamo Router index them and score candidates](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/6bf09e19-7c11-465d-b758-02d73c7dbcc8.svg align="center")

Both routers work from the same raw material. The engine splits the KV cache into fixed-size token blocks, hashes each block as a chain, and publishes `BlockStored` and `BlockRemoved` events as blocks enter and leave each tier. The router hashes the incoming prompt the same way and asks its index which pod already holds the longest matching prefix.

They differ in what happens next. llm-d's `precise-prefix-cache-producer` maintains the index and hands per-pod prefix matches to a set of weighted scorers (prefix cache with per-tier weights, load, and since router 0.11 observed TTFT); a max-score picker rotates ties so one pod doesn't absorb a whole burst. Dynamo keeps a radix tree across device, disk and Mooncake tiers, applies a cost policy that trades overlap against active load, then runs worker filters and session affinity. In both, cache hits only help if the scorer is allowed to prefer a slightly busier pod, so tune the weights on your real prompt mix.

## How a disaggregated request flows

![Prefill/decode disaggregation sequence with NIXL transfer and the conditional bypass](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/f4a00c9c-726f-4f1b-9dfb-a2eb96193936.svg align="center")

Prefill turns the prompt into KV blocks and is compute-heavy. Decode reads those blocks to generate tokens and is memory-bandwidth-bound. Putting them on separate pools stops long prompts from stalling everyone else's token stream, and lets each pool use different parallelism.

In the llm-d flow, the request lands on the decode pod, where a sidecar runs the prefill leg on a prefill pod with `max_tokens=1`. The prefill side returns `kv_transfer_params`, and the decode engine pulls the blocks with a NIXL read over UCX (InfiniBand or RoCE) or libfabric (EFA). Dynamo moves the KV the same way, but its Router orchestrates the two legs itself instead of a sidecar. The router decides where a request runs; the transfer layer moves state. They fail differently, so monitor them separately.

The cost is real. You keep two pools warm, you depend on the RDMA path (a fallback to plain TCP isn't the same deployment), and short requests pay a transfer they didn't need. Both stacks now address that last point. Dynamo's conditional bypass sends short requests straight to decode, and in llm-d the `prefix-based-pd-decider` plugin decides per request whether remote prefill is worth it, based on how many uncached tokens remain. Note one llm-d behaviour change: when a required prefill stage has no endpoint, the request now fails instead of quietly running prefill on the decode pod. Below sustained concurrency, an aggregated worker is usually the better choice.

## Scaling is two loops, not one

![Autoscaling control loops: KServe KEDA loop, Dynamo Planner loop, and the shared GPU capacity loop](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/342a5d7b-0342-4eb3-8b13-8cc4aeec497f.svg align="center")

The KServe path is an assembled loop: vLLM and EPP metrics go to Prometheus, a KEDA `ScaledObject` turns a PromQL query into an HPA target, and the HPA scales the workload. It's flexible, and llm-d 0.10 made it the recommended path, but the policy lives in your queries and thresholds. Dynamo's Planner reads forward-pass metrics and router load directly and can target `throughput`, `latency`, `load` or `sla`. In SLA mode, a throughput-based floor from a performance model combines with load-based bursts above it, all clamped by guardrails such as minimum endpoints and GPU power caps ([Planner docs](https://docs.nvidia.com/dynamo/latest/components/planner.md)). llm-d's router can now route on observed TTFT, but turning a TTFT target into a replica count is still a loop you build on the KServe side.

Neither stack owns the second loop. A new replica that doesn't fit stays `Pending` until a node autoscaler provisions a GPU node, the GPU Operator (or DRA driver) makes it schedulable, and the weights load. That chain usually dominates scale-out time, so measure it before tuning either replica loop.

## Who owns what

![Ownership matrix layer by layer for KServe + llm-d and NVIDIA Dynamo](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/7a858609-9fc6-4c83-9171-e628711420e0.svg align="center")

The two stacks look most alike at the edges (gateway, GPU capacity) and differ most in the middle: who keeps the KV index, who drives the P/D handoff, and whether replica count comes from a generic autoscaler or an inference-specific planner.

## Where they meet

These ecosystems now plug into each other more than the "vs" framing suggests:

*   Dynamo's EPP discovers workers from a Gateway API Inference Extension `InferencePool`, so Dynamo can sit behind the same Gateway API front door as llm-d.
    
*   Dynamo 1.4 added a vLLM-compatible `/inference/v1/generate` endpoint so vLLM and llm-d clients can target Dynamo's token path without reconfiguration, and 1.5 ships an llm-d Batch Gateway example.
    
*   Both use NIXL for transfer, and both can work with LMCache and Mooncake.
    

That makes one rule more important, not less. **Pick one authoritative endpoint picker per model pool.** LiteLLM, Envoy AI Gateway, llm-d EPP and the Dynamo Router can all "route". If two of them make independent replica decisions for the same pool, cache affinity breaks and every incident becomes a question of which router lied. Outer gateways route by model name and policy; exactly one inner component picks the replica.

## How I'd choose

![Decision flow: when to stay simple, when to pick KServe + llm-d or Dynamo, and when to disaggregate](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/3bcaa17f-bd59-42a0-b935-980ba70196e5.svg align="center")

**Dynamo first** if inference is the main workload, you run large or MoE models across nodes, prompts share long prefixes (agents, RAG, coding assistants), and you want TTFT/ITL-targeted scaling without writing it yourself. Budget time for its graph model and its release pace: 44 breaking changes in 1.5 alone.

**KServe + llm-d first** if you serve many model types, already run Gateway API, KEDA, Prometheus and GitOps, and want each layer replaceable. Inference-aware routing and P/D are available here too; you assemble more of the control loop. The pace isn't slower on this side either: the last llm-d router release removed a batch of plugins and metric families in one go.

**Neither yet** if traffic is low or bursty, prefixes rarely repeat, or you have no baseline numbers. One vLLM or SGLang deployment behind a gateway is a legitimate production architecture.

## How I'd evaluate

1.  **Baseline first.** Same model, quantization, GPU shape and engine version on both paths. Record P50/P95/P99 TTFT, inter-token latency, tokens/s, queue depth and GPU memory.
    
2.  **Use real prompts.** Replay traffic with your actual prefix reuse and cache misses. Check whether cache-aware routing reduces prefill work or just moves load around.
    
3.  **Test P/D separately.** Measure transfer latency on the production network, and compare against the aggregated baseline at your real concurrency.
    
4.  **Burst and break it.** Time scale-up end to end (decision, GPU scheduling, weight load, ready). Kill a decode pod mid-stream, drain a node, roll back a release.
    
5.  **Rehearse the upgrade.** Both stacks rename metrics and remove config between minor versions. Run your dashboards, alerts and KEDA queries against the new release before production does.
    
6.  **Hold governance constant.** Same TLS, auth, quotas and logging in front of both. Benchmarking a raw internal endpoint against a governed public one tells you nothing.
    

## Where I landed

For a platform already built on KServe, Envoy AI Gateway, LiteLLM and vLLM, I'd keep that baseline and adopt llm-d's routing and the new direct KEDA scaling incrementally. In parallel I'd run a Dynamo proof of concept on the workloads most likely to benefit: long shared prompts, sustained high concurrency, and strict latency targets. Same gateway in front, same load test, matched engine versions.

The architecture worth keeping is the one whose routing, cache behaviour and failure modes you can explain under real traffic. Feature lists change every month; that test doesn't.

## Sources

**NVIDIA Dynamo**

*   [Dynamo v1.5.0 release notes](https://github.com/ai-dynamo/dynamo/releases/tag/v1.5.0) and [docs mirror](https://docs.nvidia.com/dynamo/dev/reference/releases/v1-5-0)
    
*   [Dynamo v1.5.1 patch notes](https://github.com/ai-dynamo/dynamo/releases/tag/v1.5.1)
    
*   [Dynamo compatibility matrix](https://docs.nvidia.com/dynamo/reference/compatibility)
    
*   [Dynamo Planner overview](https://docs.nvidia.com/dynamo/latest/components/planner.md)
    
*   [Dynamo secure deployment guidelines](https://docs.nvidia.com/dynamo/dev/security/secure-deployment-guidelines)
    
*   [Dynamo GitHub releases](https://github.com/ai-dynamo/dynamo/releases)
    

**KServe and llm-d**

*   [KServe v0.21.0 release notes](https://github.com/kserve/kserve/releases/tag/v0.21.0)
    
*   [KServe `LLMInferenceService` configuration guide](https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-configuration)
    
*   [llm-d v0.10.0 release notes](https://github.com/llm-d/llm-d/releases/tag/v0.10.0)
    
*   [llm-d-router v0.11.0 release notes](https://github.com/llm-d/llm-d-router/releases/tag/v0.11.0)
    
*   [llm-d documentation](https://llm-d.ai/)
    

**Previous article**

*   [Running LLMs on Kubernetes in Production: KServe, vLLM, llm-d, Envoy AI Gateway, and LiteLLM](https://blog.shubhamtatvamasi.com/running-llms-on-kubernetes-in-production-kserve-vllm-llm-d-envoy-ai-gateway-and-litellm)