# Inside NVIDIA Dynamo: A Deep Technical Tour of the Datacenter-Scale Inference Stack

Serving a model on one GPU is mostly a solved problem. You pick vLLM, SGLang, or TensorRT-LLM, start the server, and point a client at it. The hard part starts later: you have 64 GPUs, a model that doesn't fit on one node, traffic that triples every morning, and coding agents that send 40 requests per session, each one carrying the same 30K-token prefix. A fast engine on every GPU doesn't add up to a fast system.

[NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) is built for that gap. I spent time reading its source tree, design docs, and release notes to understand how it actually works, not just what the launch slides say. This post is the result: a component-by-component tour of the architecture, with the algorithms, data structures, and configuration that make it tick.

Everything here reflects the repository as of November 2026, where the latest release is **v1.6.0** (GA on October 22, 2026).

## What Dynamo is, and what it isn't

Dynamo is an **orchestration layer that sits above inference engines**. It doesn't replace vLLM, SGLang, or TensorRT-LLM. It takes many instances of them and turns the whole cluster into one coordinated serving system. The core is written in Rust for performance, with Python bindings for extensibility, and the project is Apache 2.0 licensed.

Some context on where it came from. NVIDIA announced Dynamo at GTC in March 2025 as the successor to Triton Inference Server. Version 1.0 went GA in March 2026. Since then the team has shipped a feature release roughly every five or six weeks. v1.6.0 alone merged 654 PRs from 148 contributors, and the repo sits at around 8.3k stars and 1.7k forks.

![The Dynamo stack: clients on top, Dynamo's orchestration layer in the middle, inference engines below, and GPU, network, and storage hardware at the bottom](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/ffd562b7-3c4d-4f50-8940-fb903907dcc2.png align="center")

The stack has four layers. Clients speak OpenAI, Anthropic, or KServe protocols to the **Frontend**. Dynamo's own components (Router, Planner, KV Block Manager, NIXL, and the Kubernetes Operator with Grove) decide where work runs and how state moves. Engines do the actual math on the GPU. Underneath is the hardware: HBM, NVLink, InfiniBand or RoCE, host DRAM, NVMe, and object storage. Several of these pieces (NIXL, KVBM, Grove, ModelExpress) are usable on their own, and NIXL in particular is already used inside vLLM, SGLang, TensorRT-LLM, and llm-d.

The README is honest about scope: if you run one model on one GPU, your engine alone is probably enough. Dynamo starts paying off when you coordinate many GPUs or nodes.

## Why a single engine stops scaling

To see why Dynamo is shaped the way it is, you need the two phases of LLM inference in your head.

![Prefill vs decode: prefill is compute-bound and drives TTFT, decode is memory-bandwidth-bound and drives ITL](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/b5ed475c-6941-4778-b1df-fbf380759f73.png align="center")

**Prefill** processes the entire prompt in parallel. It's compute-bound, scales with prompt length, and produces the KV cache: the keys and values for every token at every layer. It determines **time to first token (TTFT)**.

**Decode** generates one token per step per sequence. Each step reads the full KV cache and all the weights, so it's memory-bandwidth-bound. It determines **inter-token latency (ITL)**.

Put both in one engine and they interfere. A long prompt arriving mid-stream steals compute from every sequence that's decoding. Chunked prefill softens this but doesn't remove it. The Dynamo docs put the trade-off precisely: a larger prefill chunk causes fewer but longer interruptions to decode, and a smaller chunk causes more frequent, shorter ones.

The second problem is the KV cache itself. It's expensive to compute and it lives in one GPU's memory. If turn 2 of a conversation lands on a different worker than turn 1, the whole prefix gets recomputed. With N workers and naive load balancing, the odds of hitting the same worker are about 1/N.

Dynamo attacks both problems with four ideas: split prefill from decode, route requests to where their cache already lives, let the cache spill into cheaper memory tiers, and autoscale each pool against latency targets. The rest of this post is how each of those is built.

## The request path, end to end

Here's what happens to a single request in a disaggregated deployment.

![The nine-step Dynamo request flow from client through frontend, prefill router, prefill worker, NIXL transfer, decode worker, and back](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/a94a903b-1ed5-4029-897d-465a59d9819c.png align="center")

1. Request: The client sends an HTTP request to the Frontend (port 8000 by default).
2. Preprocess: The Frontend applies the chat template, tokenizes, and validates. It only needs the model's config and tokenizer files, not the weights.
3. Route to prefill: The `PrefillRouter` picks a prefill worker, using KV-aware scoring or plain load balancing.
4. Prefill: The prefill worker computes the KV cache for the prompt.
5. Return metadata: It returns `disaggregated_params`, which hold backend-specific transfer metadata.
6. Route to decode: The router injects that metadata into a decode request and picks a decode worker.
7. KV transfer: The decode worker pulls the KV blocks directly from the prefill worker's GPU memory through NIXL.
8. Decode: The decode worker generates tokens.
9. Response: Tokens stream back through the Frontend, which detokenizes and formats them for the client.

The Frontend is more than a proxy. It serves `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/responses`, image and video generation endpoints, an experimental Anthropic-style `/v1/messages`, and KServe gRPC v2, all on top of one internal representation. That's why one deployment can back several different agent harnesses at once.

## The distributed runtime: three planes

Underneath every component is the Rust `DistributedRuntime` (in `lib/runtime`). Python components use the same runtime through bindings in `lib/bindings/python`.

![DistributedRuntime hierarchy of namespace, component, and endpoint, with the discovery, request, and event planes](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/a50cd899-85dc-4307-bc7f-eefce5098270.png align="center")

The runtime organizes services in a four-level hierarchy:

- `DistributedRuntime` owns connections, background tasks, and cancellation.
- `Namespace` isolates one logical deployment or model group.
- `Component` groups workers that play the same role (for example, `prefill` or `decode`).
- `Endpoint` is a callable service such as `generate`, `clear_kv_blocks`, or `load_metrics`.

A client resolves a path like `namespace.component.endpoint`, watches membership changes, and dispatches with random, round-robin, or direct selection.

The interesting design decision is that discovery, requests, and events travel on **separate planes**, and each plane can use a different transport.

| Plane | What it carries | Default | Alternatives |
| --- | --- | --- | --- |
| Discovery | Who's alive, where endpoints live | Kubernetes CRs (`DynamoWorkerMetadata`) + `EndpointSlice` on K8s; etcd on bare metal | file and in-memory backends for dev |
| Request | RPC traffic between components | Direct pooled TCP, `msgpack` codec | NATS; `json` codec; opt-in batched QUIC response plane (v1.6) |
| Event | KV cache events, forward-pass metrics, telemetry | ZMQ, discovered through the discovery plane | NATS Core |

This split matters operationally. Early Dynamo releases required both etcd and NATS. Since v0.9, a Kubernetes deployment needs neither: discovery uses native Kubernetes resources and requests go over TCP. Each endpoint advertises its own codec, so one client can talk to endpoints using different codecs.

There's also a small but useful failure-handling detail. When a routed request fails, the local runtime temporarily **inhibits** that worker (5 seconds by default, set by `DYN_RUNTIME_INHIBITED_DURATION_SECS`) while discovery catches up. Discovery stays authoritative and can restore or remove the worker sooner.

## Disaggregated serving

Disaggregation means running prefill and decode on separate worker pools, each sized and parallelized for its own bottleneck. For example, you can give decode a larger tensor-parallel degree to get more aggregate memory for KV cache, while prefill runs at lower TP for better compute efficiency.

The `PrefillRouter` orchestrates the handoff.

![Sequence diagram of disaggregated serving: client, frontend, prefill router, prefill worker, and decode worker exchanging requests and a NIXL KV transfer](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/be49800a-efab-4b8e-a016-a57b58abbeeb.png align="center")

The engines don't all hand off the same way, and the difference affects latency:

| Backend | Transfer metadata | Behavior |
| --- | --- | --- |
| SGLang | `bootstrap_info` (host, port, room_id) | Prefill can run as a background task, so decode starts immediately while KV transfer proceeds in parallel |
| vLLM | `kv_transfer_params` (block IDs + remote worker info) | Prefill runs synchronously; decode waits for it |
| TensorRT-LLM | `opaque_state` (serialized TRT-LLM metadata) | Prefill runs synchronously; decode waits for it |

The KV transfer is non-blocking from the GPU's point of view: forward passes for other requests continue while blocks move. There's no shared storage in the hot path. Each worker keeps its own KV cache and blocks move worker to worker.

The topology is **runtime-reconfigurable xPyD** (x prefill workers, y decode workers). A new worker registers with discovery and publishes its `RuntimeConfig`, including KV capacity. A removed worker drains its in-flight requests and deregisters. The router picks up both changes automatically.

![Aggregated vs disaggregated timelines: in aggregated serving long prefills stall decode; in disaggregated serving each pool runs its own phase](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/ff2d91f5-eaee-4133-8b84-88414002be98.png align="center")

Disaggregation is not free, and the docs say so plainly. At low concurrency an aggregated worker usually wins because it avoids transfer overhead and pool fragmentation. If decode KV capacity is your bottleneck, moving GPUs into a prefill pool can reduce total capacity. And the network matters a lot: without RDMA, transfers fall back to TCP with severe latency penalties. The sensible approach is to measure, which is what the Planner and the profiler (both covered below) are for.

## KV-aware routing

The router is where a lot of Dynamo's engineering effort shows. Its job is to pick the worker that minimizes total work: reuse cached prefix where possible, but don't pile everything onto the one worker with the warmest cache.

### Block identity: two hashes per block

Engines store KV cache in fixed-size blocks (PagedAttention in vLLM, RadixAttention in SGLang). Dynamo identifies each block with two hashes.

![Block hashing: tokens split into fixed-size blocks, each gets a position-independent local hash and a rolling sequence hash chained from its predecessor](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/eec6fa62-5f01-42ab-b99e-0feb8fb1e9a0.png align="center")

- The **local block hash** is a content hash of the tokens inside one block. It's position-independent, so the frontend can hash all blocks of a query in parallel.
- The **sequence block hash** is a rolling hash of the entire prefix up to that block: `seq_hash[i] = hash(seq_hash[i-1] || local_hash[i])`. It's position-dependent.

When a LoRA adapter is active, its name is mixed into the hash, so blocks cached under different adapters never collide. Engines that use Python's built-in `hash()` for event IDs should set `PYTHONHASHSEED=0` to keep IDs deterministic across workers.

### The cost function

For each candidate worker the router computes a cost, then picks the lowest. Here's the formula from the router design doc:

```plaintext
effective_device_credit = overlap_score_credit * overlap_score_credit_decay_factor

adjusted_prefill_blocks = max(0,
      prefill_blocks
    - effective_device_credit * device_overlap_blocks
    - host_cache_hit_weight   * host_overlap_blocks
    - disk_cache_hit_weight   * disk_overlap_blocks
    - shared_cache_multiplier * (shared_beyond_blocks + fetchable_blocks))

active_request_blocks = decode_active_request_weight * active_requests

cost = prefill_load_scale * adjusted_prefill_blocks
     + potential_decode_blocks
     + active_request_blocks
```

The first term is "how much prompt would I actually have to compute here," discounted by cache hits in each tier (GPU, host memory, disk, shared storage, and blocks fetchable from a peer). The second term is decode load. The third, off by default, charges for batch size when decode latency depends on it.

![Worked example of the router cost function across three workers, with worker 2 selected at the lowest cost of 10](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/d848b8eb-8c48-4a2f-bd56-89aa8fec68b3.png align="center")

A worked example with `overlap_score_credit = 1.0`, a 10-block prompt, and the default `decode_active_request_weight = 0`:

| Worker | Cached overlap | Adjusted prefill | Decode blocks | Cost |
| --- | --- | --- | --- | --- |
| 1 | 2 | 8 | 10 | **18** |
| 2 | 5 | 5 | 5 | **10** (selected) |
| 3 | 8 | 2 | 9 | **11** |

Worker 3 has the most cache, but worker 2 wins because it's less loaded. That's the whole point: a high overlap credit favors cache reuse (better TTFT), and a low one favors even load (better ITL). If you set `router_temperature` above zero, the router samples from a softmax over normalized costs instead of always taking the minimum, which spreads load further.

### Keeping a global view of the cache: KV events

The router knows what's cached where because engines tell it. A publisher next to each engine emits two kinds of events: **stored** (a block was cached) and **removed** (a block was evicted). Explicit events are needed because engines keep blocks well past request lifetime, and their eviction policies are opaque. You can't infer cache state from request and response traffic alone.

![KV event flow: engines with local indexers publish stored and removed events over the event plane to router replicas, with gap detection and recovery](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/f0ea2550-c973-48a2-8626-76ef9a9de6e9.png align="center")

By default every worker also runs a **local indexer**, its own radix tree of what it holds. That's what makes the system recoverable:

1. Each worker numbers its events with monotonically increasing IDs starting at 0.
2. The router tracks the last ID it saw per worker.
3. If an event arrives with `event_id > last_id + 1`, the router has missed something.
4. It asks the worker for events starting at `last_id + 1`.
5. The worker replays from its buffer, or sends a full tree dump if the buffer no longer holds that ID.

When a new worker is discovered, the router ingests its full local index. When a worker disappears, its blocks are dropped from the global tree. If you'd rather not run events at all, `--no-router-kv-events` switches the router to an approximate mode that predicts cache state from its own routing decisions.

### Many routers, one picture of load

You usually run several router replicas. Each one knows only the requests it routed. With `--router-replica-sync`, replicas exchange three best-effort lifecycle events: `AddRequest` (request assigned to a worker), `MarkPrefillCompleted` (request moved from prefill to decode), and `Free` (request done). These improve each router's estimate of active load. They're deliberately not authoritative: publication is fire-and-forget, and a full outbound queue drops the newest event. Cache state still comes from workers.

### The Flash Indexer

At scale, the global index becomes a hot data structure: every request does a lookup, and every prefill sprays dozens of store events. Dynamo's team rewrote it six times, and the story is a nice case study in data structure design.

![Flash Indexer evolution: from a naive Python dictionary at 385K ops/s, to the v0.1.0 radix tree at 4M ops/s, to the positional jump-search indexer at 170M ops/s](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/9fe15dca-c29c-4dd3-9617-ae5dafc8e5e9.png align="center")

The progression went from a Python nested dictionary, to a single-threaded Rust actor, to an inverted index, to a radix tree, to a concurrent radix tree, and finally to a **positional indexer with jump search**. The final design throws away the tree and indexes by position:

```rust
enum SeqEntry {
    Single(ExternalHash, HashSet<WorkerWithDpRank>),
    Multi(HashMap<ExternalHash, HashSet<WorkerWithDpRank>>),
}

struct PositionalIndexer {
    // index[position] -> { local_hash -> SeqEntry }
    index: Vec<DashMap<LocalHash, SeqEntry>>,
    worker_blocks: DashMap<WorkerWithDpRank, LevelIndex>,
    jump_size: usize,
}
```

Because any position is O(1) to reach, the lookup can **jump** ahead by a stride (say 64 blocks), check how many workers still match at the checkpoint, and only scan backwards when some worker dropped out in the skipped range. The `Single`/`Multi` split means the expensive rolling sequence hash is only computed when two prefixes collide on the same chunk hash at the same position, which is rare. Measured on real traces, it sustains about **170M ops/s**, 42x the radix tree that shipped in v0.1.0 and 440x the naive version. It has been the default indexer since v1.0.

### Queueing and policy classes

When every worker is busy, dispatching anyway just builds queues inside engines where the router can't reorder them. Dynamo can hold requests in the router instead and release them as capacity frees up, so each routing decision uses the freshest load data.

In v1.6 this moved into a policy file passed with `--router-policy-config`:

```yaml
default_policy_class: standard
policy_classes:
  - name: priority
    quantum: 800
    slo_ttft_ms: 1000
    controlled_delay: true
  - name: standard
    quantum: 200
    slo_ttft_ms: 3000
```

Clients pick a class with the `x-dynamo-meta-policy-class` header. Classes share capacity through deficit round-robin (the `quantum` values). Within a class, the queue policy can be `edf` (earliest due time, where due time is router ingress plus `slo_ttft_ms`), `fcfs` (good for tail TTFT), or `wspt` (weighted shortest processing time, good for average TTFT, where cost is uncached prompt tokens). With `controlled_delay: true`, a request still waiting when its deadline passes is rejected instead of served late. v1.6 also added a request-classifier plugin hook that can defer, reorder, or steer requests.

### Two ways to put the router in front of traffic

You can let the Dynamo Frontend own the entry point (`client → Frontend → Router → workers`), or use the Kubernetes **Gateway API Inference Extension**, where the gateway calls Dynamo's Endpoint Picker Plugin (EPP) and forwards to a Frontend sidecar running in `--router-mode direct`. The second option fits platforms that already standardize auth, rate limits, and observability at the gateway. Since v1.6, both paths share one worker-selection core.

## KVBM: treating KV cache as a memory hierarchy

GPU memory is the most expensive place to keep KV cache, and it's small. The **KV Block Manager (KVBM)** lets blocks spill into cheaper tiers instead of being evicted and recomputed later.

![KVBM tiers G1 GPU HBM, G2 pinned host DRAM, G3 local NVMe, G4 object storage, with the typestate block lifecycle below](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/6a6dcbb3-0c3d-4543-a655-3c801872348e.png align="center")

The tiers are named G1 through G4:

| Tier | Storage | Enable with |
| --- | --- | --- |
| G1 | GPU HBM (the engine's own KV cache) | always on |
| G2 | Pinned host DRAM | `DYN_KVBM_CPU_CACHE_GB` |
| G3 | Local disk / NVMe | `DYN_KVBM_DISK_CACHE_GB` |
| G4 | S3-compatible object store | `DYN_KVBM_OBJECT_ENABLED=1` plus bucket settings |

A typical worker config looks like this:

```bash
export DYN_KVBM_CPU_CACHE_GB=200
export DYN_KVBM_DISK_CACHE_GB=1000
export DYN_KVBM_DISK_CACHE_DIR=/mnt/nvme/kvbm
```

One warning from the docs is worth repeating: make the CPU cache **larger** than the engine's GPU KV cache (at least 100 GB is the rule of thumb), and the disk cache at least as large as the CPU cache. If the host tier is smaller than the device tier, KVBM churns, offloading after every forward pass, and performance gets worse instead of better.

Inside, the Rust implementation is split into crates. `kvbm-logical` manages block lifecycle with a compile-time typestate machine:

```text
MutableBlock<T> → CompleteBlock<T> → ImmutableBlock<T> ⇄ WeakBlock<T>
   (reset)           (staged)          (registered)        (non-owning)
```

A `MutableBlock` is writable and returns to the reset pool when dropped. A `CompleteBlock` has a sequence hash but isn't registered yet. An `ImmutableBlock` is registered and can be shared; while a strong reference exists it can't be evicted, and dropping it moves it to an inactive pool where it stays matchable. A `WeakBlock` doesn't prevent eviction but can be upgraded back if the block survived. The type parameter `T` marks the tier, so the compiler stops you from mixing up a GPU block and a host block. The registry can track access frequency with TinyLFU, and eviction plugs in as a backend such as LRU.

`kvbm-physical` handles the bytes: memory layouts (fully contiguous or layer-separate), registration with NIXL, and a `TransferManager` that moves blocks between tiers or workers, with BLAKE3 checksums available for verifying transfers.

Blocks are deduplicated by sequence hash in a global registry. Once registered, a block is immutable and addressable by any worker that can reach its tier. This is what lets a subagent on worker B load a system prompt that worker A already computed, instead of recomputing it. Backend support varies, so check the feature matrix: vLLM uses KVBM through its KV connector interface, SGLang has its own HiCache, and the TensorRT-LLM KVBM connector was removed in a recent release.

## NIXL: the data plane

Everything that moves bytes between machines in Dynamo goes through **NIXL**, the NVIDIA Inference Xfer Library. It's a separate repo (`ai-dynamo/nixl`) and installs with `pip install nixl`.

![NIXL architecture: a transfer agent per process with memory sections, a backend interface (UCX, GPUDirect Storage, object storage), and a metadata handler exchanging descriptors with remote agents](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/d13c47bc-e7ee-49b7-8910-85c72002c896.png align="center")

Each inference process runs one **NIXL agent**, which abstracts three things:

- Memory sections, which are registered address ranges across DRAM, VRAM, NVMe-oF, files, and object storage, all described by the same buffer-list primitive.
- Transfer backends, such as UCX (NVLink, InfiniBand, RoCE, TCP), GPUDirect Storage, and object storage plugins. NIXL picks the best backend from the source and destination memory types and what both agents support. DRAM to VRAM might go over UCX; VRAM to a parallel filesystem might use GDS.
- A metadata handler that holds serialized connection info and remote segment descriptors, exchanged once over a side channel or through a metadata service like etcd or Redis, then cached.

The usage model is register-once, transfer-many:

```text
create_agent(name)
for backend in available_backends: create_transfer_backend(backend)
register_memory(kv_cache_regions)          # once, at init
md = get_local_metadata()                  # share with peers
load_remote_metadata(peer_md)              # cache peer descriptors
handle = post_transfer(src_descs, dst_descs, peer)   # async
check_status(handle)
```

Caching metadata is what makes elastic scaling cheap: adding a worker means handing its metadata to its peers, and removing one means invalidating their caches. In practice the knobs you touch are UCX settings. A disaggregated recipe in the repo sets things like `UCX_TLS=rc_x,rc,cuda_copy,cuda_ipc` and `UCX_RNDV_SCHEME=get_zcopy` for zero-copy RDMA reads.

## The Planner: autoscaling against SLAs

A fixed prefill-to-decode ratio is only right for one traffic shape. The **Planner** is Dynamo's autoscaler, and it scales prefill and decode pools independently to hit TTFT and ITL targets at minimum cost.

![Planner pipeline: OBSERVE, PREDICT, PROPOSE, RECONCILE and CONSTRAIN, EXECUTE, with a slow throughput loop setting a floor and a fast load loop reacting above it](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/f69f0a8a-c61d-423e-9489-0bfadfe3cdc9.png align="center")

It runs two control loops on different time horizons.

The **throughput-based loop** is slow and predictive. Each interval it pulls request count, average input and output lengths (ISL/OSL), KV hit rate, and speculative-decoding accept length from Prometheus. It forecasts the next interval with one of four predictors:

| Predictor | Algorithm | Best for |
| --- | --- | --- |
| Constant | next = current | Stable workloads |
| ARIMA | Auto-ARIMA, optional log1p transform | Trends and seasonality |
| Kalman | Local linear trend filter | Bursty traffic |
| Prophet | Facebook Prophet | Complex seasonality |

It then asks a performance model how much prefill and decode capacity is needed under the SLA. That model is bootstrapped from the best available source: worker self-benchmarks, AI Configurator interpolation, offline profiling data, or live regression. The result becomes a **replica floor**.

The **load-based loop** is fast and reactive. Every engine emits per-iteration `ForwardPassMetrics` (wall time, prefill tokens scheduled, decode KV tokens, queue depth). The Planner fits an online regression on those numbers to estimate TTFT and ITL right now, and scales up or down by one above the floor. This catches prediction errors and sudden bursts without waiting for the slow loop.

Internally both loops run as plugins in one pipeline: OBSERVE, PREDICT, PROPOSE, RECONCILE/CONSTRAIN (merge proposals, enforce minimums and GPU budgets), then EXECUTE. External plugins can join over gRPC. Execution goes through a connector. The `KubernetesConnector` patches replica counts on the DynamoGraphDeployment and lets the operator reconcile. The `VirtualConnector` writes decisions into the runtime so a non-Kubernetes orchestrator can poll and acknowledge them.

Public numbers: Alibaba reported 80% fewer SLA breaches at 5% lower TCO using Planner autoscaling. v1.6 added runtime changes to GPU budgets through `/v1/min-endpoints`, and an experimental Planner Gym for comparing Planner, KEDA-style, reactive, and static policies against replayed traffic.

## Kubernetes: DGD, DGDR, the Operator, and Grove

On Kubernetes you describe deployments with custom resources and the **Dynamo Operator** reconciles them.

A **DynamoGraphDeployment (DGD)** describes one inference graph: a Frontend plus workers. Here's a disaggregated vLLM deployment using the `v1beta1` API:

```yaml
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: qwen3-32b-disagg
spec:
  components:
  - name: Frontend
    type: frontend
    replicas: 1
    podTemplate:
      spec:
        containers:
        - name: main
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0
  - name: prefill
    type: prefill
    replicas: 2
    sharedMemorySize: 16Gi
    podTemplate:
      spec:
        containers:
        - name: main
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0
          envFrom:
          - secretRef: { name: hf-token-secret }
          command: [python3, -m, dynamo.vllm]
          args:
          - --model
          - Qwen/Qwen3-32B
          - --disaggregation-mode
          - prefill
          - --kv-transfer-config
          - '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
  - name: decode
    type: decode
    replicas: 1
    sharedMemorySize: 16Gi
    podTemplate:
      spec:
        containers:
        - name: main
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0
          envFrom:
          - secretRef: { name: hf-token-secret }
          command: [python3, -m, dynamo.vllm]
          args:
          - --model
          - Qwen/Qwen3-32B
          - --disaggregation-mode
          - decode
          - --kv-transfer-config
          - '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
```

Getting parallelism and replica counts right by hand is tedious, so there's a second resource. A **DynamoGraphDeploymentRequest (DGDR)** is deploy-by-intent: you state the model and your SLA, and Dynamo works out the rest.

```yaml
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid    # rapid (~30s, simulated) | thorough (2-4h, real GPUs)
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true
```

![DGDR lifecycle: the request goes through profiling with AISimulate or real GPUs, generates a DGD, which the operator turns into Grove PodCliques gang-scheduled by KAI Scheduler, with the Planner closing the loop](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/468d2da7-057f-4046-8412-e9fa260c913c.png align="center")

A DGDR moves through `Pending → Profiling → Ready → Deploying → Deployed`. The operator auto-detects GPU SKU, VRAM, and count. With `searchStrategy: rapid`, the profiler uses AISimulate performance models and finishes in about 30 seconds without touching a GPU. With `thorough`, it benchmarks candidate configurations on real hardware. Either way it emits a DGD, plus a Planner configuration if you enable autoscaling.

Multi-node workers need gang scheduling. A TP=16 worker spread across two nodes is useless if only half its pods get scheduled. **Grove** is the default orchestrator for this, and it pairs with the **KAI Scheduler** for actual gang semantics, because the standard Kubernetes scheduler doesn't provide them. Grove understands topology (racks, hosts, NUMA nodes, NVL72 domains) so it can place tightly coupled pods close together. LeaderWorkerSet with Volcano is the supported alternative.

## Fault tolerance: workers fail, requests don't

GPUs fail, nodes get preempted, and pods get evicted. Dynamo's **request migration** keeps a client's stream alive through that.

![Request migration: the migrator accumulates generated tokens, and when worker A fails mid-stream the request is reissued to worker B with prompt plus generated tokens, so the client sees one continuous stream](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/96c54eca-8200-4379-91a5-0386312e98e4.png align="center")

A Migration operator sits in the pipeline between frontend preprocessing and the backend. As tokens stream back, it appends each one to the request's token sequence and decrements the remaining token budget. If the worker becomes unreachable before the stream starts, the request is simply retried elsewhere. If the stream breaks mid-generation with a migratable error, a new stream is created on a healthy worker with the **prompt plus every token generated so far**. The new worker prefills that context and continues from the exact point of failure. No tokens are lost or duplicated, and the client sees one continuous stream. The migration limit is set on the Frontend. Combined with KV-aware routing, the new worker may already hold much of that prefix in cache, which keeps the recovery prefill cheap.

Around this sit canary health checks, request cancellation that propagates to engines, graceful shutdown that drains in-flight work, and admission control. v1.6 also added **Dynamo Snapshot**\-backed active/standby engine failover for SGLang and vLLM.

## Built for agents and multimodal

Agentic traffic is shaped differently from chat, and Dynamo's recent releases lean into that. NVIDIA's analysis of Claude Code sessions found an 11.7x cache read-to-write ratio: the system prompt and growing conversation are computed once and then read on every call. Keeping those blocks warm and routable is most of the game.

![Agentic optimizations: harness sends nvext agent hints, the router uses priority and expected output length, and shared KV storage lets subagents load a computed prefix instead of recomputing it](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/66521c31-9c32-4191-bf80-bf1f3a085aca.png align="center")

The harness can tell Dynamo what it knows through **agent hints**, sent in the `nvext` field on any of the three chat APIs:

```json
{
  "model": "MiniMaxAI/MiniMax-M2.5",
  "messages": [...],
  "nvext": {
    "agent_hints": {
      "priority": 10,
      "osl": 256,
      "speculative_prefill": true
    }
  }
}
```

`priority` affects both router queue order and engine-level scheduling (Dynamo normalizes each engine's own priority polarity, and SGLang can use it for priority-aware cache eviction). `osl` is the harness's estimate of output length, which helps the router predict how long a worker stays busy. `speculative_prefill` asks Dynamo to warm a likely worker's cache before the full request is ready, which is handy when a tool call is about to return.

The router's Python bindings (`best_worker()`, `get_potential_loads()`, `generate()`) let teams write custom strategies. The NeMo Agent Toolkit team built a Thompson-sampling router on them and measured about 4x lower p50 TTFT and 1.5x higher p50 tokens per second compared with default routing.

Multimodal requests get a third disaggregated stage: **encode / prefill / decode (EPD)**. Vision encoding for multi-image or video requests can take hundreds of milliseconds and blocks text-only requests sharing the same worker. Dynamo can run encoder workers separately (colocated on the same GPU, or on a cheaper GPU tier) and ship embeddings to the PD workers through NIXL, with an embedding cache to skip repeated work. The project reports around 30% faster TTFT on image-heavy workloads. Video generation is also supported natively through FastVideo and SGLang Diffusion.

## Cold starts: ModelExpress and Snapshot

Autoscaling is only as good as your scale-up time. Loading a 600B-parameter model from object storage can take many minutes, which is longer than most traffic spikes.

![ModelExpress loading strategy chain: serving peer GPU over NIXL RDMA, server cache, local safetensors, ModelStreamer, GPUDirect Storage, and the engine's default loader](https://cdn.hashnode.com/uploads/covers/63cc2392587fc59d3cd765f9/3b16aa96-f04a-4708-9aba-574fa967c990.png align="center")

**ModelExpress** asks a simple question before loading: where does a compatible copy of these weights already live? It tries a fixed chain of strategies and runs the first one that applies:

1. **Serving peer to GPU** over NIXL RDMA: copy post-processed weights from a replica that's already running.
2. **Server cache to runtime** when there's no shared storage.
3. **Local safetensors to GPU** with InstantTensor.
4. **Remote or local storage to GPU** with ModelStreamer, overlapping reads with GPU placement.
5. **GPUDirect Storage** from local disk.
6. **The engine's default loader** as a last resort.

If a strategy fails after weights may have landed, ModelExpress reinitializes the model before trying the next one, so a partially loaded model is never served. Each new replica joins the source pool, so scale-out turns into GPU-to-GPU fan-out. It can also copy JIT caches (TorchInductor, Triton, DeepGEMM, FlashInfer, and others) so new replicas skip recompiling kernels. The ModelExpress README reports DeepSeek-V4-Pro weights loading from a serving replica in 11 seconds on 8xB200, a 48x speedup over a cold Hugging Face pull, and process-start-to-API-ready dropping from about 8 minutes to 1 minute 44 seconds.

**Dynamo Snapshot** goes further and checkpoints the whole engine process, combining `cuda-checkpoint` and CRIU so a replica restores with weights loaded and kernels warm.

## Hands-on: run it yourself

You don't need a cluster to get a feel for it. The prebuilt container ships the Frontend, router, and engine. With one GPU:

```bash
docker run --gpus all --network host --rm -it \
  nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0

# inside the container
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file &
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file \
  --kv-events-config '{"enable_kv_cache_events": false}' &

curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Explain KV cache in one sentence."}],
  "max_tokens": 100
}' | jq
```

`--discovery-backend file` replaces etcd for local use. With two GPUs you can try disaggregation and KV-aware routing (in approximate mode, since events are off):

```bash
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file \
  --router-mode kv --no-router-kv-events &

CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B \
  --discovery-backend file --disaggregation-mode decode \
  --kv-events-config '{"enable_kv_cache_events": false}' \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' &

VLLM_NIXL_SIDE_CHANNEL_PORT=20098 CUDA_VISIBLE_DEVICES=1 \
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B \
  --discovery-backend file --disaggregation-mode prefill \
  --kv-events-config '{"enable_kv_cache_events": false}' \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' &
```

The repo has ready-made launch scripts for many more combinations under `examples/backends/<engine>/launch/`, and full Kubernetes recipes for models like DeepSeek-R1, Qwen3, and Kimi under `recipes/`. For benchmarking, the project uses AIPerf. If you work with a coding agent, the repo also ships agent skills for deploying, debugging, and benchmarking Dynamo.

## When should you reach for Dynamo?

It earns its complexity when:

- You serve across **multiple GPUs or nodes** and need them to behave as one system.
- Your traffic has **shared prefixes** (multi-turn chat, RAG with common context, agents), where KV-aware routing cuts redundant prefill.
- Prefill and decode have **different bottlenecks** in your workload and you want to scale them separately.
- You have **latency SLAs** and want autoscaling that targets them directly instead of GPU utilization.
- Cold starts are hurting your ability to scale with demand.

The trade-offs are real, though:

- **Networking.** Disaggregation without RDMA (InfiniBand, RoCE, or EFA) falls back to TCP and loses most of its benefit.
- **Operational surface.** Operator, CRDs, Grove, KAI Scheduler, Prometheus, and possibly KVBM storage tiers are a lot of moving parts compared with one engine behind a load balancer.
- **Pace of change.** The release cadence is fast. v1.6.0 lists 41 breaking changes and several deprecated router flags scheduled for removal in v1.7. Pin versions and read the deprecation ledger before upgrading.
- **Workload fit.** Disaggregation can lose to aggregated serving at low concurrency. Profile before committing to a topology.

The headline numbers in the README give a sense of the upside when the fit is right: up to 7x higher throughput per GPU for DeepSeek-R1 on GB200 NVL72 with Dynamo versus B200 without it (SemiAnalysis InferenceX), and 2x faster TTFT from KV-aware routing in Baseten's Qwen3-Coder 480B deployment.

What I find most interesting about Dynamo is less any single feature and more the stance behind it: inference at scale is a distributed-systems problem, where KV cache is state to be indexed, moved, tiered, and scheduled around. Once you look at it that way, routing, disaggregation, tiered caching, and SLA-driven scaling stop being separate tricks and become parts of one design.

## References

- Dynamo repository: [github.com/ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo)
- Documentation: [docs.nvidia.com/dynamo](https://docs.nvidia.com/dynamo/)
- NIXL: [github.com/ai-dynamo/nixl](https://github.com/ai-dynamo/nixl)
- ModelExpress: [github.com/ai-dynamo/modelexpress](https://github.com/ai-dynamo/modelexpress)
- Grove: [github.com/ai-dynamo/grove](https://github.com/ai-dynamo/grove)
- Design docs in the repo: `docs/fern/pages/developer-guide/knowledge-base/` (architecture, router design, planner design, request migration)
- Engineering posts in the repo: Flash Indexer, agentic inference optimizations, EPD disaggregation (`docs/fern/pages/blog/2026/`)
- v1.6.0 release notes: `docs/fern/pages/reference/general/releases/dynamo-v1-6-0.mdx`
