Enabling APC for hybrid models in vLLM

Hybrid models that go beyond conventional full attention — Nemotron 3 Ultra, Qwen3.6, and others — are rapidly becoming mainstream. At the same time, reasoning and agentic workloads are making Automatic Prefix Caching (APC) increasingly important: long shared prefixes create substantial opportunities for cache reuse, and APC can dramatically reduce Time-To-First-Token (TTFT). In this talk, we present our contributions along vLLM's journey to making APC practical for hybrid models. Supporting APC for hybrid models is not straightforward. Checkpointing their state can drive high memory usage or make the cache too sparse to hit reliably. Worse, early hybrid APC support introduced decode-path overhead, causing performance regressions when cache hit rates were low. Our work improved cache utilization while keeping memory usage under control, and with a specialized Triton kernel we removed ~20% decode overhead by streamlining CPU–GPU interactions. The resulting APC implementation for hybrid models delivers >10x TTFT improvement on long-prefix cache hits, introduces no overhead in low-hit-rate scenarios, and remains compatible with advanced serving features such as Multi-Token Prediction.