Offloading KV Cache to Secondary Memory: A Three-Tier Hierarchy Emulator using vLLM

A new class of memory hardware is emerging between HBM and host DRAM: CXL-attached memory pools, NVLink-reachable DDR, and other near-accelerator fabrics that are larger than HBM and faster than host DRAM. These "secondary fast memory" systems are emerging across the industry as spill targets for hot KV blocks that no longer fit in HBM. However, this hardware is not yet broadly available to the inference community. This talk shows how we extend vLLM's SimpleCPUOffloadConnector to emulate a three-level hierarchy (HBM ↔ secondary fast memory ↔ slow host DRAM) using two pinned-host CPU pools, so the community can iterate on placement, metadata, and worker plumbing today and swap in real hardware later with no scheduler changes.

We walk through: a CPU-Tier abstraction that keeps the scheduler symmetric across tiers; a partitioned placement model that admits requests by priority; per-tier copy backends, events, and prefix-cache lookup; and backward compatibility with the existing single-pool config. You leave with the design pattern, connector hooks, and a working emulator to evaluate placement policies for KV-cache workloads — ready for secondary-memory hardware as it ships.