Train More with Less: Fewer GPUs, Less Data Movement, Better Cluster Utilization

PyTorch documentation advises against using CUDA Unified Virtual Memory (UVM) for deep learning because page faults, evictions, and repeated CPU–GPU transfers can make it significantly slower than explicit memory placement.

We show that these limitations are not inherent to UVM, but rather to how it is managed. With the right runtime policies, UVM can efficiently support LLM training on up to 50% fewer GPUs while maintaining comparable MFU.

To demonstrate this, we built TRANSIT (TRANsparent Scale-In for multi-node Training), a runtime that transparently integrates with existing PyTorch training stacks. TRANSIT minimizes CPU–GPU data movement through dynamic UVM policies, including prefetching and zero-copy, requiring no software changes.

Evaluated with TorchTitan on NVIDIA H100 cluster over a RoCE network, TRANSIT outperforms TorchTitan Offloading, ZeRO-Offload, and ZeRO-Infinity by up to 68%, 59%, and 42%, respectively. By reducing each job's GPU footprint, TRANSIT also lowers inter-node network traffic by up to 49% and reduces cluster queueing delays by up to 95%.