Real Shapes, Real Bugs: Model-Derived Operator Tests for Quick Out-of-Tree Accelerator Bring-Up

Out-of-tree backends must keep pace with PyTorch's bi-monthly releases, and the ecosystem responds with cross-repo CI relays and device-agnostic test instantiation. These fix when and where tests run. We address a different gap: what they run on. Op-level tests built from synthetic inputs pass while real models fail. We present an open-source, hardware-agnostic framework that derives operator tests from real models.

Takeaways:

• Real models emit dtype combinations, broadcast patterns, and non-contiguous strides that random generators rarely produce - the cases break a fresh backend
• The gap is widening: accelerator-optimized dtypes surface silent dtype promotions, MoE routing produces unique shapes, and agent-generated kernels outpace hand-written coverage
• A TorchDynamo tracer emits per-op YAML with observed shapes, dtypes, strides, and offsets
• A pytest runner on PyTorch's test ecosystem with the YAML and validates each op against a reference
• The YAML decouples the two, so a corpus captures once run on any target
• Coverage and defect detection across 116 operators from 8 models, plus a taxonomy of bring-up traps

We also chat lessons learned for bringing up accelerators.