The PyTorch Foundation released PyTorch 2.14. The release adds NVGEMM, a backend that picks the fastest kernel for matrix maths. It also lands nccl2, a rewritten backend for training across many machines. Fault tolerance moves into the core: a training group can be rebuilt in place after a node fails. Apple Silicon Macs gain native linear algebra routines and more hand-written Metal kernels. New control-flow operations torch.switch and torch.while_loop help models with branching logic compile. A @dynamic_spec decorator declares which tensor dimensions may change at runtime. Compiled code now overlaps communication with computation by default. Support for complex-valued tensors in torch.compile is experimental and incomplete. Binary support arrives for Python 3.15, including the no-GIL build, but torch.compile raises an error there.
What changed
PyTorch 2.13 added torchcomms and a CuTeDSL code path, but no in-tree nccl2 backend, no Python 3.15 wheels, and no multi-way branching op.
What it unlocks
Rebuilding a distributed training group in place after a node fails, instead of restarting the job.
- 2,995 commits from 487 contributors
- ROCm 7.14 wheels
- 8.5x slowdown fixed on MPS decode
What you need to act on it
- install from download.pytorch.org for Python 3.15 wheels
- Python 3.14 or earlier to use torch.compile
Sources