ML admins will be happy to learn that one bad GPU may no longer mean starting an entire training job from scratch. The latest 2. 15 release of PyTorch includes a rewritten backend for Nvidia’s Collective Communications Library (NCCL), designed to improve fault diagnosis and recovery when GPUs work together. This latest feature shows how its creators continue to advance the software from being a tool for researchers to being a cornerstone in production AI work. The interface to the networking library in PyTorch needed a rewrite – it was drowning in technical debt, said PyTorch core developer Tristan Rice , a Meta software engineer specializing in distributed computing, during a live YouTube livestream on Thursday. NCCL is pronounced "Nickel" and stands for the Nvidia Collective Communications Library. It provides PyTorch with a set of inter-GPU communication primitives, allowing GPUs to communicate within a single-node or across multi-node clusters, which is vital for running inference on large neural networks and for other tasks in building and maintaining LLMs. The old NCCL interface was too chaotic The previous PyTorch backend for NCCL, called ProcessGroupNCCL , could spawn dozens of communicators per process group, leading to intermittent deadlocks and unpredictable performance. The new version of the NCCL backend comes with a strict rule: One process group gets only one communicator during initialization. By default, this rule eliminates crosstalk between multiple communicators and all the complexities that entails. If a program requires parallelism, it will have to dispatch more process groups. The updated library also better handles stragglers by allowing for easier reconfiguration. If a GPU crashes or runs slowly, a third-party orchestrator can exclude the laggard and even assign a replacement worker. A “flight recorder” has also been added to give admins a better idea of what went wrong…
