Decoupled DiLoCo: Resilient, Distributed AI Training at Scale
Google’s new distributed architecture keeps AI training runs on track across distant data centers, with exceptional efficiency – even when hardware fails.
TL;DR
- Decoupled DiLoCo is a new distributed architecture for training large AI models across distant data centers.
- It uses decoupled compute "islands" and asynchronous data flow to isolate local disruptions and maintain training progress.
- The system is resilient to hardware failures and can seamlessly reintegrate failed compute units.
- Decoupled DiLoCo requires significantly less bandwidth and avoids communication delays common in previous distributed methods.
- It enables training of large models (e.g., 12 billion parameters) across multiple regions with achievable network speeds.
- The architecture supports training with mixed hardware generations (e.g., TPU v6e and TPU v5p) in a single run.
- Testing showed Decoupled DiLoCo maintains higher cluster availability and matches ML performance of traditional methods, even with hardware failures.