Notes on pretraining parallelisms and failed training runs.
Deeply researched interviews
TL;DR
- Pretraining runs fail due to breaking causality (e.g., expert routing, token dropping) and introducing bias (e.g., FP16 numerical precision bugs).
- Expert routing in training can break causality by allocating tokens based on future dependencies, unlike inference.
- FP16's limited granularity can cause significant numerical errors when summing many small values, as seen in an early GPT-4 training issue.
- Fully Sharded Data Parallel (FSDP) is a default parallelism strategy, offering a good balance of communication and computation, but can be limited by communication crossover points.
- Pipeline Parallelism is introduced when FSDP scales poorly, but it suffers from bubbles (underutilized GPUs) and architectural constraints that hinder research.
- Optimizing communication in large-scale training involves techniques like hierarchical collectives to minimize bottlenecks.
- The complexity of AI training suggests that new issues will continue to emerge at larger scales, making fully automated kernel writing a distant goal.