mlsanityData to 7 Oct 2026

Data movement limits to frontier model training

arXiv 2411.01137 ↗First posted 2 Nov 2024cs.DC cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We present a theoretical model of distributed training, and use it to analyze how far dense and sparse training runs can be scaled. Under our baseline assumptions, given a three month training duration, data movement bottlenecks begin to significantly lower hardware utilization for training runs exceeding about 1028 FLOP, two orders of magnitude above the largest training run to date, suggesting the arrival of fundamental barriers to scaling in three years given recent rates of growth. A training run exceeding about 1031 FLOP is infeasible even at low utilization. However, more aggressive batch size scaling and/or shorter and fatter model shapes, if achievable, have the potential to permit much larger training runs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

27 of 41 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Layer Normalizationcited by 5,579
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
15 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  6. 2023The Falcon Series of Open Language Modelscited by 526
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  8. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  9. 2018An Empirical Model of Large-Batch Trainingcited by 268
  10. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  11. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  12. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  13. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  14. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  15. 2023Zero Bubble Pipeline Parallelismcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 5
  2. 2025Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directionscited by 2
  3. 2026Throughput Optimization as a Strategic Lever in Large-Scale AI Systems: Evidence from Dataloader and Memory Profiling Innovationscited by 0