I've Got 99 Problems But FLOPS Ain't One
Not on the map. Why?
Abstract (arXiv)
Hyperscalers dominate the landscape of large network deployments, yet they rarely share data or insights about the challenges they face. In light of this supremacy, what problems can we find to solve in this space? We take an unconventional approach to find relevant research directions, starting from public plans to build a 100 billion datacenter for machine learning applications. Leveraging the language models scaling laws, we discover what workloads such a datacenter might carry and explore the challenges one may encounter in doing so, with a focus on networking research. We conclude that building the datacenter and training such models is technically possible, but this requires novel wide-area transports for inter-DC communication, a multipath transport and novel datacenter topologies for intra-datacenter communication, high speed scale-up networks and transports, outlining a rich research agenda for the networking community.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 73 references that are in the map. First posted on arXiv 1 Jul 2024.
Builds on
30 of 73 references.
References · 30
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
18 more
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
- 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
- 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
- 2018Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities (Complete Version)cited by 9
Built on by
Citing papers per year: 2026: 3.
3 papers.