mlsanityData to 14 Oct 2026

Compute Requirements for Algorithmic Innovation in Frontier AI Models

arXiv 2507.10618 ↗First posted 13 Jul 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Algorithmic innovation in the pretraining of large language models has driven a massive reduction in the total compute required to reach a given level of capability. In this paper we empirically investigate the compute requirements for developing algorithmic innovations. We catalog 36 pre-training algorithmic innovations used in Llama 3 and DeepSeek-V3. For each innovation we estimate both the total FLOP used in development and the FLOP/s of the hardware utilized. Innovations using significant resources double in their requirements each year. We then use this dataset to investigate the effect of compute caps on innovation. Our analysis suggests that compute caps alone are unlikely to dramatically slow AI algorithmic progress. Even stringent compute caps -- such as capping total operations to the compute used to train GPT-2 or capping hardware capacity to 8 H100 GPUs -- could still have allowed for half of the cataloged innovations.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 13 Jul 2025.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2020GLU Variants Improve Transformercited by 1,804
31 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  12. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  13. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  14. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  15. 20242 OLMo 2 Furiouscited by 340
  16. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  17. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  18. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  19. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  20. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  21. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  22. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  23. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  24. 2024Computing Power and the Governance of Artificial Intelligencecited by 40
  25. 2023AI capabilities can be significantly improved without expensive retrainingcited by 28
  26. 20228-bit Numerical Formats for Deep Neural Networkscited by 25
  27. 2024INTELLECT-1 Technical Reportcited by 21
  28. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 14
  29. 2024Governing Through the Cloud: The Intermediary Role of Compute Providers in AI Regulationcited by 10
  30. 2025Mechanisms to Verify International Agreements About AI Developmentcited by 9
  31. 2023Keep the Future Human: Why and How We Should Close the Gates to AGI and Superintelligence, and What We Should Build Insteadcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025On the Origin of Algorithmic Progress in AIcited by 6
  2. 2026Signals in the Noise: Open Source Intelligence (OSINT) for AI Loss of Control Detectioncited by 0