mlsanityData to 7 Oct 2026

The rising costs of training frontier AI models

arXiv 2405.21015 ↗First posted 31 May 2024cs.CYcited by 91

Not on the map. Why?

Abstract (arXiv)

The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 May 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 41; 2026: 44.

91 papers.

Sort:

By area:

  1. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  2. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  3. 2025Virtual Agent Economiescited by 17
  4. 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 15
  5. 2024The Asymptotic Behavior of Attention in Transformerscited by 12
  6. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  7. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  8. 2025When to Forget? Complexity Trade-offs in Machine Unlearningcited by 9
  9. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  10. 2025Safety Cases: A Scalable Approach to Frontier AI Safetycited by 8
  11. 2024Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Modelscited by 7
  12. 2025BTS: Harmonizing Specialized Experts into a Generalist LLMcited by 6
38 more
  1. 2025Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillationcited by 5
  2. 2025ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updatescited by 5
  3. 2025Multilingual Test-Time Scaling via Initial Thought Transfercited by 5
  4. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  5. 2025Predicting LLM Reasoning Performance with Small Proxy Modelcited by 5
  6. 2025Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systemscited by 4
  7. 2025Trends in AI Supercomputerscited by 4
  8. 2026CREDIT: Certified Ownership Verification of Deep Neural Networks Against Model Extraction Attackscited by 4
  9. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 4
  10. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  11. 2025Fair Foundation Models for Medical Image Analysis: Challenges and Perspectivescited by 3
  12. 2025Function-Space Learning Ratescited by 3
  13. 2025TrainVerify: Equivalence-Based Verification for Distributed LLM Trainingcited by 3
  14. 2025AI Should Sense Better, Not Just Scale Bigger: Adaptive Sensing as a Paradigm Shiftcited by 3
  15. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  16. 2025Generalist Foundation Models Are Not Clinical Enough for Hospital Operationscited by 3
  17. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision Peoplecited by 3
  18. 2026RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Modelscited by 3
  19. 2024A Comprehensive Study of Shapley Value in Data Analyticscited by 2
  20. 2025Practical Principles for AI Cost and Compute Accountingcited by 2
  21. 2025Green MLOps to Green GenOps: An Empirical Study of Energy Consumption in Discriminative and Generative AI Operationscited by 2
  22. 2025An entropy-optimal path to humble AIcited by 2
  23. 2025Hashed Watermark as a Filter: Defeating Forging and Overwriting Attacks in Weight-based Neural Network Watermarkingcited by 2
  24. 2026Modern Neuromorphic AI: From Intra-Token to Inter-Token Processingcited by 2
  25. 2026M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Datacited by 2
  26. 2026AI-Supervisor: Autonomous AI Research Supervision via a Persistent Research World Modelcited by 2
  27. 2026The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Productscited by 2
  28. 2026Zero knowledge verification for frontier AI training is possiblecited by 2
  29. 2024Efficient Semantic Diffusion Architectures for Model Training on Synthetic Echocardiogramscited by 1
  30. 2025Beyond Release: Access Considerations for Generative AI Systemscited by 1
  31. 2025KAIROS: Scalable Model-Agnostic Data Valuationcited by 1
  32. 2025Are neural scaling laws leading quantum chemistry astray?cited by 1
  33. 2025The Role of Computing Resources in Publishing Foundation Model Researchcited by 1
  34. 2026Capability-Aware Early-Stage Research Idea Evaluationcited by 1
  35. 2026Buy versus Build an LLM: A Decision Framework for Governmentscited by 1
  36. 2026Evolutionary Context Search for Automated Skill Acquisitioncited by 1
  37. 2026FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolationcited by 1
  38. 2026Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systemscited by 1

50 most cited of 91 All 91 (JSON)