# The rising costs of training frontier AI models

arXiv: https://arxiv.org/abs/2405.21015 · HTML: https://mlsanity.com/paper/2405.21015/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2024.

## Built on by (most cited first, 30 of 95)

- [Quartet: Native FP4 Training Can Be Optimal for Large Language Models](https://mlsanity.com/paper/2505.14669/) (2025)
- [Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization](https://mlsanity.com/paper/2409.12903/) (2024)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [The Asymptotic Behavior of Attention in Transformers](https://mlsanity.com/paper/2412.02682/) (2024)
- [Position: The Most Expensive Part of an LLM should be its Training Data](https://mlsanity.com/paper/2504.12427/) (2025)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification](https://mlsanity.com/paper/2604.17112/) (2026)
- [When to Forget? Complexity Trade-offs in Machine Unlearning](https://mlsanity.com/paper/2502.17323/) (2025)
- [Safety Cases: A Scalable Approach to Frontier AI Safety](https://mlsanity.com/paper/2503.04744/) (2025)
- [RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models](https://mlsanity.com/paper/2603.14819/) (2026)
- [Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models](https://mlsanity.com/paper/2408.05241/) (2024)
- [BTS: Harmonizing Specialized Experts into a Generalist LLM](https://mlsanity.com/paper/2502.00075/) (2025)
- [Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation](https://mlsanity.com/paper/2504.01898/) (2025)
- [ZenFlow: Enabling Stall-Free Offloading for LLM Training](https://mlsanity.com/paper/2505.12242/) (2025)
- [Multilingual Test-Time Scaling via Initial Thought Transfer](https://mlsanity.com/paper/2505.15508/) (2025)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [Predicting LLM Reasoning Performance with Small Proxy Model](https://mlsanity.com/paper/2509.21013/) (2025)
- [CREDIT: Certified Ownership Verification of Deep Neural Networks Against Model Extraction Attacks](https://mlsanity.com/paper/2602.20419/) (2026)
- [Reasoning-Driven Synthetic Data Generation and Evaluation](https://mlsanity.com/paper/2603.29791/) (2026)
- [Function-Space Learning Rates](https://mlsanity.com/paper/2502.17405/) (2025)
- [Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems](https://mlsanity.com/paper/2503.01470/) (2025)
- [Trends in AI Supercomputers](https://mlsanity.com/paper/2504.16026/) (2025)
- [Sparse Checkpointing for Fast and Reliable MoE Training](https://mlsanity.com/paper/2412.15411/) (2024)
- [Practical Principles for AI Cost and Compute Accounting](https://mlsanity.com/paper/2502.15873/) (2025)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)
- [TrainVerify: Equivalence-Based Verification for Distributed LLM Training](https://mlsanity.com/paper/2506.15961/) (2025)
- [AI Should Sense Better, Not Just Scale Bigger: Adaptive Sensing as a Paradigm Shift](https://mlsanity.com/paper/2507.07820/) (2025)
- [Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector](https://mlsanity.com/paper/2509.07177/) (2025)
- [Generalist Foundation Models Are Not Clinical Enough for Hospital Operations](https://mlsanity.com/paper/2511.13703/) (2025)

## Abstract (arXiv)

The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.