# Two AI Metrics Diverged: Will it Make All the Difference?

arXiv: https://arxiv.org/abs/2607.00913 · HTML: https://mlsanity.com/paper/2607.00913/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 28 of 28)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [Scaling Scaling Laws with Board Games](https://mlsanity.com/paper/2104.03113/) (2021)
- [How Do Large Language Monkeys Get Their Power (Laws)?](https://mlsanity.com/paper/2502.17578/) (2025)
- [Scaling Laws for a Multi-Agent Reinforcement Learning Model](https://mlsanity.com/paper/2210.00849/) (2022)
- [How predictable is language model benchmark performance?](https://mlsanity.com/paper/2401.04757/) (2024)
- [A Rosetta Stone for AI Benchmarks](https://mlsanity.com/paper/2512.00193/) (2025)
- [EvilGenie: A Reward Hacking Benchmark](https://mlsanity.com/paper/2511.21654/) (2025)
- [Scaling Laws for Deep Learning](https://mlsanity.com/paper/2108.07686/) (2021)
- [On the Origin of Algorithmic Progress in AI](https://mlsanity.com/paper/2511.21622/) (2025)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [The Importance of (Exponentially More) Computing Power](https://mlsanity.com/paper/2206.14007/) (2022)
- [Forecasting AI Time Horizon Under Compute Slowdowns](https://mlsanity.com/paper/2511.19492/) (2025)
- [Is there "Secret Sauce'' in Large Language Model Development?](https://mlsanity.com/paper/2602.07238/) (2026)
- [Meek Models Shall Inherit the Earth](https://mlsanity.com/paper/2507.07931/) (2025)
- [How Much Progress Has There Been in NVIDIA Datacenter GPUs?](https://mlsanity.com/paper/2601.20115/) (2026)

## Abstract (arXiv)

As exponential compute scaling continues, will the capabilities of frontier AI models outstrip what is accessible to developers on a small fixed budget? Or will capabilities converge, with "meek models inheriting the earth"? Building on Gundlach et al. (2025b), we show that the answer depends on how we value and measure AI capabilities. We discuss conventional performance measures and show that, while validation loss shows a shrinking gap, on other metrics frontier models grow their lead forever. Classifying performance metrics by their functional forms in relation to training (and inference) compute, we provide tight mathematical conditions for determining which metrics favor meek models, and show that bounded performance metrics always do. But careful interpretation of performance metrics is essential: we show that many common bounded metrics have closely-related counterpart metrics that are unbounded (and vice versa). Determining the apt metric in a domain is a prerequisite for policy, since bounded and unbounded metrics may suggest opposing policy responses. If a particular capability -- like software engineering, synthetic biology, or rhetorical persuasiveness -- is unbounded when measured in the terms we care about, frontier-level capability will likely be concentrated in the hands of a few wealthy actors. Conversely, if that capability is instead bounded, frontier-level capabilities proliferate through meek models into the hands of the many.