# General Scales Unlock AI Evaluation with Explanatory and Predictive Power

arXiv: https://arxiv.org/abs/2503.06378 · HTML: https://mlsanity.com/paper/2503.06378/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2025.

## Built on by (most cited first, 30 of 31)

- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning](https://mlsanity.com/paper/2507.00432/) (2025)
- [Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory](https://mlsanity.com/paper/2505.15055/) (2025)
- [AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games](https://mlsanity.com/paper/2602.17594/) (2026)
- [Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings](https://mlsanity.com/paper/2510.26384/) (2025)
- [Agent psychometrics: Task-level performance prediction in agentic coding benchmarks](https://mlsanity.com/paper/2604.00594/) (2026)
- [11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis](https://mlsanity.com/paper/2508.20068/) (2025)
- [ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models](https://mlsanity.com/paper/2601.15812/) (2026)
- [Capabilities Ain't All You Need: Measuring Propensities in AI](https://mlsanity.com/paper/2602.18182/) (2026)
- [Evaluating Language Models' Evaluations of Games](https://mlsanity.com/paper/2510.10930/) (2025)
- [Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights](https://mlsanity.com/paper/2507.06893/) (2025)
- [Measuring What AI Systems Might Do: Towards A Measurement Science in AI](https://mlsanity.com/paper/2603.00063/) (2026)
- [Interactive Evaluation Requires a Design Science](https://mlsanity.com/paper/2605.17829/) (2026)
- [Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systems](https://mlsanity.com/paper/2506.05370/) (2025)
- [Rethinking the Illusion of Thinking](https://mlsanity.com/paper/2507.01231/) (2025)
- [Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight](https://mlsanity.com/paper/2512.19691/) (2025)
- [Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models](https://mlsanity.com/paper/2603.07777/) (2026)
- [Scaling Data Difficulty: Improving Coding Models via Reinforcement Learning on Fresh and Challenging Problems](https://mlsanity.com/paper/2603.07779/) (2026)
- [Cost-Efficient Estimation of General Abilities Across Benchmarks](https://mlsanity.com/paper/2604.01418/) (2026)
- [Thomson: Continual Learning of Frontier Models for SovereignAI](https://mlsanity.com/paper/2608.27147/) (2026)
- [AutoScientist-Quant: Self-Evolving Coding Agents for Automatic Research in Quantitative Investment](https://mlsanity.com/paper/2608.28632/) (2026)
- [From Human-Level AI Tales to AI Leveling Human Scales](https://mlsanity.com/paper/2602.18911/) (2026)
- [Cross-LLM Consistency in Inference: Evidence from Shared Interactions](https://mlsanity.com/paper/2606.08129/) (2026)
- [Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs](https://mlsanity.com/paper/2607.20464/) (2026)
- [Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?](https://mlsanity.com/paper/2609.00482/) (2026)
- [Emergent evaluation hubs in a decentralizing large language model ecosystem](https://mlsanity.com/paper/2510.01286/) (2025)
- [An Interpretable and Scalable Framework for Evaluating Large Language Models](https://mlsanity.com/paper/2605.07046/) (2026)
- [Expectation Alignment of Language Models for Real-World User Expectations](https://mlsanity.com/paper/2607.20485/) (2026)
- [Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Models](https://mlsanity.com/paper/2608.01319/) (2026)
- [Using profiles of cognitive capability to assess AI suitability for workplace tasks](https://mlsanity.com/paper/2608.25623/) (2026)

## Abstract (arXiv)

Ensuring safe and effective use of AI requires understanding and anticipating its performance on novel tasks, from advanced scientific challenges to transformed workplace activities. So far, benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems, given the low transferability across diverse tasks. In this paper, we introduce general scales for AI evaluation that can explain what common AI benchmarks really measure, extract ability profiles of AI systems, and predict their performance for new task instances, in- and out-of-distribution. Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate. Illustrated for 15 large language models and 63 tasks, high explanatory power is unleashed from inspecting the demand and ability profiles, bringing insights on the sensitivity and specificity exhibited by different benchmarks, and how knowledge, metacognition and reasoning are affected by model size, chain-of-thought and distillation. Surprisingly, high predictive power at the instance level becomes possible using these demand levels, providing superior estimates over black-box baseline predictors based on embeddings or finetuning, especially in out-of-distribution settings (new tasks and new benchmarks). The scales, rubrics, battery, techniques and results presented here represent a major step for AI evaluation, underpinning the reliable deployment of AI in the years ahead. (Collaborative platform: https://kinds-of-intelligence-cfi.github.io/ADELE.)