# Revealing the structure of language model capabilities

arXiv: https://arxiv.org/abs/2306.10062 · HTML: https://mlsanity.com/paper/2306.10062/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)

## Built on by (most cited first, 30 of 40)

- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Dynamic Evaluation of Large Language Models by Meta Probing Agents](https://mlsanity.com/paper/2402.14865/) (2024)
- [LLM-as-a-Judge & Reward Model: What They Can and Cannot Do](https://mlsanity.com/paper/2409.11239/) (2024)
- [100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances](https://mlsanity.com/paper/2409.03563/) (2024)
- [Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families](https://mlsanity.com/paper/2412.06540/) (2024)
- [Evaluating General-Purpose AI with Psychometrics](https://mlsanity.com/paper/2310.16379/) (2023)
- [Evaluating Large Language Models with Psychometrics](https://mlsanity.com/paper/2406.17675/) (2024)
- [Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches](https://mlsanity.com/paper/2404.12744/) (2024)
- [Train-before-Test Harmonizes Language Model Rankings](https://mlsanity.com/paper/2507.05195/) (2025)
- [Language Models as a Service: Overview of a New Paradigm and its Challenges](https://mlsanity.com/paper/2309.16573/) (2023)
- [What do Language Models Learn and When? The Implicit Curriculum Hypothesis](https://mlsanity.com/paper/2604.08510/) (2026)
- [From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation](https://mlsanity.com/paper/2507.20208/) (2025)
- [When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives](https://mlsanity.com/paper/2406.12084/) (2024)
- [Capabilities Ain't All You Need: Measuring Propensities in AI](https://mlsanity.com/paper/2602.18182/) (2026)
- [Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach](https://mlsanity.com/paper/2403.15250/) (2024)
- [A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities](https://mlsanity.com/paper/2603.02540/) (2026)
- [Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation](https://mlsanity.com/paper/2311.09136/) (2023)
- [Psychometric Personality Shaping Modulates Capabilities and Safety in Language Models](https://mlsanity.com/paper/2509.16332/) (2025)
- [BenchScope: How Many Independent Signals Does Your Benchmark Provide?](https://mlsanity.com/paper/2603.29357/) (2026)
- [AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems](https://mlsanity.com/paper/2605.25272/) (2026)
- [You Don't Need to Run Every Eval](https://mlsanity.com/paper/2606.24020/) (2026)
- [RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instruction](https://mlsanity.com/paper/2502.18308/) (2025)
- [A Latent Variable Framework for Scaling Laws in Large Language Models](https://mlsanity.com/paper/2512.06553/) (2025)
- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)
- [Cost-Efficient Estimation of General Abilities Across Benchmarks](https://mlsanity.com/paper/2604.01418/) (2026)
- [From Human-Level AI Tales to AI Leveling Human Scales](https://mlsanity.com/paper/2602.18911/) (2026)
- [Pressure Reveals Character: Behavioural Alignment Evaluation at Depth](https://mlsanity.com/paper/2602.20813/) (2026)
- [SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling](https://mlsanity.com/paper/2605.28179/) (2026)
- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)

## Abstract (arXiv)

Building a theoretical understanding of the capabilities of large language models (LLMs) is vital for our ability to predict and explain the behavior of these systems. Here, we investigate the structure of LLM capabilities by extracting latent capabilities from patterns of individual differences across a varied population of LLMs. Using a combination of Bayesian and frequentist factor analysis, we analyzed data from 29 different LLMs across 27 cognitive tasks. We found evidence that LLM capabilities are not monolithic. Instead, they are better explained by three well-delineated factors that represent reasoning, comprehension and core language modeling. Moreover, we found that these three factors can explain a high proportion of the variance in model performance. These results reveal a consistent structure in the capabilities of different LLMs and demonstrate the multifaceted nature of these capabilities. We also found that the three abilities show different relationships to model properties such as model size and instruction tuning. These patterns help refine our understanding of scaling laws and indicate that changes to a model that improve one ability might simultaneously impair others. Based on these findings, we suggest that benchmarks could be streamlined by focusing on tasks that tap into each broad model ability.