# Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests

arXiv: https://arxiv.org/abs/2502.14359 · HTML: https://mlsanity.com/paper/2502.14359/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 84 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [NATURAL PLAN: Benchmarking LLMs on Natural Language Planning](https://mlsanity.com/paper/2406.04520/) (2024)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents](https://mlsanity.com/paper/2305.13455/) (2023)
- [Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought](https://mlsanity.com/paper/2401.06836/) (2024)
- [Holmes: A Benchmark to Assess the Linguistic Competence of Language Models](https://mlsanity.com/paper/2404.18923/) (2024)
- [InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context](https://mlsanity.com/paper/2406.12203/) (2024)
- [Working Memory Capacity of ChatGPT: An Empirical Study](https://mlsanity.com/paper/2305.03731/) (2023)
- [What A Situated Language-Using Agent Must be Able to Do: A Top-Down Analysis](https://mlsanity.com/paper/2302.08590/) (2023)

## Built on by (most cited first, 12 of 12)

- [A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench](https://mlsanity.com/paper/2507.08491/) (2025)
- [Playpen: An Environment for Exploring Learning Through Conversational Interaction](https://mlsanity.com/paper/2504.08590/) (2025)
- [Do You Get the Hint? Benchmarking LLMs on the Board Game Concept](https://mlsanity.com/paper/2510.13271/) (2025)
- [AI Through the Human Lens: Investigating Cognitive Theories in Machine Psychology](https://mlsanity.com/paper/2506.18156/) (2025)
- [Debiasing International Attitudes: LLM Agents for Simulating US-China Perception Changes](https://mlsanity.com/paper/2508.08837/) (2025)
- [Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time](https://mlsanity.com/paper/2604.08340/) (2026)
- [Assessing mentalization in humans and large language models](https://mlsanity.com/paper/2608.26291/) (2026)
- [CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models](https://mlsanity.com/paper/2607.24999/) (2026)
- [Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks](https://mlsanity.com/paper/2608.03340/) (2026)
- [Investigating Knowledge Transfer Across Interactive Dialogue Games](https://mlsanity.com/paper/2608.23969/) (2026)
- [Bringing AI to Autonomous Systems -- From Cognition to Collective Intelligence](https://mlsanity.com/paper/2609.30291/) (2026)
- [Reputation, Strategy, and Emotion Effects on Generative AI Cooperation: A Comparison Across Reasoning and Non-Reasoning Models](https://mlsanity.com/paper/2610.01222/) (2026)

## Abstract (arXiv)

We examine three evaluation paradigms: standard benchmarks (e.g., MMLU and BBH), interactive games (e.g., Signalling Games or Taboo), and cognitive tests (e.g., for working memory or theory of mind). First, we investigate which of the former two-benchmarks or games-is most effective at discriminating LLMs of varying quality. Then, inspired by human cognitive assessments, we compile a suite of targeted tests that measure cognitive abilities deemed essential for effective language use, and we investigate their correlation with model performance in benchmarks and games. Our analyses reveal that interactive games are superior to standard benchmarks in discriminating models. Causal and logical reasoning correlate with both static and interactive tests, while differences emerge regarding core executive functions and social/emotional skills, which correlate more with games. We advocate for the development of new interactive benchmarks and targeted cognitive tasks inspired by assessing human abilities but designed specifically for LLMs.