mlsanityData to 7 Oct 2026

GameArena: Evaluating LLM Reasoning through Live Computer Games

arXiv 2412.06394 ↗First posted 9 Dec 2024cs.AI cs.CLcited by 32

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Evaluating the reasoning abilities of large language models (LLMs) is challenging. Existing benchmarks often depend on static datasets, which are vulnerable to data contamination and may get saturated over time, or on binary live human feedback that conflates reasoning with other abilities. As the most prominent dynamic benchmark, Chatbot Arena evaluates open-ended questions in real-world settings, but lacks the granularity in assessing specific reasoning capabilities. We introduce GameArena, a dynamic benchmark designed to evaluate LLM reasoning capabilities through interactive gameplay with humans. GameArena consists of three games designed to test specific reasoning capabilities (e.g., deductive and inductive reasoning), while keeping participants entertained and engaged. We analyze the gaming data retrospectively to uncover the underlying reasoning processes of LLMs and measure their fine-grained reasoning capabilities. We collect over 2000 game sessions and provide detailed assessments of various reasoning capabilities for five state-of-the-art LLMs. Our user study with 100 participants suggests that GameArena improves user engagement compared to Chatbot Arena. For the first time, GameArena enables the collection of step-by-step LLM reasoning data in the wild.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 30 of its 57 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

30 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Self-playing Adversarial Language Game Enhances LLM Reasoning16 Apr

Other references · 29

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  11. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  12. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
17 more
  1. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  3. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  6. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  7. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  8. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  9. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  10. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  11. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  12. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 40
  13. 2022Language Models as Inductive Reasonerscited by 32
  14. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 25
  15. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  16. 2022A Property Induction Framework for Neural Language Modelscited by 8
  17. 2024A LLM Benchmark based on the Minecraft Builder Dialog Agent Taskcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 12.

32 papers.

Sort:

By area:

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  2. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  3. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  4. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 29
  5. 2025TextArenacited by 24
  6. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  8. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  9. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  10. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Playcited by 10
  11. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  12. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
20 more
  1. 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluationcited by 7
  2. 2025GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketingcited by 7
  3. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  4. 2025Can Large Models Fool the Eye? A New Turing Test for Biological Animationcited by 6
  5. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Makingcited by 5
  6. 2025EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVercited by 5
  7. 2026DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrationscited by 5
  8. 2026OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamicscited by 4
  9. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  10. 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
  11. 2025Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Modelscited by 2
  12. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  13. 2025Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interactioncited by 1
  14. 2026MineNPC-Task: Task Suite for Memory-Aware Minecraft Agentscited by 1
  15. 2026Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Gamecited by 0
  16. 2026Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+cited by 0
  17. 2026Experience-Sensitive Game Learning: A Behavioral Study of Humans and Language Agentscited by 0
  18. 2026Clueing up LLMs with Tool-Augmented Deductive Reasoningcited by 0
  19. 2026WordPolo: Evaluating Language Models Through Iterative Semantic Feedbackcited by 0
  20. 2026Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  2. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  3. 2025ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models
  4. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
  5. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  6. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  7. 2025Do You Get the Hint? Benchmarking LLMs on the Board Game Concept
  8. 2025Baba is LLM: Reasoning in a Game with Dynamic Rules