mlsanityData to 14 Oct 2026

BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

arXiv 2411.13543 ↗First posted 20 Nov 2024cs.AIcited by 146

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies-areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as several models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community. Code and Leaderboard at balrogai.com.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 76 of its 95 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

76 of 95 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Enhance Reasoning for Large Language Models in the Game Werewolf4 Feb
  2. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks2 Feb

Other references · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
38 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  13. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  14. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  15. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  16. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  17. 2022A Generalist Agentcited by 808
  18. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  19. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  20. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  21. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  22. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  23. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  24. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  25. 2024Automated Design of Agentic Systemscited by 407
  26. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  27. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  28. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  29. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  30. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  31. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  32. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  33. 2020The NetHack Learning Environmentcited by 198
  34. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  35. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  36. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  37. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  38. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 68; 2026: 75.

146 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  3. 2025Towards a Science of Scaling Agent Systemscited by 138
  4. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  5. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  6. 2025Agentic Large Language Models, a surveycited by 108
  7. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 100
  8. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  9. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  10. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  11. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  12. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
38 more
  1. 2025A Rosetta Stone for AI Benchmarkscited by 36
  2. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  3. 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoningcited by 32
  4. 2025ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflectioncited by 31
  5. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 31
  6. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 31
  7. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  8. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoningcited by 28
  9. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
  10. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  11. 2025Sudoku-Bench: Evaluating creative reasoning with Sudoku variantscited by 22
  12. 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacracited by 22
  13. 2026Large Language Model Reasoning Failurescited by 21
  14. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agentscited by 20
  15. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Taskscited by 15
  16. 2025Synthesizing world models for bilevel planningcited by 14
  17. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 13
  18. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  19. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representationscited by 13
  20. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedbackcited by 12
  21. 2026AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Gamescited by 12
  22. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 12
  23. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 11
  24. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 11
  25. 2026textttYC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Executioncited by 11
  26. 2025Benchmarking LLMs' Swarm intelligencecited by 10
  27. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 9
  28. 2026The Illusion of Stochasticity in LLMscited by 9
  29. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 8
  30. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  31. 2025Factorio Learning Environmentcited by 7
  32. 2025FlySearch: Exploring how vision-language models explorecited by 7
  33. 2025Enhancing Decision-Making of Large Language Models via Actor-Criticcited by 7
  34. 2025StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valleycited by 7
  35. 2025LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactionscited by 7
  36. 2026Workspace Optimization: How to Train Your Agentcited by 7
  37. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  38. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 7

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  2. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments
  3. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
  4. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
  5. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
  6. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  7. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  8. 2026SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire