mlsanityData to 14 Oct 2026

lmgame-Bench: How Good are LLMs at Playing Games?

arXiv 2505.15146 ↗First posted 21 May 2025cs.AIcited by 55

Not on the map. Why?

Cited by 31 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Playing video games requires perception, memory, and planning, exactly the faculties modern large language model (LLM) agents are expected to master. We study the major challenges in using popular video games to evaluate modern LLMs and find that directly dropping LLMs into games cannot make an effective evaluation, for three reasons -- brittle vision perception, prompt sensitivity, and potential data contamination. We introduce lmgame-Bench to turn games into reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and narrative games delivered through a unified Gym-style API and paired with lightweight perception and memory scaffolds, and is designed to stabilize prompt variance and remove contamination. Across 13 leading models, we show lmgame-Bench is challenging while still separating models well. Correlation analysis shows that every game probes a unique blend of capabilities often tested in isolation elsewhere. More interestingly, performing reinforcement learning on a single game from lmgame-Bench transfers both to unseen games and to external planning tasks. Our evaluation code is available at https://github.com/lmgame-org/GamingAgent/lmgame-bench.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 44 of its 81 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

44 of 81 references.

References · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  10. 2016OpenAI Gymcited by 1,736
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
32 more
  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  4. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  5. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  6. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  7. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  8. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  10. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  11. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
  12. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  13. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  14. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  15. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  16. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
  17. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  18. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  19. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  20. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 95
  21. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 61
  22. 2018General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithmscited by 45
  23. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 37
  24. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  25. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  26. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  27. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challengescited by 19
  28. 2025Are Large Vision Language Models Good Game Players?cited by 19
  29. 2024Can LLM-Augmented autonomous agents cooperate?, An evaluation of their cooperative capabilities through Melting Potcited by 11
  30. 2024GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Mapscited by 8
  31. 2025TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoningcited by 8
  32. 2025Factorio Learning Environmentcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 42.

55 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  3. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 62
  4. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  5. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 36
  6. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 31
  7. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMscited by 28
  8. 2025Code World Models for General Game Playingcited by 26
  9. 2026MMSkills: Towards Multimodal Skills for General Visual Agentscited by 16
  10. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Taskscited by 15
  11. 2025Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planningcited by 12
  12. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 12
38 more
  1. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 11
  2. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Playcited by 11
  3. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 11
  4. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 9
  5. 2026GameVerse: Can Vision-Language Models Learn from Video-based Reflection?cited by 8
  6. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  7. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Gamescited by 5
  8. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMscited by 5
  9. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 4
  10. 2026OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamicscited by 4
  11. 2025FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Gamescited by 3
  12. 2026Gym-V: A Unified Vision Environment System for Agentic Vision Researchcited by 3
  13. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 3
  14. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  15. 2026Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargainingcited by 3
  16. 2026AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agentscited by 3
  17. 2026CAST: Game Solvers as Turn-Level Teachers for LLM Agentscited by 3
  18. 2026SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Modelscited by 2
  19. 2026Under the Influence: Quantifying Persuasion and Vigilance in Large Language Modelscited by 2
  20. 2026Do Vision Language Models Understand Human Engagement in Games?cited by 1
  21. 2026TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMscited by 1
  22. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  23. 2025The STAR-XAI Protocol: A Framework for Inducing and Verifying Agency, Reasoning, and Reliability in AI Agentscited by 0
  24. 2025Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observationcited by 0
  25. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playingcited by 0
  26. 2026MOSAIC: A Universal Agent-Level Interface for Cross-Paradigm Agent Mixing and Human-AI Collaborationcited by 0
  27. 2026Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Servedcited by 0
  28. 2026Spatial Competence Benchmarkcited by 0
  29. 2026Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Playcited by 0
  30. 2026PTCG-Bench: Can LLM Agents Master Pok'emon Trading Card Game?cited by 0
  31. 2026Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Gamescited by 0
  32. 2026Experience-Sensitive Game Learning: A Behavioral Study of Humans and Language Agentscited by 0
  33. 2026Investigating Knowledge Transfer Across Interactive Dialogue Gamescited by 0
  34. 2026V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environmentscited by 0
  35. 2026Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilizationcited by 0
  36. 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agentscited by 0
  37. 2026Self-Play Search Distillation for Large Language Model Reasoningcited by 0
  38. 2026Witness: Discovery, Deciphering, and Epiphany in Interactive Puzzle Environmentscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  2. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
  3. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  4. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs
  5. 2025Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Games
  6. 2025Diamonds in the rough: Transforming SPARCs of imagination into a game concept by leveraging medium sized LLMs ↗
  7. 2026Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
  8. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors