mlsanityData to 7 Oct 2026

Are Large Vision Language Models Good Game Players?

arXiv 2503.02358 ↗First posted 4 Mar 2025cs.CV cs.AI cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Vision Language Models (LVLMs) have demonstrated remarkable abilities in understanding and reasoning about both visual and textual information. However, existing evaluation methods for LVLMs, primarily based on benchmarks like Visual Question Answering and image captioning, often fail to capture the full scope of LVLMs' capabilities. These benchmarks are limited by issues such as inadequate assessment of detailed visual perception, data contamination, and a lack of focus on multi-turn reasoning. To address these challenges, we propose method, a game-based evaluation framework designed to provide a comprehensive assessment of LVLMs' cognitive and reasoning skills in structured environments. method uses a set of games to evaluate LVLMs on four core tasks: Perceiving, Question Answering, Rule Following, and End-to-End Playing, with each target task designed to assess specific abilities, including visual perception, reasoning, decision-making, etc. Based on this framework, we conduct extensive experiments that explore the limitations of current LVLMs, such as handling long structured outputs and perceiving detailed and dense elements. Code and data are publicly available at https://github.com/xinke-wang/LVLM-Playground.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 57 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

42 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs2 Jul

Other references · 41

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2019Towards VQA Models That Can Readcited by 1,696
29 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2018nocaps: novel object captioning at scalecited by 444
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  10. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  11. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  12. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  13. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  14. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  15. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  16. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  17. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  18. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  19. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  20. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  21. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  22. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  23. 2024Needle In A Multimodal Haystackcited by 56
  24. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  25. 2024Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Casecited by 23
  26. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  27. 2024ModaVerse: Efficiently Transforming Modalities with LLMscited by 18
  28. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 6
  29. 2024Towards Flexible Evaluation for Generative Visual Question Answeringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  2. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  3. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  4. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  5. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  6. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  7. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  8. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  9. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  10. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  11. 2026OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamicscited by 4
  12. 2025From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentarycited by 3
7 more
  1. 2025FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Gamescited by 3
  2. 2026ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenascited by 3
  3. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzlescited by 2
  4. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  5. 2025Who is a Better Player: LLM against LLMcited by 1
  6. 2025StarBench: A Turn-Based RPG Benchmark for Agentic Multimodal Decision-Making and Information Seekingcited by 1
  7. 2026OptiArena: Can LLMs Improve Executable Algorithms under Fixed Resource Budgets?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoGameBench: Can Vision-Language Models complete popular video games?
  2. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
  3. 2025Where do Large Vision-Language Models Look at when Answering Questions?
  4. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision
  5. 2024A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models
  6. 2025Vision-Language Model Dialog Games for Self-Improvement
  7. 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning
  8. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models