mlsanityData to 14 Oct 2026

VideoGameBunny: Towards vision assistants for video games

arXiv 2407.15295 ↗First posted 21 Jul 2024cs.CV cs.SEcited by 11

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large multimodal models (LMMs) hold substantial promise across various domains, from personal assistance in daily tasks to sophisticated applications like medical diagnostics. However, their capabilities have limitations in the video game domain, such as challenges with scene understanding, hallucinations, and inaccurate descriptions of video game content, especially in open-source models. This paper describes the development of VideoGameBunny, a LLaVA-style model based on Bunny, specifically tailored for understanding images from video games. We release intermediate checkpoints, training logs, and an extensive dataset comprising 185,259 video game images from 413 titles, along with 389,565 image-instruction pairs that include image captions, question-answer pairs, and a JSON representation of 16 elements of 136,974 images. Our experiments show that our high quality game-related data has the potential to make a relatively small model outperform the much larger state-of-the-art model LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study paves the way for future research in video game understanding on tasks such as playing, commentary, and debugging. Code and data are available at https://videogamebunny.github.io/

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 62 references that are in the map. First posted on arXiv 21 Jul 2024.

Builds on

45 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023GlitchBench: Can large multimodal models detect video game glitches?8 Dec

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
32 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  13. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  14. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  15. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  16. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  17. 2024What matters when building vision-language models?cited by 326
  18. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  19. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  20. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 176
  21. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  22. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 144
  23. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  24. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  25. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  26. 2024Large Language Models and Games: A Survey and Roadmapcited by 93
  27. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
  28. 2024When Do We Not Need Larger Vision Models?cited by 67
  29. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worldscited by 45
  30. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  31. 2022Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectorscited by 14
  32. 2023AdaRefiner: Refining Decisions of Language Models with Adaptive Feedbackcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 7.

11 papers.

  1. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  2. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  3. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  4. 2026RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Modelscited by 3
  5. 2026Combined Flicker-banding and Moire Removal for Screen-Captured Imagescited by 2
  6. 2026Order from Chaos: Physical World Understanding from Glitchy Gameplay Videoscited by 1
  7. 2026How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videoscited by 1
  8. 2025Grasp-HGN: Grasping the Unexpectedcited by 0
  9. 2026TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videoscited by 0
  10. 2026Bricker to BRACE: A Bracket Exposure RAW Dataset and Restoration Model for Flicker-Bandingcited by 0
  11. 2026WorldAuditBench: Interactive 3D World Auditing with Multimodal Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoGameBench: Can Vision-Language Models complete popular video games?
  2. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models
  3. 2026PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
  4. 2025Are Large Vision Language Models Good Game Players?
  5. 2026Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
  6. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models
  7. 2025An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Models
  8. 2026Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training