mlsanityData to 14 Oct 2026

GlitchBench: Can large multimodal models detect video game glitches?

arXiv 2312.05291 ↗First posted 8 Dec 2023cs.CV cs.AI cs.CLcited by 23

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large multimodal models (LMMs) have evolved from large language models (LLMs) to integrate multiple input modalities, such as visual inputs. This integration augments the capacity of LLMs for tasks requiring visual comprehension and reasoning. However, the extent and limitations of their enhanced abilities are not fully understood, especially when it comes to real-world tasks. To address this gap, we introduce GlitchBench, a novel benchmark derived from video game quality assurance tasks, to test and evaluate the reasoning capabilities of LMMs. Our benchmark is curated from a variety of unusual and glitched scenarios from video games and aims to challenge both the visual and linguistic reasoning powers of LMMs in detecting and interpreting out-of-the-ordinary events. We evaluate multiple state-of-the-art LMMs, and we show that GlitchBench presents a new challenge for these models. Code and data are available at: https://glitchbench.github.io/

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 86 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

65 of 86 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct

Other references · 64

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  17. 2016A Diagram Is Worth A Dozen Imagescited by 804
  18. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  19. 2018Object Hallucination in Image Captioningcited by 669
  20. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  21. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  22. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  23. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  24. 2018nocaps: novel object captioning at scalecited by 444
  25. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  26. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  27. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  28. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  29. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  30. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  31. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  32. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  33. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  34. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  35. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80
  36. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 72
  37. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  38. 2019AI2D-RST: A multimodal corpus of 1000 primary school science diagramscited by 60

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 10.

23 papers.

Sort:

By area:

  1. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  2. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  3. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systemscited by 26
  4. 2024Large Language Models and Video Games: A Preliminary Scoping Reviewcited by 23
  5. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  6. 2024VideoGameBunny: Towards vision assistants for video gamescited by 11
  7. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 11
  8. 2024GPT for Games: An Updated Scoping Review (2020-2024)cited by 10
  9. 2024Level Up Your Tutorials: VLMs for Game Tutorials Quality Assessmentcited by 3
  10. 2025CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environmentscited by 3
  11. 2026GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agentscited by 3
  12. 2026Open-Ended Video Game Glitch Detection with Agentic Reasoning and Temporal Groundingcited by 3
11 more
  1. 2026RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Modelscited by 3
  2. 2026AI for Games in the Foundation Model Eracited by 3
  3. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  4. 2025A Hybrid Co-Finetuning Approach for Visual Bug Detection in Video Gamescited by 2
  5. 2026Order from Chaos: Physical World Understanding from Glitchy Gameplay Videoscited by 1
  6. 2026How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videoscited by 1
  7. 2025Runtime Failure Hunting for Physics Engine Based Software Systems: How Far Can We Go?cited by 0
  8. 2026TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videoscited by 0
  9. 2026Typed Temporal Interaction Features for Simulation-Backed Forecasting of Open-Source Game Release Incidentscited by 0
  10. 2026WorldAuditBench: Interactive 3D World Auditing with Multimodal Agentscited by 0
  11. 2026SpeedrunBench: Challenging LLM Agents with Video Game Speedrunningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoGameBench: Can Vision-Language Models complete popular video games?
  2. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
  3. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models
  4. 2026Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
  5. 2025PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving
  6. 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
  7. 2026See2Think: Do Multimodal Models Really Use Intermediate Visual States?
  8. 2025VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance