# VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance

arXiv: https://arxiv.org/abs/2505.15952 · HTML: https://mlsanity.com/paper/2505.15952/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models](https://mlsanity.com/paper/2502.09696/) (2025)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [Provable Scaling Laws for the Test-Time Compute of Large Language Models](https://mlsanity.com/paper/2411.19477/) (2024)
- [HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance](https://mlsanity.com/paper/2407.06937/) (2024)
- [PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos](https://mlsanity.com/paper/2412.01800/) (2024)
- [PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance](https://mlsanity.com/paper/2411.02327/) (2024)
- [Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors](https://mlsanity.com/paper/2210.02506/) (2022)
- [Detecting Human Artifacts from Text-to-Image Models](https://mlsanity.com/paper/2411.13842/) (2024)
- [Learning to Identify Perceptual Bugs in 3D Video Games](https://mlsanity.com/paper/2202.12884/) (2022)
- [Evaluating and Predicting Distorted Human Body Parts for Generated Images](https://mlsanity.com/paper/2503.00811/) (2025)
- [Weak Supervision for Label Efficient Visual Bug Detection](https://mlsanity.com/paper/2309.11077/) (2023)
- [Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities](https://mlsanity.com/paper/2410.08534/) (2024)

## Built on by (most cited first, 11 of 11)

- [GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?](https://mlsanity.com/paper/2608.21833/) (2026)
- [WP-CLIP: Leveraging CLIP to Predict W"olfflin's Principles in Visual Art](https://mlsanity.com/paper/2508.12668/) (2025)
- [GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents](https://mlsanity.com/paper/2603.24329/) (2026)
- [Open-Ended Video Game Glitch Detection with Agentic Reasoning and Temporal Grounding](https://mlsanity.com/paper/2604.07818/) (2026)
- [RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Models](https://mlsanity.com/paper/2604.11082/) (2026)
- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos](https://mlsanity.com/paper/2603.22706/) (2026)
- [EA-Swin: An Embedding-Agnostic Swin Transformer for AI-Generated Video Detection](https://mlsanity.com/paper/2602.17260/) (2026)
- [TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos](https://mlsanity.com/paper/2605.21443/) (2026)
- [Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA](https://mlsanity.com/paper/2607.25921/) (2026)
- [WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents](https://mlsanity.com/paper/2609.40325/) (2026)

## Abstract (arXiv)

With video games now generating the highest revenues in the entertainment industry, optimizing game development workflows has become essential for the sector's sustained growth. Recent advancements in Vision-Language Models (VLMs) offer considerable potential to automate and enhance various aspects of game development, particularly Quality Assurance (QA), which remains one of the industry's most labor-intensive processes with limited automation options. To accurately evaluate the performance of VLMs in video game QA tasks and determine their effectiveness in handling real-world scenarios, there is a clear need for standardized benchmarks, as existing benchmarks are insufficient to address the specific requirements of this domain. To bridge this gap, we introduce VideoGameQA-Bench, a comprehensive benchmark that covers a wide array of game QA activities, including visual unit testing, visual regression testing, needle-in-a-haystack tasks, glitch detection, and bug report generation for both images and videos of various games. Code and data are available at: https://asgaardlab.github.io/videogameqa-bench/