# RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Models

arXiv: https://arxiv.org/abs/2604.11082 · HTML: https://mlsanity.com/paper/2604.11082/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [GlitchBench: Can large multimodal models detect video game glitches?](https://mlsanity.com/paper/2312.05291/) (2023)
- [PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos](https://mlsanity.com/paper/2412.01800/) (2024)
- [CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning](https://mlsanity.com/paper/2203.11096/) (2022)
- [Instruction Tuning with Human Curriculum](https://mlsanity.com/paper/2310.09518/) (2023)
- [VideoGameBunny: Towards vision assistants for video games](https://mlsanity.com/paper/2407.15295/) (2024)
- [VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance](https://mlsanity.com/paper/2505.15952/) (2025)
- [Using Deep Convolutional Neural Networks to Detect Rendered Glitches in Video Games](https://mlsanity.com/paper/2406.08231/) (2024)
- [Using Gameplay Videos for Detecting Issues in Video Games](https://mlsanity.com/paper/2307.14749/) (2023)
- [Finding the Needle in a Haystack: Detecting Bug Occurrences in Gameplay Videos](https://mlsanity.com/paper/2311.10926/) (2023)
- [Players' Perception of Bugs and Glitches in Video Games: An Exploratory Study](https://mlsanity.com/paper/2504.15408/) (2025)
- [Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models](https://mlsanity.com/paper/2508.04895/) (2025)

## Built on by (most cited first, 3 of 3)

- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos](https://mlsanity.com/paper/2605.21443/) (2026)
- [Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA](https://mlsanity.com/paper/2607.25921/) (2026)

## Abstract (arXiv)

Visual glitches in video games degrade player experience and perceived quality, yet manual quality assurance cannot keep pace with the growing test surface of modern game development. Prior automation efforts, particularly those using vision-language models (VLMs), largely operate on isolated frames without sufficient context to judge whether a glitch is present. We introduce RefGlitch-Bench, a benchmark for reference-based video game glitch detection with VLMs. The key idea is to formulate glitch detection as an explicit within-video comparison problem: given a test frame, a reference frame provides a visual baseline that helps the model distinguish true glitches from benign visual variation. RefGlitch-Bench includes a controlled synthetic dataset with five injected glitch types and manually annotated reference/test frame pairs, enabling an oracle-reference evaluation that isolates the potential benefit of reference guidance. We further establish four initial baselines for automatically selecting references from earlier frames in the same video, with LastCleanFrame performing best and transferring across VLMs. Finally, we evaluate automatic reference guidance on real-world gameplay data, where it improves frame-level glitch detection beyond the controlled setting while revealing reference reliability and error propagation as key challenges. Code and data are available at: https://github.com/PipiZong/RefGlitch-Bench.git.