# TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos

arXiv: https://arxiv.org/abs/2605.21443 · HTML: https://mlsanity.com/paper/2605.21443/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [GlitchBench: Can large multimodal models detect video game glitches?](https://mlsanity.com/paper/2312.05291/) (2023)
- [PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos](https://mlsanity.com/paper/2412.01800/) (2024)
- [A Survey of Video Game Testing](https://mlsanity.com/paper/2103.06431/) (2021)
- [CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning](https://mlsanity.com/paper/2203.11096/) (2022)
- [Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors](https://mlsanity.com/paper/2210.02506/) (2022)
- [VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models](https://mlsanity.com/paper/2603.06148/) (2026)
- [VideoGameBunny: Towards vision assistants for video games](https://mlsanity.com/paper/2407.15295/) (2024)
- [VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance](https://mlsanity.com/paper/2505.15952/) (2025)
- [Towards Automated Video Game Testing: Still a Long Way to Go](https://mlsanity.com/paper/2202.12777/) (2022)
- [RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Models](https://mlsanity.com/paper/2604.11082/) (2026)
- [Players' Perception of Bugs and Glitches in Video Games: An Exploratory Study](https://mlsanity.com/paper/2504.15408/) (2025)
- [Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models](https://mlsanity.com/paper/2508.04895/) (2025)
- [Are Game Engines Software Frameworks? A Three-perspective Study](https://mlsanity.com/paper/2004.05705/) (2020)

## Abstract (arXiv)

Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames. A preliminary study confirms this gap, showing that temporal glitches are substantially harder for VLMs to detect than spatial ones. To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection. TempGlitch covers five temporal glitch types with balanced per-category samples, together with paired glitch-free videos that enable reliable binary evaluation. We evaluate 12 proprietary and open-weight VLMs across multiple frame-sampling settings. Our results show that current VLMs remain near chance on TempGlitch, often collapsing into either overly conservative behavior that misses most glitches or overly sensitive behavior that flags clean videos as glitchy. Moreover, denser frame sampling and larger model size do not reliably resolve these failures. TempGlitch provides a focused testbed for temporal reasoning, robust gameplay understanding, and automated glitch detection with VLMs. Code and data are available at the project website.