# Human-AI Collaborative Game Testing with Vision Language Models

arXiv: https://arxiv.org/abs/2501.11782 · HTML: https://mlsanity.com/paper/2501.11782/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 54 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [Contrastive Learning Reduces Hallucination in Conversations](https://mlsanity.com/paper/2212.10400/) (2022)
- [DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems](https://mlsanity.com/paper/2407.10701/) (2024)
- [Uncalibrated Models Can Improve Human-AI Collaboration](https://mlsanity.com/paper/2202.05983/) (2022)
- [Automated Video Game Testing Using Synthetic and Human-Like Agents](https://mlsanity.com/paper/1906.00317/) (2019)
- [Dissonance Between Human and Machine Understanding](https://mlsanity.com/paper/2101.07337/) (2021)
- [A Survey of Video Game Testing](https://mlsanity.com/paper/2103.06431/) (2021)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation](https://mlsanity.com/paper/2207.08379/) (2022)
- [Honest AI: Fine-Tuning "Small" Language Models to Say "I Don't Know", and Reducing Hallucination in RAG](https://mlsanity.com/paper/2410.09699/) (2024)
- [An Agent-based Approach to Automated Game Testing: an Experience Report](https://mlsanity.com/paper/2211.06386/) (2022)
- [RESTful API Automated Test Case Generation](https://mlsanity.com/paper/1901.01538/) (2019)
- [SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning](https://mlsanity.com/paper/2203.05566/) (2022)

## Built on by (most cited first, 1 of 1)

- [VRScout: Towards Real-Time, Autonomous Testing of Virtual Reality Games](https://mlsanity.com/paper/2511.00002/) (2025)

## Abstract (arXiv)

As modern video games become increasingly complex, traditional manual testing methods are proving costly and inefficient, limiting the ability to ensure high-quality game experiences. While advancements in Artificial Intelligence (AI) offer the potential to assist human testers, the effectiveness of AI in truly enhancing real-world human performance remains underexplored. This study investigates how AI can improve game testing by developing and experimenting with an AI-assisted workflow that leverages state-of-the-art machine learning models for defect detection. Through an experiment involving 800 test cases and 276 participants of varying backgrounds, we evaluate the effectiveness of AI assistance under four conditions: with or without AI support, and with or without detailed knowledge of defects and design documentation. The results indicate that AI assistance significantly improves defect identification performance, particularly when paired with detailed knowledge. However, challenges arise when AI errors occur, negatively impacting human decision-making. Our findings show the importance of optimizing human-AI collaboration and implementing strategies to mitigate the effects of AI inaccuracies. By this research, we demonstrate AI's potential and problems in enhancing efficiency and accuracy in game testing workflows and offers practical insights for integrating AI into the testing process.