# Leveraging LLM Agents for Automated Video Game Testing

arXiv: https://arxiv.org/abs/2509.22170 · HTML: https://mlsanity.com/paper/2509.22170/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models](https://mlsanity.com/paper/2311.05997/) (2023)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [STEVE-1: A Generative Model for Text-to-Behavior in Minecraft](https://mlsanity.com/paper/2306.00937/) (2023)
- [Scaling Instructable Agents Across Many Simulated Worlds](https://mlsanity.com/paper/2404.10179/) (2024)
- [CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents](https://mlsanity.com/paper/2401.10568/) (2024)
- [A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2403.10249/) (2024)
- [A Survey of Video Game Testing](https://mlsanity.com/paper/2103.06431/) (2021)
- [Playing NetHack with LLMs: Potential & Limitations as Zero-Shot Agents](https://mlsanity.com/paper/2403.00690/) (2024)
- [Towards Automated Video Game Testing: Still a Long Way to Go](https://mlsanity.com/paper/2202.12777/) (2022)
- [Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing](https://mlsanity.com/paper/1602.01226/) (2016)
- [Exploiting Machine Unlearning for Backdoor Attacks in Deep Learning System](https://mlsanity.com/paper/2310.10659/) (2023)
- [Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation](https://mlsanity.com/paper/2508.07745/) (2025)
- [Automatic Bug Detection in LLM-Powered Text-Based Games Using LLMs](https://mlsanity.com/paper/2406.04482/) (2024)
- [On Automating Video Game Regression Testing by Planning and Learning](https://mlsanity.com/paper/2402.12393/) (2024)
- [Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents](https://mlsanity.com/paper/2505.03947/) (2025)

## Built on by (most cited first, 5 of 5)

- [GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers](https://mlsanity.com/paper/2604.02648/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [CA2: Code-Aware Agent for Automated Game Testing](https://mlsanity.com/paper/2605.13918/) (2026)
- [Game-Creation-Agents: Autonomous Game Production](https://mlsanity.com/paper/2610.16776/) (2026)

## Abstract (arXiv)

Testing MMORPGs (Massively Multiplayer Online Role-Playing Games) is a critical yet labor-intensive task in game development due to their complexity and frequent updating nature. Traditional automated game testing approaches struggle to achieve high state coverage and efficiency in these rich, open-ended environments, while existing LLM-based game-playing approaches are limited to shallow reasoning ability in understanding complex game state-action spaces and long-complex tasks. To address these challenges, we propose TITAN, an effective LLM-driven agent framework for intelligent MMORPG testing. TITAN incorporates four key components to: (1) perceive and abstract high-dimensional game states, (2) proactively optimize and prioritize available actions, (3) enable long-horizon reasoning with action trace memory and reflective self-correction, and (4) employ LLM-based oracles to detect potential functional and logic bugs with diagnostic reports. We implement the prototype of TITAN and evaluate it on two large-scale commercial MMORPGs spanning both PC and mobile platforms. In our experiments, TITAN achieves significantly higher task completion rates (95%) and bug detection performance compared to existing automated game testing approaches. An ablation study further demonstrates that each core component of TITAN contributes substantially to its overall performance. Notably, TITAN detects four previously unknown bugs that prior testing approaches fail to identify. We provide an in-depth discussion of these results, which offer guidance for new avenues of advancing intelligent, general-purpose testing systems. Moreover, TITAN has been deployed in eight real-world game QA pipelines, underscoring its practical impact as an LLM-driven game testing framework.