# Automatic Bug Detection in LLM-Powered Text-Based Games Using LLMs

arXiv: https://arxiv.org/abs/2406.04482 · HTML: https://mlsanity.com/paper/2406.04482/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 62 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 4 of 4)

- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Player-Driven Emergence in LLM-Driven Game Narrative](https://mlsanity.com/paper/2404.17027/) (2024)
- [ChatGPT4PCG Competition: Character-like Level Generation for Science Birds](https://mlsanity.com/paper/2303.15662/) (2023)
- [Automatically Detecting Visual Bugs in HTML5 <canvas> Games](https://mlsanity.com/paper/2208.02335/) (2022)

## Built on by (most cited first, 2 of 2)

- [Leveraging LLM Agents for Automated Video Game Testing](https://mlsanity.com/paper/2509.22170/) (2025)
- [Tricking LLM-Based NPCs into Spilling Secrets](https://mlsanity.com/paper/2508.19288/) (2025)

## Abstract (arXiv)

Advancements in large language models (LLMs) are revolutionizing interactive game design, enabling dynamic plotlines and interactions between players and non-player characters (NPCs). However, LLMs may exhibit flaws such as hallucinations, forgetfulness, or misinterpretations of prompts, causing logical inconsistencies and unexpected deviations from intended designs. Automated techniques for detecting such game bugs are still lacking. To address this, we propose a systematic LLM-based method for automatically identifying such bugs from player game logs, eliminating the need for collecting additional data such as post-play surveys. Applied to a text-based game DejaBoom!, our approach effectively identifies bugs inherent in LLM-powered interactive games, surpassing unstructured LLM-powered bug-catching methods and filling the gap in automated detection of logical and design flaws.