mlsanityData to 7 Oct 2026

LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents

arXiv 2410.02829 ↗First posted 1 Oct 2024cs.AI cs.HC cs.LGcited by 5

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have demonstrated their potential as autonomous agents across various tasks. One emerging application is the use of LLMs in playing games. In this work, we explore a practical problem for the gaming industry: Can LLMs be used to measure game difficulty? We propose a general game-testing framework using LLM agents and test it on two widely played strategy games: Wordle and Slay the Spire. Our results reveal an interesting finding: although LLMs may not perform as well as the average human player, their performance, when guided by simple, generic prompting techniques, shows a statistically significant and strong correlation with difficulty indicated by human players. This suggests that LLMs could serve as effective agents for measuring game difficulty during the development process. Based on our experiments, we also outline general principles and guidelines for incorporating LLMs into the game testing process.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 79 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

25 of 79 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024A Survey on Large Language Model-Based Game Agents2 Apr
  2. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents19 Jan
  3. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach19 Dec
  4. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game29 Oct
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 20

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  5. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  6. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  7. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activitiescited by 154
  8. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  9. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  10. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  11. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  12. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
8 more
  1. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
  2. 2023Large Language Models Can Be Used to Estimate the Latent Positions of Politicianscited by 33
  3. 2019Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creatorscited by 31
  4. 2020Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kingscited by 18
  5. 2021A Survey of Video Game Testingcited by 18
  6. 2018Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithmscited by 13
  7. 2021Player-AI Interaction: What Neural Network Games Reveal About AI as Playcited by 8
  8. 2020From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AIcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoningcited by 8
  2. 2025Exploring Large Language Models for Word Games:Who is the Spy?cited by 6
  3. 2025Do You Get the Hint? Benchmarking LLMs on the Board Game Conceptcited by 2
  4. 2026AI for Games in the Foundation Model Eracited by 2
  5. 2026Nemobot Games: Crafting Strategic AI Gaming Agents for Interactive Learning with Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
  2. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agents
  3. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
  4. 2025Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Games
  5. 2025LLMs as Agentic Cooperative Players in Multiplayer UNO
  6. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors
  7. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
  8. 2025Leveraging LLM Agents for Automated Video Game Testing