mlsanityData to 14 Oct 2026

Digital Player: Evaluating Large Language Models based Human-like Agent in Games

arXiv 2502.20807 ↗First posted 28 Feb 2025cs.LGcited by 6

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

With the rapid advancement of Large Language Models (LLMs), LLM-based autonomous agents have shown the potential to function as digital employees, such as digital analysts, teachers, and programmers. In this paper, we develop an application-level testbed based on the open-source strategy game "Unciv", which has millions of active players, to enable researchers to build a "data flywheel" for studying human-like agents in the "digital players" task. This "Civilization"-like game features expansive decision-making spaces along with rich linguistic interactions such as diplomatic negotiations and acts of deception, posing significant challenges for LLM-based agents in terms of numerical reasoning and long-term planning. Another challenge for "digital players" is to generate human-like responses for social interaction, collaboration, and negotiation with human players. The open-source project can be found at https:/github.com/fuxiAIlab/CivAgent.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 48 of its 81 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

48 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory25 May
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023Mistral 7Bcited by 3,164
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
33 more
  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  6. 2022A Generalist Agentcited by 808
  7. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  8. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  12. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  13. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  14. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  15. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  16. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  17. 2022TALM: Tool Augmented Language Modelscited by 197
  18. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  19. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  20. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 145
  21. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  22. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  23. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  24. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 98
  25. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 58
  26. 2023Large Language Models Are Semi-Parametric Reinforcement Learning Agentscited by 42
  27. 2021Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabicited by 40
  28. 2022Large Language Models and the Reverse Turing Testcited by 38
  29. 2023Does GPT-4 pass the Turing test?cited by 37
  30. 2024FlowMind: Automatic Workflow Generation with LLMscited by 30
  31. 2024Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialoguescited by 29
  32. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 24
  33. 2021A Survey of Text Games for Reinforcement Learning informed by Natural Languagecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  2. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 23
  3. 2025Vox Deorum: A Hybrid LLM Architecture for 4X / Grand Strategy Game AI -- Lessons from Civilization Vcited by 4
  4. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  5. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  6. 2026To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs as Agentic Cooperative Players in Multiplayer UNO
  2. 2024A Survey on Large Language Model-Based Game Agents
  3. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios
  4. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systems
  5. 2025Evaluating LLMs in Open-Source Games
  6. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  7. 2024Large Language Models as Agents in Two-Player Games
  8. 2025Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity?