mlsanityData to 14 Oct 2026

DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments

arXiv 2503.06047 ↗First posted 8 Mar 2025cs.AI cs.CLcited by 23

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language model (LLM)-based agents are increasingly applied to complex strategic environments that demand long-horizon reasoning, multi-agent interaction, and decision-making under uncertainty. However, common existing benchmarks either assess isolated skills, lack environmental diversity, or rely on broad overall metrics. To address these issues, we introduce DSGBench, a more rigorous evaluation platform for strategic decision-making tasks. Firstly, it incorporates six complex strategic games which serve as ideal testbeds due to their long-term and multi-dimensional decision-making demands and flexibility in customizing tasks with various difficulty levels and targets. Secondly, DSGBench employs a fine-grained evaluation scoring system which examines the decision-making capabilities by looking into the performance in five specific dimensions, offering a comprehensive assessment in a better-designed fashion. Furthermore, DSGBench also incorporates an automated decision-tracking mechanism which enables in-depth analysis of agent behaviour patterns and the turning points in their strategies. We evaluate six popular LLM agents, including open-source and closed-source models, and observe distinct strengths and limitations among various tasks. Through decision trajectory analysis, we further identify systemic limitations in different LLMs. These findings offer valuable insights for model selection and future LLM-based agent development.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 47 of its 66 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

47 of 66 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Digital Player: Evaluating Large Language Models based Human-like Agent in Games28 Feb
  2. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun
  3. 2024A Survey on Large Language Model-Based Game Agents2 Apr
  4. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments18 Mar
  5. 2024SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models31 Jan
  6. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents19 Jan
  7. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach19 Dec
  8. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration14 Nov
  9. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agents2 Oct
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
25 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  4. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  6. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  7. 2024A Survey on Large Language Models for Code Generationcited by 658
  8. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  9. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  10. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  11. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  12. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  13. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 163
  14. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  15. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  16. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  17. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  18. 2020Thinking Fast and Slow in AIcited by 70
  19. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  20. 2023Strategic Reasoning with Language Modelscited by 64
  21. 2023Lyfe Agents: Generative agents for low-cost real-time social interactionscited by 49
  22. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 45
  23. 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 45
  24. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  25. 2024FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  2. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 31
  3. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 13
  4. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  5. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 9
  6. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  7. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  8. 2025PuzzleJAX: A Benchmark for Reasoning and Learningcited by 4
  9. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  10. 2025Matching Markets Meet LLMs: Algorithmic Reasoning with Ranked Preferencescited by 2
  11. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  12. 2026AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agentscited by 2
11 more
  1. 2025Do LLMs Know When to Flip a Coin? Strategic Randomization through Reasoning and Experiencecited by 1
  2. 2025SAG-Agent: Enabling Long-Horizon Reasoning in Strategy Games via Dynamic Knowledge Graphscited by 1
  3. 2025Beyond the Black Box: Demystifying Multi-Turn LLM Reasoning with VISTAcited by 1
  4. 2026To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulationcited by 1
  5. 2026Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discoverycited by 1
  6. 2025WereWolf-Plus: An Update of Werewolf Game setting Based on DSGBenchcited by 0
  7. 2026LATTICE: Evaluating Decision Support Utility of Crypto Agentscited by 0
  8. 2026The Evaluation Game: Beyond Static LLM Benchmarkingcited by 0
  9. 2026Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?cited by 0
  10. 2026Experience-Sensitive Game Learning: A Behavioral Study of Humans and Language Agentscited by 0
  11. 2026Choosing LLMs for Strategic Agent Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making
  2. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
  3. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
  4. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs
  5. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  6. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
  7. 2025CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs
  8. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors