mlsanityData to 14 Oct 2026

CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V

arXiv 2604.07733 ↗First posted 9 Apr 2026cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide all three, and fewer still offer evaluation signals rich enough for long-horizon, multi-agent play. We introduce CivBench, a benchmark for LLM strategists (a model under an agentic setup) in multiplayer Civilization V. Because terminal win/loss is too sparse a signal in games spanning hundreds of turns and multiple opponents, CivBench trains estimators of a state-value function, the probability that a player eventually wins given the game state. cross 307 games with 7 LLMs and multiple CivBench agent conditions, we demonstrate CivBench's potential to estimate strategic capabilities in Civilization V as an unsaturated benchmark, reveal model-specific effects of agentic setup, and outline distinct strategic profiles not visible through outcome-only evaluation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 41 of its 54 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

41 of 54 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration2 Mar
  2. 2025Digital Player: Evaluating Large Language Models based Human-like Agent in Games28 Feb
  3. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  4. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun
  5. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments26 Feb
  6. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents19 Jan
  7. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach19 Dec

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  6. 2017Deep Setscited by 1,421
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  12. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
22 more
  1. 2025Survey on Evaluation of LLM-based Agentscited by 204
  2. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 178
  3. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  4. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  5. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  6. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
  7. 2024Escalation Risks from Language Models in Military and Diplomatic Decision-Makingcited by 63
  8. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 53
  9. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  10. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 46
  11. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  12. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agentscited by 28
  13. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  14. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systemscited by 26
  15. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMscited by 24
  16. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  17. 2024Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulationscited by 21
  18. 2019A Bayesian Approach to In-Game Win Probability in Soccercited by 9
  19. 2024OpenSkill: A faster asymmetric multi-team, multiplayer rating systemcited by 9
  20. 2026AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crisescited by 7
  21. 2024Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Controlcited by 4
  22. 2025Vox Deorum: A Hybrid LLM Architecture for 4X / Grand Strategy Game AI -- Lessons from Civilization Vcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulationcited by 1
  2. 2026Symbolic Reasoning Frameworks Trigger Memory-Mediated Ecosystem Dynamics in Multi-Agent LLM Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments
  2. 2026CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds
  3. 2026CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
  4. 2026Choosing LLMs for Strategic Agent Interactions
  5. 2026LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo
  6. 2026RuleSmith: Multi-Agent LLMs for Automated Game Balancing
  7. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors
  8. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making