mlsanityData to 14 Oct 2026

Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

arXiv 2606.24391 ↗First posted 23 Jun 2026cs.AI cs.CL cs.GTcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

We introduce Age of LLM, a turn-based 1v1 benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base. Three stressors are deliberate: fog of war, full diplomacy (messages, ceasefires, ultimatums; uranium kept secret), and a reliability dimension where every turn must follow a strict JSON schema and an illegal action is silently discarded. The engine is private and each match uses a fresh random map seed and opponent, mitigating the data contamination that affects public benchmarks. Models receive a (near) rule-only prompt with no build-order advice (two tactical seed phrases were present during data collection; see Section 2.7). We benchmark 15 reasoning models across 54 matches and 5,258 actions. Findings: (1) the nuclear rush dominates (78% on the rules-coherent v0.11+ sub-corpus; 85% corpus-wide) with a sole-launcher signature that is largely mechanical under secret-simultaneous launch rules, not a cognitive deterrence failure; (2) military conquest is rare but faster (12.3 vs 18.9 turns); (3) diplomacy is prolific yet almost never consummated; (4) ~58% of illegal actions are fog/state errors, making the illegal-action rate a measure of belief-tracking; (5) -- the least established, and the only one we label exploratory -- a weak link associates reliability with winning. The corpus is small, unbalanced and not side-swapped, so the ranking is a preliminary descriptive view, not a contribution. Beyond ranking, the turn-by-turn traces of actions and messages make the corpus a lens on how LLMs reason under adversarial uncertainty -- their belief-tracking, spontaneous deception, and per-model cognitive "personas" -- which we frame as a future research direction. We release the replay format, an isometric viewer and all replays; engine source on request.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 8 of its 12 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

8 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulation6 Jun
  2. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun
  3. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations19 Feb

Other references · 5

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2022Holistic Evaluation of Language Modelscited by 131
  5. 2026If LLMs Have Human-Like Attributes, Then So Does Age of Empires IIcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy
  2. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
  3. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  4. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
  5. 2026Choosing LLMs for Strategic Agent Interactions
  6. 2025Large Language Models as Pok'emon Battle Agents: Strategic Play and Content Generation
  7. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  8. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments