When AI Navigates the Fog of War
Not on the map. Why?
Abstract (arXiv)
Can AI reason about a war before its trajectory becomes historically obvious? Analyzing this capability is difficult because retrospective geopolitical prediction is heavily confounded by training-data leakage. We address this challenge through a temporally grounded case study of the early stages of the 2026 Middle East conflict, which unfolded after the training cutoff of current frontier models. We construct 11 critical temporal nodes, 42 node-specific verifiable questions, and 5 general exploratory questions, requiring models to reason only from information that would have been publicly available at each moment. This design substantially mitigates training-data leakage concerns, creating a setting well-suited for studying how models analyze an unfolding crisis under the fog of war, and provides, to our knowledge, the first temporally grounded analysis of LLM reasoning in an ongoing geopolitical conflict. Our analysis reveals three main findings. First, current state-of-the-art large language models often display a striking degree of strategic realism, reasoning beyond surface rhetoric toward deeper structural incentives. Second, this capability is uneven across domains: models are more reliable in economically and logistically structured settings than in politically ambiguous multi-actor environments. Finally, model narratives evolve over time, shifting from early expectations of rapid containment toward more systemic accounts of regional entrenchment and attritional de-escalation. Since the conflict remains ongoing at the time of writing, this work can serve as an archival snapshot of model reasoning during an unfolding geopolitical crisis, enabling future studies without the hindsight bias of retrospective analysis.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 17 Mar 2026.
Builds on
35 of 49 references.
References · 35
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
- 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
- 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
- 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
23 more
- 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
- 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
- 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
- 2024Approaching Human-Level Forecasting with Language Modelscited by 102
- 2023Can we trust the evaluation on ChatGPT?cited by 63
- 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Modelscited by 61
- 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 57
- 2023A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localizationcited by 50
- 2023Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluationcited by 47
- 2023An Open Source Data Contamination Report for Large Language Modelscited by 47
- 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
- 2025A Survey on Data Contamination for Large Language Modelscited by 33
- 2024Data Contamination Can Cross Language Barrierscited by 26
- 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
- 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
- 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
- 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
- 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
- 2025Benchmarking LLMs for Political Science: A United Nations Perspectivecited by 9
- 2025FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmarkcited by 8
- 2026Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoffcited by 6
- 2025CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoningcited by 4
- 2026ThinkTank-ME: A Multi-Expert Framework for Middle East Event Forecastingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026LLMs as Strategic Actors: Behavioral Alignment, Risk Calibration, and Argumentation Framing in Geopolitical Simulations
- 2024Escalation Risks from Language Models in Military and Diplomatic Decision-Making
- 2026Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperform
- 2026Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War
- 2025Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting
- 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars
- 2026AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises
- 2026Towards a Belief-Based World Model for LLM Agents