# Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations

arXiv: https://arxiv.org/abs/2403.03407 · HTML: https://mlsanity.com/paper/2403.03407/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 56 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Escalation Risks from Language Models in Military and Diplomatic Decision-Making](https://mlsanity.com/paper/2401.03408/) (2024)
- [The Fourth International Verification of Neural Networks Competition (VNN-COMP 2023): Summary and Results](https://mlsanity.com/paper/2312.16760/) (2023)
- [Student of Games: A unified learning algorithm for both perfect and imperfect information games](https://mlsanity.com/paper/2112.03178/) (2021)
- [AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy](https://mlsanity.com/paper/2402.07862/) (2024)

## Built on by (most cited first, 21 of 21)

- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation](https://mlsanity.com/paper/2406.11852/) (2024)
- [Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations](https://mlsanity.com/paper/2410.13204/) (2024)
- [GPT for Games: An Updated Scoping Review (2020-2024)](https://mlsanity.com/paper/2411.00308/) (2024)
- [AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises](https://mlsanity.com/paper/2602.14740/) (2026)
- [Markovian Transformers for Informative Language Modeling](https://mlsanity.com/paper/2404.18988/) (2024)
- [Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgment](https://mlsanity.com/paper/2503.02976/) (2025)
- [The Prompt War: How AI Decides on a Military Intervention](https://mlsanity.com/paper/2507.06277/) (2025)
- [Scenarios in Computing Research: A Systematic Review of the Use of Scenario Methods for Exploring the Future of Computing Technologies in Society](https://mlsanity.com/paper/2506.05605/) (2025)
- [Managing Escalation in Off-the-Shelf Large Language Models](https://mlsanity.com/paper/2508.01056/) (2025)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V](https://mlsanity.com/paper/2604.07733/) (2026)
- [When Trust Collides: Decoding Human-LLM Cooperation Dynamics through the Prisoner's Dilemma](https://mlsanity.com/paper/2503.07320/) (2025)
- [Developing and Validating the Arabic Version of the Attitudes Toward Large Language Models Scale](https://mlsanity.com/paper/2510.13009/) (2025)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [The Foreign Policy AI Evaluation Gap](https://mlsanity.com/paper/2607.02955/) (2026)
- [No One Wins in Nuclear War: A Social Simulation of Military Decision-making](https://mlsanity.com/paper/2608.01868/) (2026)
- [Don't Want Your LLM to Recommend Nuclear Strike? Try Asking It in Japanese](https://mlsanity.com/paper/2608.12373/) (2026)
- [Language models judge war differently when tested for alignment](https://mlsanity.com/paper/2609.05009/) (2026)
- [Position: AI Is Not Ready for Strategic Conflicts](https://mlsanity.com/paper/2609.16189/) (2026)
- [MA-Bench: Measuring Operational Planning in Language Models](https://mlsanity.com/paper/2610.19138/) (2026)

## Abstract (arXiv)

To some, the advent of artificial intelligence (AI) promises better decision-making and increased military effectiveness while reducing the influence of human error and emotions. However, there is still debate about how AI systems, especially large language models (LLMs) that can be applied to many tasks, behave compared to humans in high-stakes military decision-making scenarios with the potential for increased risks towards escalation. To test this potential and scrutinize the use of LLMs for such purposes, we use a new wargame experiment with 214 national security experts designed to examine crisis escalation in a fictional U.S.-China scenario and compare the behavior of human player teams to LLM-simulated team responses in separate simulations. Here, we find that the LLM-simulated responses can be more aggressive and significantly affected by changes in the scenario. We show a considerable high-level agreement in the LLM and human responses and significant quantitative and qualitative differences in individual actions and strategic tendencies. These differences depend on intrinsic biases in LLMs regarding the appropriate level of violence following strategic instructions, the choice of LLM, and whether the LLMs are tasked to decide for a team of players directly or first to simulate dialog between a team of players. When simulating the dialog, the discussions lack quality and maintain a farcical harmony. The LLM simulations cannot account for human player characteristics, showing no significant difference even for extreme traits, such as "pacifist" or "aggressive sociopath." When probing behavioral consistency across individual moves of the simulation, the tested LLMs deviated from each other but generally showed somewhat consistent behavior. Our results motivate policymakers to be cautious before granting autonomy or following AI-based strategy recommendations.