# Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

arXiv: https://arxiv.org/abs/2503.06263 · HTML: https://mlsanity.com/paper/2503.06263/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 89 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Escalation Risks from Language Models in Military and Diplomatic Decision-Making](https://mlsanity.com/paper/2401.03408/) (2024)
- [GPT-4 Can't Reason](https://mlsanity.com/paper/2308.03762/) (2023)
- [Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters](https://mlsanity.com/paper/2312.17476/) (2023)
- [What Do NLP Researchers Believe? Results of the NLP Community Metasurvey](https://mlsanity.com/paper/2208.12852/) (2022)
- [AI-Powered Autonomous Weapons Risk Geopolitical Instability and Threaten AI Research](https://mlsanity.com/paper/2405.01859/) (2024)
- [Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations](https://mlsanity.com/paper/2410.13204/) (2024)
- [Open-Ended Wargames with Large Language Models](https://mlsanity.com/paper/2404.11446/) (2024)

## Built on by (most cited first, 7 of 7)

- [Military AI Needs Technically-Informed Regulation to Safeguard AI Research and its Applications](https://mlsanity.com/paper/2505.18371/) (2025)
- [The Prompt War: How AI Decides on a Military Intervention](https://mlsanity.com/paper/2507.06277/) (2025)
- [Managing Escalation in Off-the-Shelf Large Language Models](https://mlsanity.com/paper/2508.01056/) (2025)
- ["As Eastern Powers, I will veto." : An Investigation of Nation-level Bias of Large Language Models in International Relations](https://mlsanity.com/paper/2511.10695/) (2025)
- [Multi-Agent Strategic Games with LLMs](https://mlsanity.com/paper/2605.03604/) (2026)
- [Can AI Make Conflicts Worse? An Alignment Failure in LLM Deployment Across Conflict Contexts](https://mlsanity.com/paper/2605.22720/) (2026)
- [The Foreign Policy AI Evaluation Gap](https://mlsanity.com/paper/2607.02955/) (2026)

## Abstract (arXiv)

As national security institutions increasingly integrate Artificial Intelligence (AI) into decision-making and content generation processes, understanding the inherent biases of large language models (LLMs) is crucial. This study presents a novel benchmark designed to evaluate the biases and preferences of seven prominent foundation models-Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, GPT-4o, Gemini 1.5 Pro-002, Mixtral 8x22B, Claude 3.5 Sonnet, and Qwen2 72B-in the context of international relations (IR). We designed a bias discovery study around core topics in IR using 400-expert crafted scenarios to analyze results from our selected models. These scenarios focused on four topical domains including: military escalation, military and humanitarian intervention, cooperative behavior in the international system, and alliance dynamics. Our analysis reveals noteworthy variation among model recommendations based on scenarios designed for the four tested domains. Particularly, Qwen2 72B, Gemini 1.5 Pro-002 and Llama 3.1 8B Instruct models offered significantly more escalatory recommendations than Claude 3.5 Sonnet and GPT-4o models. All models exhibit some degree of country-specific biases, often recommending less escalatory and interventionist actions for China and Russia compared to the United States and the United Kingdom. These findings highlight the necessity for controlled deployment of LLMs in high-stakes environments, emphasizing the need for domain-specific evaluations and model fine-tuning to align with institutional objectives.