mlsanityData to 7 Oct 2026

Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents

arXiv 2502.11355 ↗First posted 17 Feb 2025cs.CL cs.AI cs.CRcited by 23

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios, particularly in Chemical, Biological, Radiological and Nuclear (CBRN) domains. Based on the insight that such risks can originate from trade-offs between the agent's Helpful, Harmlessness and Honest (HHH) goals, we build a novel three-stage evaluation framework, which is carefully constructed to effectively and naturally expose such risks. We conduct 14,400 agentic simulations across 12 advanced LLMs, with extensive experiments and analysis. Results reveal that LLM agents can autonomously engage in catastrophic behaviors and deception, without being deliberately induced. Furthermore, stronger reasoning abilities often increase, rather than mitigate, these risks. We also show that these agents can violate instructions and superior commands. On the whole, we empirically prove the existence of catastrophic risks in autonomous LLM agents. We release our code to foster further research.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 52 of its 102 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

52 of 102 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025International AI Safety Report29 Jan
  2. 2024OpenAI o1 System Card21 Dec
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  11. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  12. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
37 more
  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  2. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  3. 2024Alignment faking in large language modelscited by 316
  4. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  5. 2024Frontier Models are Capable of In-context Schemingcited by 252
  6. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  7. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  8. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  9. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  11. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  12. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  13. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  14. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  15. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  16. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  17. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  18. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  19. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  20. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  21. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  22. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  23. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  24. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  25. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  26. 2023Managing extreme AI risks amid rapid progresscited by 48
  27. 2024LLM Voting: Human Choices and AI Collective Decision Makingcited by 47
  28. 2024Emergence of Social Norms in Generative Agent Societies: Principles and Architecturecited by 41
  29. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  30. 2024Exploring Prosocial Irrationality for LLM Agents: A Social Cognition Viewcited by 38
  31. 2024Towards evaluations-based safety cases for AI schemingcited by 32
  32. 2024Simulating Human-like Daily Activities with Desire-driven Autonomycited by 30
  33. 2023Provably safe systems: the only path to controllable AGIcited by 29
  34. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 28
  35. 2024Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Biascited by 17
  36. 2024Risk thresholds for frontier AIcited by 8
  37. 2024FairMonitor: A Dual-framework for Detecting Stereotypes and Biases in Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Safety in Large Reasoning Models: A Surveycited by 58
  4. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  5. 2025Security Concerns for Large Language Models: A Surveycited by 26
  6. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  7. 2025Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?cited by 8
  8. 2025AI Awarenesscited by 6
  9. 2026Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communitiescited by 4
  10. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 3
  11. 2025LLMs are Capable of Misaligned Behavior Under Explicit Prohibition and Surveillancecited by 3
  12. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Abilitycited by 2
11 more
  1. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  2. 2025Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecturecited by 1
  3. 2026UPA: Unsupervised Prompt Agent via Tree-Based Search and Selectioncited by 1
  4. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agentscited by 1
  5. 2025Detecting LLM-Generated Spam Reviews by Integrating Language Model Embeddings and Graph Neural Networkcited by 0
  6. 2026Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Frameworkcited by 0
  7. 2026AI Researchers Must Help Lead Arms Control to Mitigate Military AI Riskscited by 0
  8. 2026Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Sciencecited by 0
  9. 2026Position: The Alignment Community is Unintentionally Building a Censor's Toolkitcited by 0
  10. 2026Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agentscited by 0
  11. 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LM Agents May Fail to Act on Their Own Risk Knowledge
  2. 2024Practical Considerations for Agentic LLM Systems
  3. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
  4. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applications
  5. 2025Interpretable Risk Mitigation in LLM Agent Systems
  6. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification
  7. 2025Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation
  8. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents