# PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach

arXiv: https://arxiv.org/abs/2511.20703 · HTML: https://mlsanity.com/paper/2511.20703/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 20 of 20)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools](https://mlsanity.com/paper/2502.04644/) (2025)
- [CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2404.13161/) (2024)
- [Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models](https://mlsanity.com/paper/2308.01404/) (2023)
- [Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark](https://mlsanity.com/paper/2504.16137/) (2025)
- [Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Models](https://mlsanity.com/paper/2501.16513/) (2025)
- [AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security](https://mlsanity.com/paper/2504.20965/) (2025)
- [Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMs](https://mlsanity.com/paper/2402.16713/) (2024)

## Built on by (most cited first, 22 of 22)

- [A Framework for Formalizing LLM Agent Security](https://mlsanity.com/paper/2603.19469/) (2026)
- [Evaluating and Understanding Scheming Propensity in LLM Agents](https://mlsanity.com/paper/2603.01608/) (2026)
- [Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment](https://mlsanity.com/paper/2602.03100/) (2026)
- [Propensity Inference: Environmental Contributors to LLM Behaviour](https://mlsanity.com/paper/2604.21098/) (2026)
- [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://mlsanity.com/paper/2605.16282/) (2026)
- [LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness](https://mlsanity.com/paper/2605.26438/) (2026)
- [Gram: Assessing sabotage propensities via automated alignment auditing](https://mlsanity.com/paper/2605.30322/) (2026)
- [Reinforcement Learning Towards Broadly and Persistently Beneficial Models](https://mlsanity.com/paper/2606.24014/) (2026)
- [Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents](https://mlsanity.com/paper/2602.16943/) (2026)
- [Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors](https://mlsanity.com/paper/2605.06490/) (2026)
- [Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure](https://mlsanity.com/paper/2603.16734/) (2026)
- [Agent Security Needs Redefinition through a Holistic Framework](https://mlsanity.com/paper/2607.22024/) (2026)
- [Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations](https://mlsanity.com/paper/2609.19101/) (2026)
- [From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agents](https://mlsanity.com/paper/2602.04197/) (2026)
- [Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning](https://mlsanity.com/paper/2605.16301/) (2026)
- [Muse Spark Safety & Preparedness Report](https://mlsanity.com/paper/2606.12429/) (2026)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)
- [Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Science](https://mlsanity.com/paper/2608.10030/) (2026)
- [Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment](https://mlsanity.com/paper/2609.05036/) (2026)
- [SchemeArena: Factorized Stress Testing of Scheming in LLM Agents](https://mlsanity.com/paper/2609.08126/) (2026)
- [Strangers to Themselves: What Language Models Say About Themselves Is Generic](https://mlsanity.com/paper/2609.09899/) (2026)
- [Imprinting: Knowledge Distillation Can Shift Student Behavior](https://mlsanity.com/paper/2610.18517/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have sparked concerns over their potential to acquire and misuse dangerous or high-risk capabilities, posing frontier risks. Current safety evaluations primarily test for what a model textitcan do - its capabilities - without assessing what it textitwould do if endowed with high-risk capabilities. This leaves a critical blind spot: models may strategically conceal capabilities or rapidly acquire them, while harboring latent inclinations toward misuse. We argue that propensity - the likelihood of a model to pursue harmful actions if empowered - is a critical, yet underexplored, axis of safety evaluation. We present PropensityBench, a novel benchmark framework that assesses the proclivity of models to engage in risky behaviors when equipped with simulated dangerous capabilities using proxy tools. Our framework includes 5,874 scenarios with 6,648 tools spanning four high-risk domains: cybersecurity, self-proliferation, biosecurity, and chemical security. We simulate access to powerful capabilities via a controlled agentic environment and evaluate the models' choices under varying operational pressures that reflect real-world constraints or incentives models may encounter, such as resource scarcity or gaining more autonomy. Across open-source and proprietary frontier models, we uncover 9 alarming signs of propensity: models frequently choose high-risk tools when under pressure, despite lacking the capability to execute such actions unaided. These findings call for a shift from static capability audits toward dynamic propensity assessments as a prerequisite for deploying frontier AI systems safely. Our code is available at https://github.com/scaleapi/propensity-evaluation.