# Evaluating Frontier Models for Stealth and Situational Awareness

arXiv: https://arxiv.org/abs/2505.01420 · HTML: https://mlsanity.com/paper/2505.01420/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 2 May 2025.

## Builds on (most cited first, 25 of 25)

- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [An Approach to Technical AGI Safety and Security](https://mlsanity.com/paper/2504.01849/) (2025)
- [A sketch of an AI control safety case](https://mlsanity.com/paper/2501.17315/) (2025)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats](https://mlsanity.com/paper/2411.17693/) (2024)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [Safety case template for frontier AI: A cyber inability argument](https://mlsanity.com/paper/2411.08088/) (2024)
- [Safety cases for frontier AI](https://mlsanity.com/paper/2410.21572/) (2024)
- [Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?](https://mlsanity.com/paper/2412.12480/) (2024)
- [MISR: Measuring Instrumental Self-Reasoning in Frontier Models](https://mlsanity.com/paper/2412.03904/) (2024)

## Built on by (most cited first, 30 of 37)

- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives](https://mlsanity.com/paper/2511.06626/) (2025)
- [When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents](https://mlsanity.com/paper/2602.08235/) (2026)
- [Evaluating and Understanding Scheming Propensity in LLM Agents](https://mlsanity.com/paper/2603.01608/) (2026)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [Evidence for Limited Metacognition in LLMs](https://mlsanity.com/paper/2509.21545/) (2025)
- [Evaluating Control Protocols for Untrusted AI Agents](https://mlsanity.com/paper/2511.02997/) (2025)
- [Benchmarking Misuse Mitigation Against Covert Adversaries](https://mlsanity.com/paper/2506.06414/) (2025)
- [Practical challenges of control monitoring in frontier AI deployments](https://mlsanity.com/paper/2512.22154/) (2025)
- [AI Testing Should Account for Sophisticated Strategic Behaviour](https://mlsanity.com/paper/2508.14927/) (2025)
- [Optimizing AI Agent Attacks With Synthetic Data](https://mlsanity.com/paper/2511.02823/) (2025)
- [LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness](https://mlsanity.com/paper/2605.26438/) (2026)
- [Training Deliberative Monitors for Black-Box Scheming Detection](https://mlsanity.com/paper/2605.29601/) (2026)
- [Gram: Assessing sabotage propensities via automated alignment auditing](https://mlsanity.com/paper/2605.30322/) (2026)
- [The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy](https://mlsanity.com/paper/2510.26752/) (2025)
- [Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety](https://mlsanity.com/paper/2605.22643/) (2026)
- [Decomposing and Measuring Evaluation Awareness](https://mlsanity.com/paper/2605.23055/) (2026)
- [Measuring Reward-Seeking via Contrastive Belief Updates](https://mlsanity.com/paper/2607.18966/) (2026)
- [Sequential Bayesian Evaluation of Large Language Model Behavior](https://mlsanity.com/paper/2511.10661/) (2025)
- [Tatemae: Detecting Alignment Faking via Tool Selection in LLMs](https://mlsanity.com/paper/2604.26511/) (2026)
- [PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users](https://mlsanity.com/paper/2605.13307/) (2026)
- [The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages](https://mlsanity.com/paper/2605.27901/) (2026)
- [Realistic honeypot evaluations for scheming propensity](https://mlsanity.com/paper/2605.29729/) (2026)
- [Muse Spark Safety & Preparedness Report](https://mlsanity.com/paper/2606.12429/) (2026)
- [How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift](https://mlsanity.com/paper/2607.22676/) (2026)
- [The Lock-In Phase Hypothesis: Identity Consolidation as a Precursor to AGI](https://mlsanity.com/paper/2510.20190/) (2025)
- [An Aristotelian ontology of instrumental goals: Structural features to be managed and not failures to be eliminated](https://mlsanity.com/paper/2510.25471/) (2025)
- [Password-Activated Shutdown Protocols for Misaligned Frontier Agents](https://mlsanity.com/paper/2512.03089/) (2025)

## Abstract (arXiv)

Recent work has demonstrated the plausibility of frontier AI models scheming -- knowingly and covertly pursuing an objective misaligned with its developer's intentions. Such behavior could be very hard to detect, and if present in future advanced systems, could pose severe loss of control risk. It is therefore important for AI developers to rule out harm from scheming prior to model deployment. In this paper, we present a suite of scheming reasoning evaluations measuring two types of reasoning capabilities that we believe are prerequisites for successful scheming: First, we propose five evaluations of ability to reason about and circumvent oversight (stealth). Second, we present eleven evaluations for measuring a model's ability to instrumentally reason about itself, its environment and its deployment (situational awareness). We demonstrate how these evaluations can be used as part of a scheming inability safety case: a model that does not succeed on these evaluations is almost certainly incapable of causing severe harm via scheming in real deployment. We run our evaluations on current frontier models and find that none of them show concerning levels of either situational awareness or stealth.