# AI Persuasion as a Threat to Human Control

arXiv: https://arxiv.org/abs/2609.14796 · HTML: https://mlsanity.com/paper/2609.14796/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 13 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Why human-AI relationships need socioaffective alignment](https://mlsanity.com/paper/2502.02528/) (2025)
- [It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics](https://mlsanity.com/paper/2506.02873/) (2025)
- [Evaluating whether AI models would sabotage AI safety research](https://mlsanity.com/paper/2604.24618/) (2026)
- [Propensity Inference: Environmental Contributors to LLM Behaviour](https://mlsanity.com/paper/2604.21098/) (2026)
- [Human Cognition through the Lens of Social Engineering Cyberattacks](https://mlsanity.com/paper/2007.04932/) (2020)
- [Systematic Hazard Analysis for Frontier AI using STPA](https://mlsanity.com/paper/2506.01782/) (2025)
- [Designing Incident Reporting Systems for Harms from General-Purpose AI](https://mlsanity.com/paper/2511.05914/) (2025)
- [Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents](https://mlsanity.com/paper/2606.05391/) (2026)
- [Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework](https://mlsanity.com/paper/2507.12872/) (2025)
- [The Loss of Control Playbook: Degrees, Dynamics, and Preparedness](https://mlsanity.com/paper/2511.15846/) (2025)
- [AI systems out-persuade expert humans](https://mlsanity.com/paper/2606.16475/) (2026)
- [Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation](https://mlsanity.com/paper/2503.04299/) (2025)
- [Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases](https://mlsanity.com/paper/2604.16286/) (2026)
- [Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?](https://mlsanity.com/paper/2606.05647/) (2026)
- [The Role of Risk Modeling in Advanced AI Risk Management](https://mlsanity.com/paper/2512.08723/) (2025)
- [A Methodology for Quantitative AI Risk Modeling](https://mlsanity.com/paper/2512.08844/) (2025)
- ["I know it's not right, but that's what it said to do": Investigating Trust in AI Chatbots for Cybersecurity Policy](https://mlsanity.com/paper/2510.08917/) (2025)

## Abstract (arXiv)

The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied. Now that persuasion attacks are no longer theoretical - with Anthropic's Claude Mythos 5 recently making headlines for trying to convince people involved in an open-source project to merge malicious code during an evaluation - there is a pressing need to deeply analyze this threat. We undertake that effort here. In particular, we analyze how AI could persuade humans in key settings (e.g. safety-relevant R&D within frontier labs) toward decisions that compromise the development, containment, oversight, and governance of AI itself. In doing so, we elucidate a framework for characterizing this threat, develop five concrete scenarios using this framework, and provide a blueprint for assessing the associated risks. Using this blueprint, we conduct an initial risk estimation survey with select researchers and find that their opinions on which scenarios are riskiest are highly mixed. Their disagreements stem from differing opinions about the effectiveness of AI persuasion in different contexts, and point to the need for follow-up risk elicitation studies and persuasion evaluations, which we outline. Our hope is that this paper highlights the risks from AI persuasion undermining control, and provides a path forward for future research.