# The End of Trust: How Agentic AI Breaks Security Assumptions

arXiv: https://arxiv.org/abs/2605.16436 · HTML: https://mlsanity.com/paper/2605.16436/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 82 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 21 of 21)

- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [Deep Learning for Deepfakes Creation and Detection: A Survey](https://mlsanity.com/paper/1909.11573/) (2019)
- [Spear Phishing With Large Language Models](https://mlsanity.com/paper/2305.06972/) (2023)
- [Seeing is Living? Rethinking the Security of Facial Liveness Verification in the Deepfake Era](https://mlsanity.com/paper/2202.10673/) (2022)
- [ChatGPT: More than a Weapon of Mass Deception, Ethical challenges and responses from the Human-Centered Artificial Intelligence (HCAI) perspective](https://mlsanity.com/paper/2304.11215/) (2023)
- [Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects](https://mlsanity.com/paper/2412.00586/) (2024)
- [Personhood credentials: Artificial intelligence and the value of privacy-preserving tools to distinguish who is real online](https://mlsanity.com/paper/2408.07892/) (2024)
- [Defending Against Social Engineering Attacks in the Age of LLMs](https://mlsanity.com/paper/2406.12263/) (2024)
- [Watermarking Without Standards Is Not AI Governance](https://mlsanity.com/paper/2505.23814/) (2025)
- [Better Safe Than Sorry: An Adversarial Approach to Improve Social Bot Detection](https://mlsanity.com/paper/1904.05132/) (2019)
- [Topic-Based Watermarks for Large Language Models](https://mlsanity.com/paper/2404.02138/) (2024)
- [Content, Nudges and Incentives: A Study on the Effectiveness and Perception of Embedded Phishing Training](https://mlsanity.com/paper/2409.01378/) (2024)
- [Phishing in Organizations: Findings from a Large-Scale and Long-Term Study](https://mlsanity.com/paper/2112.07498/) (2021)
- [The Cat and Mouse Game: The Ongoing Arms Race Between Diffusion Models and Detection Methods](https://mlsanity.com/paper/2410.18866/) (2024)

## Abstract (arXiv)

For decades, the security of digital interaction has rested on an unacknowledged economic constraint. Attackers faced a tradeoff between the fidelity of a deception and the scale at which it could be deployed. Convincing impersonation required sustained human effort and was confined to a narrow set of high-value targets, while mass-market attacks sacrificed plausibility for reach. Detection systems, verification mechanisms, and user awareness training have all been implicitly calibrated to the artifacts of cheap deception that this tradeoff produced. Agentic AI collapses the tradeoff, allowing high-fidelity, individually tailored deception to be produced at mass-market scale. We argue that this shift exhausts a security paradigm rather than merely intensifying the threat landscape. We introduce the Infinite Impostor, an attack model in which an autonomous agent interposes itself between two parties who already trust each other, hijacking an existing relationship rather than building a new one from scratch. Detection-oriented defenses share an assumption that generative progress is eliminating, that synthetic outputs are distinguishable from authentic ones. We propose a suspect-by-default paradigm that shifts security from authenticating actors to evaluating actions, and examine the governance tensions that arise when platforms become the regulatory substrate of digital interaction.