Understanding Cognition-Induced Risks in Agentic AI Systems
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Frontier agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition. As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied. To address this gap, we systematically analyze risks induced by expanding cognitive capabilities, following a three-level framework defined by their cognitive scope, from physical cognition to social cognition, and finally to self-referential cognition. We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level. We finally propose strategies to mitigate these risks and enhance the controllability of agentic AI systems, ensuring their long-term safe development.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 11 of its 25 references that are in the map. First posted on arXiv 15 Aug 2026.
Builds on
11 of 25 references.
References · 11
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2024Alignment faking in large language modelscited by 360
- 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
- 2023The Science of Detecting LLM-Generated Textscited by 128
- 2026Emergent Introspective Awareness in Large Language Modelscited by 82
- 2025How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Studycited by 22
- 2025Large Language Models Report Subjective Experience Under Self-Referential Processingcited by 19
- 2024Frontier AI systems have surpassed the self-replicating red linecited by 17
- 2025The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systemscited by 3
- 2025AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Normscited by 1
- 2026Metacognition in LLMs: Foundations, Progress, and Opportunitiescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Agentic AI Needs a Systems Theory
- 2026Questionnaire Responses Do not Capture the Safety of AI Agents
- 2026A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI
- 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
- 2026Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agents
- 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
- 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
- 2026The Path Ahead for Agentic AI: Challenges and Opportunities