# Fully Autonomous AI Agents Should Not be Developed

arXiv: https://arxiv.org/abs/2502.02649 · HTML: https://mlsanity.com/paper/2502.02649/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 201 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 30 of 33)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning](https://mlsanity.com/paper/2406.11896/) (2024)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [Levels of AGI for Operationalizing Progress on the Path to AGI](https://mlsanity.com/paper/2311.02462/) (2023)
- [AirGapAgent: Protecting Privacy-Conscious Conversational Agents](https://mlsanity.com/paper/2405.05175/) (2024)
- [AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents](https://mlsanity.com/paper/2401.12963/) (2024)
- [Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance](https://mlsanity.com/paper/2410.12361/) (2024)
- [The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)](https://mlsanity.com/paper/2403.14473/) (2024)
- [Security of AI Agents](https://mlsanity.com/paper/2406.08689/) (2024)
- [AI Autonomy : Self-Initiated Open-World Continual Learning and Adaptation](https://mlsanity.com/paper/2203.08994/) (2022)
- [Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic](https://mlsanity.com/paper/2202.00881/) (2022)
- [AI-Powered Autonomous Weapons Risk Geopolitical Instability and Threaten AI Research](https://mlsanity.com/paper/2405.01859/) (2024)
- [Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment](https://mlsanity.com/paper/2406.04231/) (2024)
- [Non-Deterministic Policies in Markovian Decision Processes](https://mlsanity.com/paper/1401.3871/) (2014)
- [Good Parenting is all you need -- Multi-agentic LLM Hallucination Mitigation](https://mlsanity.com/paper/2410.14262/) (2024)
- [Catalyzing Equity in STEM Teams: Harnessing Generative AI for Inclusion and Diversity](https://mlsanity.com/paper/2402.00037/) (2024)

## Built on by (most cited first, 30 of 45)

- [Levels of Autonomy for AI Agents](https://mlsanity.com/paper/2506.12469/) (2025)
- [Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce](https://mlsanity.com/paper/2506.06576/) (2025)
- [Characterizing AI Agents for Alignment and Governance](https://mlsanity.com/paper/2504.21848/) (2025)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [Security Concerns for Large Language Models: A Survey](https://mlsanity.com/paper/2505.18889/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [Agentic AI Needs a Systems Theory](https://mlsanity.com/paper/2503.00237/) (2025)
- [Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective](https://mlsanity.com/paper/2504.03255/) (2025)
- [Generating Symbolic World Models via Test-time Scaling of Large Language Models](https://mlsanity.com/paper/2502.04728/) (2025)
- [A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy](https://mlsanity.com/paper/2506.09420/) (2025)
- [Build Agent Advocates, Not Platform Agents](https://mlsanity.com/paper/2505.04345/) (2025)
- [Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines](https://mlsanity.com/paper/2505.00875/) (2025)
- [How are AI agents used? Evidence from 177,000 MCP tools](https://mlsanity.com/paper/2603.23802/) (2026)
- [From surveillance to signalling: escalation channels as environmental controls for agentic AI](https://mlsanity.com/paper/2510.05192/) (2025)
- [Completion neq Collaboration: Scaling Collaborative Effort with Agents](https://mlsanity.com/paper/2510.25744/) (2025)
- [Modeling Distinct Human Interaction in Web Agents](https://mlsanity.com/paper/2602.17588/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [AI Agents Push Humans Out of the Loop](https://mlsanity.com/paper/2608.23642/) (2026)
- [When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration](https://mlsanity.com/paper/2506.05579/) (2025)
- [Personalizing Agent Privacy Decisions via Logical Entailment](https://mlsanity.com/paper/2512.05065/) (2025)
- [AI Must not be Fully Autonomous](https://mlsanity.com/paper/2507.23330/) (2025)
- [Overseeing Agents Without Constant Oversight: Challenges and Opportunities](https://mlsanity.com/paper/2602.16844/) (2026)
- [Autonomy and Agency in Agentic AI: Architectural Tactics for Regulated Contexts](https://mlsanity.com/paper/2605.12105/) (2026)
- [SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning](https://mlsanity.com/paper/2606.01991/) (2026)
- [Counterfactual-based Agent Influence Ranker for Agentic AI Workflows](https://mlsanity.com/paper/2510.25612/) (2025)
- [Reliable agent engineering should integrate machine-compatible organizational principles](https://mlsanity.com/paper/2512.07665/) (2025)
- [I'm Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk](https://mlsanity.com/paper/2504.00012/) (2025)
- [OpenAI's GPT-OSS-20B Model and Safety Alignment Issues in a Low-Resource Language](https://mlsanity.com/paper/2510.01266/) (2025)
- [A Criminology of Machines](https://mlsanity.com/paper/2511.02895/) (2025)
- [Semantic Deception: When Reasoning Models Can't Compute an Addition](https://mlsanity.com/paper/2512.20812/) (2025)

## Abstract (arXiv)

This paper argues that fully autonomous AI agents should not be developed. In support of this position, we build from prior scientific literature and current product marketing to delineate different AI agent levels and detail the ethical values at play in each, documenting trade-offs in potential benefits and risks. Our analysis reveals that risks to people increase with the autonomy of a system: The more control a user cedes to an AI agent, the more risks to people arise. Particularly concerning are safety risks, which affect human life and impact further values.