# AI Must not be Fully Autonomous

arXiv: https://arxiv.org/abs/2507.23330 · HTML: https://mlsanity.com/paper/2507.23330/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 103 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 21 of 21)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation](https://mlsanity.com/paper/1808.10307/) (2018)
- [The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems](https://mlsanity.com/paper/2503.03750/) (2025)
- [Fully Autonomous AI Agents Should Not be Developed](https://mlsanity.com/paper/2502.02649/) (2025)
- [Too many cooks: Bayesian inference for coordinating multi-agent collaboration](https://mlsanity.com/paper/2003.11778/) (2020)
- [AI Autonomy : Self-Initiated Open-World Continual Learning and Adaptation](https://mlsanity.com/paper/2203.08994/) (2022)
- [Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.22777/) (2025)
- [Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests](https://mlsanity.com/paper/2507.05619/) (2025)
- [The Social Impact of Generative AI: An Analysis on ChatGPT](https://mlsanity.com/paper/2403.04667/) (2024)
- [On the Limitations of Large Language Models (LLMs): False Attribution](https://mlsanity.com/paper/2404.04631/) (2024)
- [Generative AI and Teachers -- For Us or Against Us? A Case Study](https://mlsanity.com/paper/2404.03486/) (2024)
- [HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models](https://mlsanity.com/paper/2506.15065/) (2025)
- [Bipol: A Novel Multi-Axes Bias Evaluation Metric with Explainability for NLP](https://mlsanity.com/paper/2304.04029/) (2023)
- [MalGEN: A Testbed for Modeling and Evaluating Malware Behaviors](https://mlsanity.com/paper/2506.07586/) (2025)
- [Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning](https://mlsanity.com/paper/2507.12079/) (2025)
- [Truly Autonomous Machines Are Ethical](https://mlsanity.com/paper/1812.02217/) (2018)

## Built on by (most cited first, 3 of 3)

- [From the Rock Floor to the Cloud: A Systematic Survey of State-of-the-Art NLP in Battery Life Cycle](https://mlsanity.com/paper/2510.27369/) (2025)
- [BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task](https://mlsanity.com/paper/2604.26986/) (2026)
- [Counterargument for Critical Thinking as Judged by AI and Humans](https://mlsanity.com/paper/2605.05353/) (2026)

## Abstract (arXiv)

Autonomous Artificial Intelligence (AI) has many benefits. It also has many risks. In this work, we identify the 3 levels of autonomous AI. We are of the position that AI must not be fully autonomous because of the many risks, especially as artificial superintelligence (ASI) is speculated to be just decades away. Fully autonomous AI, which can develop its own objectives, is at level 3 and without responsible human oversight. However, responsible human oversight is crucial for mitigating the risks. To ague for our position, we discuss theories of autonomy, AI and agents. Then, we offer 12 distinct arguments and 6 counterarguments with rebuttals to the counterarguments. We also present 15 pieces of recent evidence of AI misaligned values and other risks in the appendix.