AI Must not be Fully Autonomous
Not on the map. Why?
Abstract (arXiv)
Autonomous Artificial Intelligence (AI) has many benefits. It also has many risks. In this work, we identify the 3 levels of autonomous AI. We are of the position that AI must not be fully autonomous because of the many risks, especially as artificial superintelligence (ASI) is speculated to be just decades away. Fully autonomous AI, which can develop its own objectives, is at level 3 and without responsible human oversight. However, responsible human oversight is crucial for mitigating the risks. To ague for our position, we discuss theories of autonomy, AI and agents. Then, we offer 12 distinct arguments and 6 counterarguments with rebuttals to the counterarguments. We also present 15 pieces of recent evidence of AI misaligned values and other risks in the appendix.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 103 references that are in the map. First posted on arXiv 31 Jul 2025.
Builds on
21 of 103 references.
References · 21
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
- 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
- 2024Alignment faking in large language modelscited by 360
- 2024Frontier Models are Capable of In-context Schemingcited by 287
- 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
- 2018Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbationcited by 134
- 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
- 2025Fully Autonomous AI Agents Should Not be Developedcited by 45
- 2020Too many cooks: Bayesian inference for coordinating multi-agent collaborationcited by 29
- 2022AI Autonomy : Self-Initiated Open-World Continual Learning and Adaptationcited by 23
- 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 21
9 more
- 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
- 2024The Social Impact of Generative AI: An Analysis on ChatGPTcited by 18
- 2024On the Limitations of Large Language Models (LLMs): False Attributioncited by 13
- 2024Generative AI and Teachers -- For Us or Against Us? A Case Studycited by 6
- 2025HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Modelscited by 6
- 2023Bipol: A Novel Multi-Axes Bias Evaluation Metric with Explainability for NLPcited by 3
- 2025MalGEN: A Testbed for Modeling and Evaluating Malware Behaviorscited by 3
- 2025Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learningcited by 3
- 2018Truly Autonomous Machines Are Ethicalcited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.