mlsanityData to 7 Oct 2026

AI Agents Should be Regulated Based on the Extent of Their Autonomous Operations

arXiv 2503.04750 ↗First posted 7 Feb 2025cs.CY cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

This position paper argues that AI agents should be regulated by the extent to which they operate autonomously. AI agents with long-term planning and strategic capabilities can pose significant risks of human extinction and irreversible global catastrophes. While existing regulations often focus on computational scale as a proxy for potential harm, we argue that such measures are insufficient for assessing the risks posed by agents whose capabilities arise primarily from inference-time computation. To support our position, we discuss relevant regulations and recommendations from scientists regarding existential risks, as well as the advantages of using action sequences -- which reflect the degree of an agent's autonomy -- as a more suitable measure of potential impact than existing metrics that rely on observing environmental states.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 56 of its 123 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

56 of 123 references.

References · 56

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
38 more
  1. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  6. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  8. 2023Large Language Modelscited by 774
  9. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  10. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  11. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  13. 2020Artificial Intelligence, Values and Alignmentcited by 443
  14. 2023Cognitive Architectures for Language Agentscited by 422
  15. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  16. 2019Control Barrier Functions: Theory and Applicationscited by 332
  17. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  18. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  19. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  20. 2024A Survey on Data Selection for Language Modelscited by 230
  21. 2023An Overview of Catastrophic AI Riskscited by 214
  22. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  23. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  24. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  25. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  26. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  27. 2019Optimal Policies Tend to Seek Powercited by 115
  28. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  29. 2016The Off-Switch Gamecited by 97
  30. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  31. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  32. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  33. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 51
  34. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  35. 2024Open Problems in Technical AI Governancecited by 45
  36. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  37. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  38. 2019Conservative Agency via Attainable Utility Preservationcited by 35

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  2. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0