mlsanityData to 14 Oct 2026

Levels of Autonomy for AI Agents

arXiv 2506.12469 ↗First posted 14 Jun 2025cs.HC cs.AIcited by 48

Not on the map. Why?

Cited by 26 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomy is a double-edged sword for AI agents, simultaneously unlocking transformative possibilities and serious risks. How can agent developers calibrate the appropriate levels of autonomy at which their agents should operate? We argue that an agent's level of autonomy can be treated as a deliberate design decision, separate from its capability and operational environment. In this work, we define five levels of escalating agent autonomy, characterized by the roles a user can take when interacting with an agent: operator, collaborator, consultant, approver, and observer. Within each level, we describe the ways by which a user can exert control over the agent and open questions for how to design the nature of user-agent interaction. We then highlight a potential application of our framework towards AI autonomy certificates to govern agent behavior in single- and multi-agent systems. We conclude by proposing early ideas for evaluating agents' autonomy. Our work aims to contribute meaningful, practical steps towards responsibly deployed and useful AI agents in the real world.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

27 of 43 references.

References · 27

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  7. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  8. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  9. 2024AI Agents That Mattercited by 219
  10. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  11. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  12. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
15 more
  1. 2024Building Machines that Learn and Think with Peoplecited by 73
  2. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
  3. 2025Authenticated Delegation and Authorized AI Agentscited by 65
  4. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  5. 2025Gradual Disempowerment: Systemic Existential Risks from Incremental AI Developmentcited by 46
  6. 2025Fully Autonomous AI Agents Should Not be Developedcited by 45
  7. 2024IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedbackcited by 42
  8. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 39
  9. 2025A sketch of an AI control safety casecited by 39
  10. 2024Safety case template for frontier AI: A cyber inability argumentcited by 19
  11. 2024Safety cases for frontier AIcited by 18
  12. 2023How does HCI Understand Human Autonomy and Agency?cited by 14
  13. 2025Measuring AI agent autonomy: Towards a scalable approach with code inspectioncited by 10
  14. 2024Language Models as Critical Thinking Tools: A Case Study of Philosopherscited by 6
  15. 2024Voice-Enabled AI Agents can Perform Common Scamscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 40.

48 papers.

Sort:

By area:

  1. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 48
  2. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 36
  3. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 30
  4. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 18
  5. 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 16
  6. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  7. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  8. 2026Mapping the Design Space of User Experience for Computer Use Agentscited by 7
  9. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
  10. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 6
  11. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 6
  12. 2025Interaction as Intelligence: Deep Research With Human-AI Partnershipcited by 5
36 more
  1. 2026AI Agents Push Humans Out of the Loopcited by 5
  2. 2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Taskscited by 4
  3. 2025On the Regulatory Potential of User Interfaces for AI Agent Governancecited by 3
  4. 2026Overseeing Agents Without Constant Oversight: Challenges and Opportunitiescited by 3
  5. 2026Autonomy and Agency in Agentic AI: Architectural Tactics for Regulated Contextscited by 3
  6. 2026Janus: a Playground for User-Involved Agentic Permission Managementcited by 3
  7. 2026Human Tool: An MCP-Style Framework for Human-Agent Collaborationcited by 2
  8. 2026Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insightcited by 2
  9. 2026Herding CATs: ALARA for Agent Harness Engineering in Portable Composable Multi-Agent Teamscited by 2
  10. 2026Post-AGI Economies: Autonomy and the First Fundamental Theorem of Welfare Economicscited by 2
  11. 2026How to Steer Your Multi-Agent System: Human-LLM Collaborative Planningcited by 2
  12. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  13. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
  14. 2025Regulating the Agency of LLM-based Agentscited by 1
  15. 2025Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agentscited by 1
  16. 2026"I Didn't Make the Micro Decisions": Measuring, Inducing, and Exposing Goal-Level AI Contributions in Collaborationcited by 1
  17. 2026The Digital Apprentice: A Framework for Human-Directed Agentic AI Developmentcited by 1
  18. 2026Managed Autonomy at Runtime: Gear-Based Safety and Governance for Single- and Multi-Agent Cyber-Physical Systemscited by 1
  19. 2025Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applicationscited by 0
  20. 2026SCSimulator: An Exploratory Visual Analytics Framework for Partner Selection in Supply Chains through LLM-driven Multi-Agent Simulationcited by 0
  21. 2026Toward Agentic Software Project Management: A Vision and Roadmapcited by 0
  22. 2026Rethinking Health Agents: From Siloed AI to Collaborative Decision Mediatorscited by 0
  23. 2026The Self Driving Portfolio: Agentic Architecture for Institutional Asset Managementcited by 0
  24. 2026The Inverse-Wisdom Law: Architectural Tribalism and the Consensus Paradox in Agentic Swarmscited by 0
  25. 2026From Tool to Agent: How Worker Needs Reorganize Across the Agentic Roles of Workplace AIcited by 0
  26. 2026Precautionary Governance of Autonomous AI: Legal Personhood as Functional Instrumentcited by 0
  27. 2026Consent Chain Degradation in Embodied Multi-Agent Systems: Bridging the Gap Between AI Agent Governance and Robot Ethicscited by 0
  28. 2026Agentic Literacy Debt: A Structural Problem the AI Literacy Field Has Not Yet Namedcited by 0
  29. 2026Do we have the knowledge we need? Rethinking human-AI decision-making in corporationscited by 0
  30. 2026SCAN: A Decision-Making Framework for Effective Task Allocation with Generative AIcited by 0
  31. 2026The Autonomous Agency Scale: A Behavioral Framework for Measuring Self-Directed Behavior in AI Systemscited by 0
  32. 2026Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarkscited by 0
  33. 2026Compliant with Local Controls, Collectively Discriminatory. A Governance Architecture for Multi-Agent AI in Regulated Financecited by 0
  34. 2026Designing the Future of User Feedback for Generative AIcited by 0
  35. 2026Ream: Unfolding Mutual Awareness in Human-Agent Workspacescited by 0
  36. 2026Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Responsecited by 0