mlsanityData to 14 Oct 2026

A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy

arXiv 2506.09420 ↗First posted 11 Jun 2025cs.AI cs.CL cs.HCcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent improvements in large language models (LLMs) have led many researchers to focus on building fully autonomous AI agents. This position paper questions whether this approach is the right path forward, as these autonomous systems still have problems with reliability, transparency, and understanding the actual requirements of human. We suggest a different approach: LLM-based Human-Agent Systems (LLM-HAS), where AI works with humans rather than replacing them. By keeping human involved to provide guidance, answer questions, and maintain control, these systems can be more trustworthy and adaptable. Looking at examples from healthcare, finance, and software development, we show how human-AI teamwork can handle complex tasks better than AI working alone. We also discuss the challenges of building these collaborative systems and offer practical solutions. This paper argues that progress in AI should not be measured by how independent systems become, but by how well they can work with humans. The most promising future for AI is not in systems that take over human roles, but in those that enhance human capabilities through meaningful partnership.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 84 of its 100 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

84 of 100 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent4 May
  2. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey1 May
  3. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review28 Apr
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges27 Mar
  5. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement20 Mar
  6. 2025A Survey on the Optimization of Large Language Model-based Agents16 Mar
  7. 2025Unified Mind Model: Reimagining Autonomous Agents in the LLM Era5 Mar
  8. 2025Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaboration17 Feb
  9. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration20 Dec
  10. 2024AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration18 Apr
  11. 2024Large Multimodal Agents: A Survey23 Feb
  12. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solving20 Feb
3 more
  1. 2024LLM Multi-Agent Systems: Challenges and Open Problems5 Feb
  2. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  3. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents5 Jun

Other references · 69

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  8. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  9. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  10. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  11. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  12. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
38 more
  1. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 113
  2. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 109
  3. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  4. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  5. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 97
  6. 2023Is RLHF More Difficult than Standard RL?cited by 96
  7. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 85
  8. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  9. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Taskscited by 82
  10. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  11. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
  12. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  13. 2025LLM Agents Making Agent Toolscited by 48
  14. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  15. 2025Fully Autonomous AI Agents Should Not be Developedcited by 45
  16. 2025Infrastructure for AI Agentscited by 41
  17. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 40
  18. 2024Moral Alignment for LLM Agentscited by 39
  19. 2024A Taxonomy for Human-LLM Interaction Modes: An Initial Explorationcited by 38
  20. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 32
  21. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  22. 2023LLM A*: Human in the Loop Large Language Models Enabled A* Search for Roboticscited by 26
  23. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  24. 2024Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?cited by 23
  25. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  26. 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
  27. 2025A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategycited by 20
  28. 2024Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attackscited by 19
  29. 2025Citations and Trust in LLM Generated Responsescited by 18
  30. 2023LLM-Based Human-Robot Collaboration Framework for Manipulation Taskscited by 17
  31. 2024Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teamscited by 16
  32. 2025ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environmentscited by 14
  33. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 13
  34. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 13
  35. 2025LLM-based Automated Grading with Human-in-the-Loopcited by 13
  36. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  37. 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedbackcited by 11
  38. 2024To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactionscited by 10

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 48
  2. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  3. 2026When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigationcited by 14
  4. 2025Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologiescited by 5
  5. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  6. 2026Overseeing Agents Without Constant Oversight: Challenges and Opportunitiescited by 3
  7. 2025MADIAVE: Multi-Agent Debate for Implicit Attribute Value Extractioncited by 2
  8. 2026Harnesses for Inference-Time Alignment over Execution Trajectoriescited by 2
  9. 2026The Boundaries of Automation: A Theory of Persistent Human Participationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents
  2. 2024Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Survey
  4. 2026Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agents
  5. 2025Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  7. 2026HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation
  8. 2025A Survey of AI Agent Protocols