mlsanityData to 14 Oct 2026

LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey

arXiv 2505.00753 ↗First posted 1 May 2025cs.CL cs.LGcited by 29

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent advances in large language models (LLMs) have sparked growing interest in building fully autonomous agents. However, fully autonomous LLM-based agents still face significant challenges, including limited reliability due to hallucinations, difficulty in handling complex tasks, and substantial safety and ethical risks, all of which limit their feasibility and trustworthiness in real-world applications. To overcome these limitations, LLM-based human-agent systems (LLM-HAS) incorporate human-provided information, feedback, or control into the agent system to enhance system performance, reliability, and safety. These human-agent collaboration systems enable humans and LLM-based agents to collaborate effectively by leveraging their complementary strengths. This paper provides the first comprehensive and structured survey of LLM-HAS. It clarifies fundamental concepts, systematically presents core components shaping these systems, including environment and profiling, human feedback, interaction types, orchestration, and communication, explores emerging applications, and discusses unique challenges and opportunities arising from human-AI collaboration. By consolidating current knowledge and offering a structured overview, we aim to foster further research and innovation in this rapidly evolving interdisciplinary field. Paper lists and resources are available at https://github.com/HenryPengZou/Awesome-Human-Agent-Collaboration-Interaction-Systems.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 62 of its 127 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

62 of 127 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Stochastic, Dynamic, Fluid Autonomy in Agentic AI: Implications for Authorship, Inventorship, and Liability5 Apr
  2. 2025Human-Artificial Interaction in the Age of Agentic AI: A System-Theoretical Approach19 Feb
  3. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMs10 Jan
  4. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration20 Dec
  5. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback7 Oct
  6. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task13 Sep
  7. 2024Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations27 Aug
  8. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planning18 Jun
  9. 2024A Human-Computer Collaborative Tool for Training a Single Large Language Model Agent into a Network through Few Examples24 Apr
  10. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solving20 Feb
  11. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  12. 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination23 Dec
4 more
  1. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents18 Oct
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  3. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  4. 2023AdaPlanner: Adaptive Planning from Feedback with Language Models26 May

Other references · 46

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  4. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  5. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
  6. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  7. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  8. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  9. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  10. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  11. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  12. 2018Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Taskscited by 121
34 more
  1. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  2. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 97
  3. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 97
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  5. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 85
  6. 2019Personalizing Dialogue Agents via Meta-Learningcited by 83
  7. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 69
  8. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
  9. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  10. 2025A Survey on LLM-powered Agents for Recommender Systemscited by 50
  11. 2025Fully Autonomous AI Agents Should Not be Developedcited by 45
  12. 2021Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AIcited by 38
  13. 2024A Taxonomy for Human-LLM Interaction Modes: An Initial Explorationcited by 38
  14. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 32
  15. 2023LLM A*: Human in the Loop Large Language Models Enabled A* Search for Roboticscited by 26
  16. 2024Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?cited by 23
  17. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  18. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  19. 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
  20. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 20
  21. 2023LLM-Based Human-Robot Collaboration Framework for Manipulation Taskscited by 17
  22. 2025ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environmentscited by 14
  23. 2024A2C: A Modular Multi-stage Collaborative Decision Framework for Human-AI Teamscited by 12
  24. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  25. 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedbackcited by 11
  26. 2025M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Qualitycited by 11
  27. 2024To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactionscited by 10
  28. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  29. 2025Beyond Turn-taking: Introducing Text-based Overlap into Human-LLM Interactionscited by 6
  30. 2025Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocolscited by 6
  31. 2024Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gaincited by 5
  32. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  33. 2024Towards Modeling Human-Agentic Collaborative Workflows: A BPMN Extensioncited by 3
  34. 2025An Empirical Study of the Role of Incompleteness and Ambiguity in Interactions with Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 22.

29 papers.

Sort:

By area:

  1. 2026CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verificationcited by 128
  2. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  3. 2026GAM: Hierarchical Graph-based Agentic Memory for LLM Agentscited by 19
  4. 2025The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerabilitycited by 12
  5. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  6. 2026AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompterscited by 8
  7. 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defensescited by 7
  8. 2026Stop When Reasoning Converges: Semantic-Preserving Early Exit for Reasoning Modelscited by 7
  9. 2025EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysiscited by 4
  10. 2026Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectoriescited by 4
  11. 2026JudgeFlow: Agentic Workflow Optimization via Block Judgecited by 3
  12. 2025Why do AI agents communicate in human language?cited by 2
17 more
  1. 2025MADIAVE: Multi-Agent Debate for Implicit Attribute Value Extractioncited by 2
  2. 2025PersonaAgent with GraphRAG: Community-Aware Knowledge Graphs for Personalized LLMcited by 2
  3. 2026ViviDoc: Generating Interactive Documents through Human-Agent Collaborationcited by 2
  4. 2026CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experimentscited by 2
  5. 2026AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoningcited by 2
  6. 2026Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Modelscited by 1
  7. 2026CollabSkill: Evaluating Human-Agent Collaboration On Real-World Taskscited by 1
  8. 2026Discovering Crystal Structure Prediction Algorithms with an AI Co-Scientistcited by 1
  9. 2026Social Structure Matters in 3D Human-Human Interaction Generationcited by 1
  10. 2026HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participationcited by 1
  11. 2026SkillHEX: Improving Agent Skills via Hypothesis-Driven Autonomous Exploration and Exploitationcited by 1
  12. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  13. 2026Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysiscited by 0
  14. 2026The Boundaries of Automation: A Theory of Persistent Human Participationcited by 0
  15. 2026Beyond Component Testing: Validating Agentic AI Systemscited by 0
  16. 2026IntelliAudit: Using Large Language Models to Evaluate Audit Controlscited by 0
  17. 2026Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  3. 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems
  4. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  5. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application
  6. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects
  7. 2024Practical Considerations for Agentic LLM Systems
  8. 2024Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction