# LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey

arXiv: https://arxiv.org/abs/2505.00753 · HTML: https://mlsanity.com/paper/2505.00753/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 62 of its 127 references that are in the map.
- First posted on arXiv 1 May 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 62)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning](https://mlsanity.com/paper/2406.11896/) (2024)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks](https://mlsanity.com/paper/1812.09755/) (2018)
- [Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2309.10228/) (2023)
- [PersonaGym: Evaluating Persona Agents and LLMs](https://mlsanity.com/paper/2407.18416/) (2024)
- [MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft](https://mlsanity.com/paper/2504.08388/) (2025)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [A Survey of LLM-based Agents in Medicine: How far are we from Baymax?](https://mlsanity.com/paper/2502.11211/) (2025)
- [Personalizing Dialogue Agents via Meta-Learning](https://mlsanity.com/paper/1905.10033/) (2019)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration](https://mlsanity.com/paper/2412.15701/) (2024)
- [InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions](https://mlsanity.com/paper/2305.18047/) (2023)
- [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024)
- [Ask-before-Plan: Proactive Language Agents for Real-World Planning](https://mlsanity.com/paper/2406.12639/) (2024)
- [A Survey on LLM-powered Agents for Recommender Systems](https://mlsanity.com/paper/2502.10050/) (2025)

## Built on by (most cited first, 29 of 29)

- [CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification](https://mlsanity.com/paper/2604.01687/) (2026)
- [TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems](https://mlsanity.com/paper/2506.04133/) (2025)
- [GAM: Hierarchical Graph-based Agentic Memory for LLM Agents](https://mlsanity.com/paper/2604.12285/) (2026)
- [The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerability](https://mlsanity.com/paper/2510.18563/) (2025)
- [A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy](https://mlsanity.com/paper/2506.09420/) (2025)
- [AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters](https://mlsanity.com/paper/2605.22645/) (2026)
- [Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses](https://mlsanity.com/paper/2605.02900/) (2026)
- [Stop When Reasoning Converges: Semantic-Preserving Early Exit for Reasoning Models](https://mlsanity.com/paper/2605.17672/) (2026)
- [EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis](https://mlsanity.com/paper/2510.00024/) (2025)
- [Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories](https://mlsanity.com/paper/2607.25560/) (2026)
- [JudgeFlow: Agentic Workflow Optimization via Block Judge](https://mlsanity.com/paper/2601.07477/) (2026)
- [Why do AI agents communicate in human language?](https://mlsanity.com/paper/2506.02739/) (2025)
- [MADIAVE: Multi-Agent Debate for Implicit Attribute Value Extraction](https://mlsanity.com/paper/2510.05611/) (2025)
- [PersonaAgent with GraphRAG: Community-Aware Knowledge Graphs for Personalized LLM](https://mlsanity.com/paper/2511.17467/) (2025)
- [ViviDoc: Generating Interactive Documents through Human-Agent Collaboration](https://mlsanity.com/paper/2603.27991/) (2026)
- [CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments](https://mlsanity.com/paper/2606.06399/) (2026)
- [AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning](https://mlsanity.com/paper/2608.29988/) (2026)
- [Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Models](https://mlsanity.com/paper/2604.13706/) (2026)
- [CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks](https://mlsanity.com/paper/2606.09833/) (2026)
- [Discovering Crystal Structure Prediction Algorithms with an AI Co-Scientist](https://mlsanity.com/paper/2606.22866/) (2026)
- [Social Structure Matters in 3D Human-Human Interaction Generation](https://mlsanity.com/paper/2606.24255/) (2026)
- [HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation](https://mlsanity.com/paper/2607.04329/) (2026)
- [SkillHEX: Improving Agent Skills via Hypothesis-Driven Autonomous Exploration and Exploitation](https://mlsanity.com/paper/2608.05628/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis](https://mlsanity.com/paper/2603.05510/) (2026)
- [The Boundaries of Automation: A Theory of Persistent Human Participation](https://mlsanity.com/paper/2607.21547/) (2026)
- [Beyond Component Testing: Validating Agentic AI Systems](https://mlsanity.com/paper/2607.29405/) (2026)
- [IntelliAudit: Using Large Language Models to Evaluate Audit Controls](https://mlsanity.com/paper/2608.07688/) (2026)
- [Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systems](https://mlsanity.com/paper/2608.14667/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have sparked growing interest in building fully autonomous agents. However, fully autonomous LLM-based agents still face significant challenges, including limited reliability due to hallucinations, difficulty in handling complex tasks, and substantial safety and ethical risks, all of which limit their feasibility and trustworthiness in real-world applications. To overcome these limitations, LLM-based human-agent systems (LLM-HAS) incorporate human-provided information, feedback, or control into the agent system to enhance system performance, reliability, and safety. These human-agent collaboration systems enable humans and LLM-based agents to collaborate effectively by leveraging their complementary strengths. This paper provides the first comprehensive and structured survey of LLM-HAS. It clarifies fundamental concepts, systematically presents core components shaping these systems, including environment and profiling, human feedback, interaction types, orchestration, and communication, explores emerging applications, and discusses unique challenges and opportunities arising from human-AI collaboration. By consolidating current knowledge and offering a structured overview, we aim to foster further research and innovation in this rapidly evolving interdisciplinary field. Paper lists and resources are available at https://github.com/HenryPengZou/Awesome-Human-Agent-Collaboration-Interaction-Systems.