mlsanityData to 7 Oct 2026

Human-Robot Interaction in GenAI Architectures via the Agent-Client Protocol

arXiv 2607.14919 ↗First posted 16 Jul 2026cs.ROcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent advances in Generative Artificial Intelligence (GenAI), particularly Large Language Models (LLMs), are driving robotic architectures toward agent-based high-level orchestration, in which natural-language instructions can be translated into context-aware action sequences. While the integration of these agents and robotic capabilities is increasingly converging toward standardization through the Model Context Protocol (MCP), the upper Human-Robot Interaction (HRI) layer remains fragmented by proprietary, ad hoc interfaces that hinder real-time human-in-the-loop collaboration. To address this fragmentation, this paper proposes the adoption of the Agent-Client Protocol (ACP) -- a communication standard originally introduced for coding agents in software engineering -- as a unified communication contract for the HRI layer in agent-based robotic systems. By combining ACP at the interface-agent link and MCP at the agent-execution link, we formulate a fully decoupled three-layer architecture that separates human interaction, deliberative orchestration, and physical execution. This topology removes rigid architectural dependencies, enabling heterogeneous user interfaces to connect to the same robotic system and allowing the underlying robotic platform to be replaced without requiring client-specific integration changes. Moreover, it provides native support for collaborative HRI capabilities such as real-time observability, explicit human authorization, and immediate task interruption. We experimentally evaluate the proposed architecture on a physical mobile robot, demonstrating interoperability across three heterogeneous user interfaces and validating real-time human-in-the-loop workflows with negligible latency overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 15 of its 21 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

15 of 21 references.

References · 15

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  10. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  11. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  12. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
3 more
  1. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  2. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  3. 2025RoboNeuron: A Middle-Layer Infrastructure for Agent-Driven Orchestration in Embodied AIcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applications
  2. 2025From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systems
  3. 2025Human-Artificial Interaction in the Age of Agentic AI: A System-Theoretical Approach
  4. 2025Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration
  5. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents
  6. 2025A Survey of AI Agent Protocols
  7. 2026Human Tool: An MCP-Style Framework for Human-Agent Collaboration
  8. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey