# A Multimodal Framework for Human-Multi-Agent Interaction

arXiv: https://arxiv.org/abs/2603.23271 · HTML: https://mlsanity.com/paper/2603.23271/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 7 of 7)

- [From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systems](https://mlsanity.com/paper/2506.20091/) (2025)
- [Using Physiological Measures, Gaze, and Facial Expressions to Model Human Trust in a Robot Partner](https://mlsanity.com/paper/2504.05291/) (2025)
- [Improving Human Motion Prediction Through Continual Learning](https://mlsanity.com/paper/2107.00544/) (2021)
- [Can you pass that tool?: Implications of Indirect Speech in Physical Human-Robot Collaboration](https://mlsanity.com/paper/2502.11720/) (2025)
- [CoHRT: A Collaboration System for Human-Robot Teamwork](https://mlsanity.com/paper/2410.08504/) (2024)
- [The Beatbots: A Musician-Informed Multi-Robot Percussion Quartet](https://mlsanity.com/paper/2502.00966/) (2025)
- [Gazing at Failure: Investigating Human Gaze in Response to Robot Failure in Collaborative Tasks](https://mlsanity.com/paper/2502.16899/) (2025)

## Abstract (arXiv)

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.