# LaMI: Large Language Models for Multi-Modal Human-Robot Interaction

arXiv: https://arxiv.org/abs/2401.15174 · HTML: https://mlsanity.com/paper/2401.15174/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2024.

## Built on by (most cited first, 21 of 21)

- [ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning](https://mlsanity.com/paper/2406.19741/) (2024)
- [Large Language Models for Multi-Robot Systems: A Survey](https://mlsanity.com/paper/2502.03814/) (2025)
- [FaceXBench: Evaluating Multimodal LLMs on Face Understanding](https://mlsanity.com/paper/2501.10360/) (2025)
- [To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions](https://mlsanity.com/paper/2403.12533/) (2024)
- [Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Libraries](https://mlsanity.com/paper/2407.21778/) (2024)
- [Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model](https://mlsanity.com/paper/2506.08967/) (2025)
- [A New Era in Human Factors Engineering: A Survey of the Applications and Prospects of Large Multimodal Models](https://mlsanity.com/paper/2405.13426/) (2024)
- [Graph-Grounded LLMs: Leveraging Graphical Function Calling to Minimize LLM Hallucinations](https://mlsanity.com/paper/2503.10941/) (2025)
- [Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task](https://mlsanity.com/paper/2404.08424/) (2024)
- [The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare](https://mlsanity.com/paper/2411.03287/) (2024)
- [Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models](https://mlsanity.com/paper/2503.15491/) (2025)
- [A Framework for Adapting Human-Robot Interaction to Diverse User Groups](https://mlsanity.com/paper/2410.11377/) (2024)
- [Plan-and-Act using Large Language Models for Interactive Agreement](https://mlsanity.com/paper/2504.01252/) (2025)
- [Population-Coded Spiking Neural Networks for High-Dimensional Robotic Control](https://mlsanity.com/paper/2510.10516/) (2025)
- [CANINE: Coaching Visually Impaired Users for Interactive Navigation with a Robot Guide Dog](https://mlsanity.com/paper/2605.19501/) (2026)
- [TransforMerger: Transformer-based Voice-Gesture Fusion for Robust Human-Robot Communication](https://mlsanity.com/paper/2504.01708/) (2025)
- [Interpretable Robot Control via Structured Behavior Trees and Large Language Models](https://mlsanity.com/paper/2508.09621/) (2025)
- [J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perception](https://mlsanity.com/paper/2510.21761/) (2025)
- [Robotic Assistant: Completing Collaborative Tasks with Dexterous Vision-Language-Action Models](https://mlsanity.com/paper/2510.25713/) (2025)
- [Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction](https://mlsanity.com/paper/2602.23312/) (2026)
- [STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration](https://mlsanity.com/paper/2608.27225/) (2026)

## Abstract (arXiv)

This paper presents an innovative large language model (LLM)-based robotic system for enhancing multi-modal human-robot interaction (HRI). Traditional HRI systems relied on complex designs for intent estimation, reasoning, and behavior generation, which were resource-intensive. In contrast, our system empowers researchers and practitioners to regulate robot behavior through three key aspects: providing high-level linguistic guidance, creating "atomic actions" and expressions the robot can use, and offering a set of examples. Implemented on a physical robot, it demonstrates proficiency in adapting to multi-modal inputs and determining the appropriate manner of action to assist humans with its arms, following researchers' defined guidelines. Simultaneously, it coordinates the robot's lid, neck, and ear movements with speech output to produce dynamic, multi-modal expressions. This showcases the system's potential to revolutionize HRI by shifting from conventional, manual state-and-flow design methods to an intuitive, guidance-based, and example-driven approach. Supplementary material can be found at https://hri-eu.github.io/Lami/