# Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models

arXiv: https://arxiv.org/abs/2403.12273 · HTML: https://mlsanity.com/paper/2403.12273/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers](https://mlsanity.com/paper/2311.10242/) (2023)
- [Understanding why SLAM algorithms fail in modern indoor environments](https://mlsanity.com/paper/2305.05313/) (2023)
- [The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language](https://mlsanity.com/paper/2401.11838/) (2024)
- [ROMR: A ROS-based Open-source Mobile Robot](https://mlsanity.com/paper/2210.01627/) (2022)
- [Facilitating Human-Robot Collaboration through Natural Vocal Conversations](https://mlsanity.com/paper/2311.13973/) (2023)

## Built on by (most cited first, 2 of 2)

- [SIL: Symbiotic Interactive Learning for Language-Conditioned Human-Agent Co-Adaptation](https://mlsanity.com/paper/2511.05203/) (2025)
- [Lemon Agent Technical Report](https://mlsanity.com/paper/2602.07092/) (2026)

## Abstract (arXiv)

In this paper, we extended the method proposed in [21] to enable humans to interact naturally with autonomous agents through vocal and textual conversations. Our extended method exploits the inherent capabilities of pre-trained large language models (LLMs), multimodal visual language models (VLMs), and speech recognition (SR) models to decode the high-level natural language conversations and semantic understanding of the robot's task environment, and abstract them to the robot's actionable commands or queries. We performed a quantitative evaluation of our framework's natural vocal conversation understanding with participants from different racial backgrounds and English language accents. The participants interacted with the robot using both spoken and textual instructional commands. Based on the logged interaction data, our framework achieved 87.55% vocal commands decoding accuracy, 86.27% commands execution success, and an average latency of 0.89 seconds from receiving the participants' vocal chat commands to initiating the robot's actual physical action. The video demonstrations of this paper can be found at https://linusnep.github.io/MTCC-IRoNL/.