mlsanityData to 7 Oct 2026

A Modern System Recipe for Situated Embodied Human-Robot Conversation with Real-Time Multimodal LLMs and Tool-Calling

arXiv 2602.04157 ↗First posted 4 Feb 2026cs.ROcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Situated embodied conversation requires robots to interleave real-time dialogue with active perception: deciding what to look at, when to look, and what to say under tight latency constraints. We present a simple, minimal system recipe that pairs a real-time multimodal language model with a small set of tool interfaces for attention and active perception. We study six home-style scenarios that require frequent attention shifts and increasing perceptual scope. Across four system variants, we evaluate turn-level tool-decision correctness against human annotations and collect subjective ratings of interaction quality. Results indicate that real-time multimodal large language models and tool use for active perception is a promising direction for practical situated embodied conversation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 42 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

7 of 42 references.

References · 7

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2025SAM 3: Segment Anything with Conceptscited by 835
  5. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 414
  6. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  7. 2023FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial Expressionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelismcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Multimodal Social Conversations with Robots: Using Vision-Language Models
  2. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Models
  3. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
  4. 2026Multimodal Large Language Models for Real-Time Situated Reasoning
  5. 2026A Multimodal Framework for Human-Multi-Agent Interaction
  6. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
  7. 2024Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models
  8. 2023PaLM-E: An Embodied Multimodal Language Model