mlsanityData to 7 Oct 2026

Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction

arXiv 2404.09705 ↗First posted 15 Apr 2024cs.ROcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

This paper presents an improved system based on our prior work, designed to create explanations for autonomous robot actions during Human-Robot Interaction (HRI). Previously, we developed a system that used Large Language Models (LLMs) to interpret logs and produce natural language explanations. In this study, we expand our approach by incorporating Vision-Language Models (VLMs), enabling the system to analyze textual logs with the added context of visual input. This method allows for generating explanations that combine data from the robot's logs and the images it captures. We tested this enhanced system on a basic navigation task where the robot needs to avoid a human obstacle. The findings from this preliminary study indicate that adding visual interpretation improves our system's explanations by precisely identifying obstacles and increasing the accuracy of the explanations provided.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 16 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

9 of 16 references.

References · 9

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  6. 2022Robot Operating System 2: Design, Architecture, and Uses In The Wildcited by 190
  7. 2020Explainable Artificial Intelligence: a Systematic Reviewcited by 79
  8. 2023Using Large Language Models for Interpreting Autonomous Robots Behaviorscited by 4
  9. 2024Explaining Autonomy: Enhancing Human-Robot Interaction through Explanation Generation with Large Language Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  2. 2024A Multi-Modal Explainability Approach for Human-Aware Robots in Multi-Party Conversationcited by 0
  3. 2025Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Modelscited by 0
  4. 2025Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robotscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models
  2. 2025Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models
  3. 2026Lightweight Visual Reasoning for Socially-Aware Robots
  4. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
  5. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  6. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
  7. 2024Task-oriented Robotic Manipulation with Vision Language Models
  8. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models