# Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction

arXiv: https://arxiv.org/abs/2404.09705 · HTML: https://mlsanity.com/paper/2404.09705/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 9 of 9)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Explainable Artificial Intelligence: a Systematic Review](https://mlsanity.com/paper/2006.00093/) (2020)
- [Using Large Language Models for Interpreting Autonomous Robots Behaviors](https://mlsanity.com/paper/2304.14844/) (2023)
- [Explaining Autonomy: Enhancing Human-Robot Interaction through Explanation Generation with Large Language Models](https://mlsanity.com/paper/2402.04206/) (2024)

## Built on by (most cited first, 4 of 4)

- [Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction](https://mlsanity.com/paper/2508.05294/) (2025)
- [A Multi-Modal Explainability Approach for Human-Aware Robots in Multi-Party Conversation](https://mlsanity.com/paper/2407.03340/) (2024)
- [Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models](https://mlsanity.com/paper/2504.05477/) (2025)
- [Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots](https://mlsanity.com/paper/2509.06768/) (2025)

## Abstract (arXiv)

This paper presents an improved system based on our prior work, designed to create explanations for autonomous robot actions during Human-Robot Interaction (HRI). Previously, we developed a system that used Large Language Models (LLMs) to interpret logs and produce natural language explanations. In this study, we expand our approach by incorporating Vision-Language Models (VLMs), enabling the system to analyze textual logs with the added context of visual input. This method allows for generating explanations that combine data from the robot's logs and the images it captures. We tested this enhanced system on a basic navigation task where the robot needs to avoid a human obstacle. The findings from this preliminary study indicate that adding visual interpretation improves our system's explanations by precisely identifying obstacles and increasing the accuracy of the explanations provided.