# I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots

arXiv: https://arxiv.org/abs/2311.08957 · HTML: https://mlsanity.com/paper/2311.08957/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [ROS for Human-Robot Interaction](https://mlsanity.com/paper/2012.13944/) (2020)
- [ROSGPT_Vision: Commanding Robots Using Only Language Models' Prompts](https://mlsanity.com/paper/2308.11236/) (2023)

## Built on by (most cited first, 4 of 4)

- [Vision Language Models as Values Detectors](https://mlsanity.com/paper/2501.03957/) (2025)
- [Towards Multimodal Social Conversations with Robots: Using Vision-Language Models](https://mlsanity.com/paper/2507.19196/) (2025)
- [A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot](https://mlsanity.com/paper/2603.21013/) (2026)
- [The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models](https://mlsanity.com/paper/2607.16318/) (2026)

## Abstract (arXiv)

In the rapidly evolving landscape of human-computer interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents an initial implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4, IDEFICS) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. By implementing this vision-enabled dialogue system, the paper envisions a future where conversational agents seamlessly blend textual and visual modalities, enabling richer, more context-aware dialogues.