mlsanityData to 14 Oct 2026

I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots

arXiv 2311.08957 ↗First posted 15 Nov 2023cs.RO cs.AI cs.HCcited by 4

Not on the map. Why?

Abstract (arXiv)

In the rapidly evolving landscape of human-computer interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents an initial implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4, IDEFICS) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. By implementing this vision-enabled dialogue system, the paper envisions a future where conversational agents seamlessly blend textual and visual modalities, enabling richer, more context-aware dialogues.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 10 of its 31 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

10 of 31 references.

References · 10

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  7. 2022Robot Operating System 2: Design, Architecture, and Uses In The Wildcited by 199
  8. 2019What Makes a Good Conversation? Challenges in Designing Truly Conversational Agentscited by 42
  9. 2020ROS for Human-Robot Interactioncited by 6
  10. 2023ROSGPT_Vision: Commanding Robots Using Only Language Models' Promptscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Vision Language Models as Values Detectorscited by 3
  2. 2025Towards Multimodal Social Conversations with Robots: Using Vision-Language Modelscited by 3
  3. 2026A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robotcited by 1
  4. 2026The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Modelscited by 0