mlsanityData to 14 Oct 2026

DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding

arXiv 2307.06924 ↗First posted 13 Jul 2023cs.RO cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Persons with visual impairments (PwVI) have difficulties understanding and navigating spaces around them. Current wayfinding technologies either focus solely on navigation or provide limited communication about the environment. Motivated by recent advances in visual-language grounding and semantic navigation, we propose DRAGON, a guiding robot powered by a dialogue system and the ability to associate the environment with natural language. By understanding the commands from the user, DRAGON is able to guide the user to the desired landmarks on the map, describe the environment, and answer questions from visual observations. Through effective utilization of dialogue, the robot can ground the user's free-form descriptions to landmarks in the environment, and give the user semantic information through spoken language. We conduct a user study with blindfolded participants in an everyday indoor environment. Our results demonstrate that DRAGON is able to communicate with the user smoothly, provide a good guiding experience, and connect users with their surrounding environment in an intuitive manner. Videos and code are available at https://sites.google.com/view/dragon-wayfinding/home.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

26 of 46 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  9. 2022Visual Language Maps for Robot Navigationcited by 365
  10. 2018Semi-parametric Topological Memory for Navigationcited by 209
  11. 2020Neural Topological SLAM for Visual Navigationcited by 195
  12. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
14 more
  1. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 178
  2. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  3. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  4. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  5. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  6. 2019Scaling Local Control to Large-Scale Topological Navigationcited by 37
  7. 2020DIET: Lightweight Language Understanding for Dialogue Systemscited by 35
  8. 2021Robotic Guide Dog: Leading a Human with Leash-Guided Hybrid Physical Interactioncited by 16
  9. 2018Personalized Dynamics Models for Adaptive Assistive Navigation Systemscited by 13
  10. 2017A Review of Methodologies for Natural-Language-Facilitated Human-Robot Cooperationcited by 8
  11. 2019Robot Sound Interpretation: Combining Sight and Sound in Learning-Based Controlcited by 7
  12. 2023"I am the follower, also the boss": Exploring Different Levels of Autonomy and Machine Forms of Guiding Robots for the Visually Impairedcited by 6
  13. 2021Learning Visual-Audio Representations for Voice-Controlled Robotscited by 5
  14. 2022"I was Confused by It; It was Confused by Me:" Exploring the Experiences of People with Visual Impairments around Mobile Service Robotscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 1.

9 papers.

  1. 2024Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossingcited by 12
  2. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  3. 2023Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recoverycited by 6
  4. 2024R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbationscited by 6
  5. 2024MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphscited by 4
  6. 2025ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusioncited by 4
  7. 2025GuideNav: User-Informed Development of a Vision-Only Robotic Navigation Assistant For Blind Travelerscited by 4
  8. 2025Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Modelscited by 1
  9. 2026Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agentscited by 0