# DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding

arXiv: https://arxiv.org/abs/2307.06924 · HTML: https://mlsanity.com/paper/2307.06924/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Scaling Local Control to Large-Scale Topological Navigation](https://mlsanity.com/paper/1909.12329/) (2019)
- [DIET: Lightweight Language Understanding for Dialogue Systems](https://mlsanity.com/paper/2004.09936/) (2020)
- [Robotic Guide Dog: Leading a Human with Leash-Guided Hybrid Physical Interaction](https://mlsanity.com/paper/2103.14300/) (2021)
- [Personalized Dynamics Models for Adaptive Assistive Navigation Systems](https://mlsanity.com/paper/1804.04118/) (2018)
- [A Review of Methodologies for Natural-Language-Facilitated Human-Robot Cooperation](https://mlsanity.com/paper/1701.08756/) (2017)
- [Robot Sound Interpretation: Combining Sight and Sound in Learning-Based Control](https://mlsanity.com/paper/1909.09172/) (2019)
- ["I am the follower, also the boss": Exploring Different Levels of Autonomy and Machine Forms of Guiding Robots for the Visually Impaired](https://mlsanity.com/paper/2302.03481/) (2023)
- [Learning Visual-Audio Representations for Voice-Controlled Robots](https://mlsanity.com/paper/2109.02823/) (2021)
- ["I was Confused by It; It was Confused by Me:" Exploring the Experiences of People with Visual Impairments around Mobile Service Robots](https://mlsanity.com/paper/2208.13894/) (2022)

## Built on by (most cited first, 9 of 9)

- [Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing](https://mlsanity.com/paper/2402.06794/) (2024)
- [Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairments](https://mlsanity.com/paper/2410.20666/) (2024)
- [Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery](https://mlsanity.com/paper/2309.14425/) (2023)
- [R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations](https://mlsanity.com/paper/2403.04924/) (2024)
- [MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs](https://mlsanity.com/paper/2403.05814/) (2024)
- [ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusion](https://mlsanity.com/paper/2509.17941/) (2025)
- [GuideNav: User-Informed Development of a Vision-Only Robotic Navigation Assistant For Blind Travelers](https://mlsanity.com/paper/2512.06147/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)
- [Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents](https://mlsanity.com/paper/2604.25161/) (2026)

## Abstract (arXiv)

Persons with visual impairments (PwVI) have difficulties understanding and navigating spaces around them. Current wayfinding technologies either focus solely on navigation or provide limited communication about the environment. Motivated by recent advances in visual-language grounding and semantic navigation, we propose DRAGON, a guiding robot powered by a dialogue system and the ability to associate the environment with natural language. By understanding the commands from the user, DRAGON is able to guide the user to the desired landmarks on the map, describe the environment, and answer questions from visual observations. Through effective utilization of dialogue, the robot can ground the user's free-form descriptions to landmarks in the environment, and give the user semantic information through spoken language. We conduct a user study with blindfolded participants in an everyday indoor environment. Our results demonstrate that DRAGON is able to communicate with the user smoothly, provide a good guiding experience, and connect users with their surrounding environment in an intuitive manner. Videos and code are available at https://sites.google.com/view/dragon-wayfinding/home.