mlsanityData to 14 Oct 2026

Advancing DialNav through Automatic Embodied Dialog Augmentation

arXiv 2606.19948 ↗First posted 18 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

For embodied agents capable of physical interaction, the capability to create and understand dialog is crucial to ensure both safety and effectiveness. While DialNav~citehan2025dialnav provides a framework for holistic evaluation of the dialog--execution loop in photorealistic indoor navigation, its performance remains limited by a critical scarcity of training data (2K episodes). To address this, we propose an automatic generation pipeline, and construct the RAINbow dataset, a large-scale training dataset with 238K episodes for DialNav. Our pipeline converts existing VLN datasets into multi-turn dialog and creates cost-efficient and high-quality dataset. Then, we introduce two additional complementary advances to unlock the data's full potential: (1) Dual-Strategy Training, a navigation training scheme to align the navigation training with the dynamic dialog-navigation loop, and (2) a localization model that leverages VLN knowledge. By combining these complementary solutions, our model substantially outperforms the baseline in success rate on both Val Seen (58.24, +89%) and Val Unseen (29.05, +100%) splits, establishing a new state of the art.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

31 of 47 references.

References · 31

  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  3. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  6. 2019Vision-and-Dialog Navigationcited by 250
  7. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  8. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  9. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  10. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  11. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  12. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
19 more
  1. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  2. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  3. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  4. 2019Executing Instructions in Situated Collaborative Interactionscited by 69
  5. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  6. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  7. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44
  8. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  9. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  10. 2020The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigationcited by 26
  11. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  12. 2020Where Are You? Localization from Embodied Dialogcited by 24
  13. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  14. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 21
  15. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 19
  16. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 12
  17. 2022Transformer-based Localization from Embodied Dialog with Large-scale Pre-trainingcited by 5
  18. 2024DiaLoc: An Iterative Approach to Embodied Dialog Localizationcited by 4
  19. 2025GOAT: A Training Framework for Goal-Oriented Agent with Toolscited by 1

Built on by

Not cited yet.