# Advancing DialNav through Automatic Embodied Dialog Augmentation

arXiv: https://arxiv.org/abs/2606.19948 · HTML: https://mlsanity.com/paper/2606.19948/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning](https://mlsanity.com/paper/1909.01871/) (2019)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [Learning Vision-and-Language Navigation from YouTube Videos](https://mlsanity.com/paper/2307.11984/) (2023)
- [Talk the Walk: Navigating New York City through Grounded Dialogue](https://mlsanity.com/paper/1807.03367/) (2018)
- [Simple and Effective Synthesis of Indoor 3D Scenes](https://mlsanity.com/paper/2204.02960/) (2022)
- [RMM: A Recursive Mental Model for Dialog Navigation](https://mlsanity.com/paper/2005.00728/) (2020)
- [The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation](https://mlsanity.com/paper/2010.12639/) (2020)
- [VLN-Trans: Translator for the Vision and Language Navigation Agent](https://mlsanity.com/paper/2302.09230/) (2023)
- [Where Are You? Localization from Embodied Dialog](https://mlsanity.com/paper/2011.08277/) (2020)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Controllable Navigation Instruction Generation with Chain of Thought Prompting](https://mlsanity.com/paper/2407.07433/) (2024)
- [NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM](https://mlsanity.com/paper/2502.11142/) (2025)
- [Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation](https://mlsanity.com/paper/2307.13368/) (2023)
- [Transformer-based Localization from Embodied Dialog with Large-scale Pre-training](https://mlsanity.com/paper/2210.04864/) (2022)
- [DiaLoc: An Iterative Approach to Embodied Dialog Localization](https://mlsanity.com/paper/2403.06846/) (2024)

## Abstract (arXiv)

For embodied agents capable of physical interaction, the capability to create and understand dialog is crucial to ensure both safety and effectiveness. While DialNav~citehan2025dialnav provides a framework for holistic evaluation of the dialog--execution loop in photorealistic indoor navigation, its performance remains limited by a critical scarcity of training data (2K episodes). To address this, we propose an automatic generation pipeline, and construct the RAINbow dataset, a large-scale training dataset with 238K episodes for DialNav. Our pipeline converts existing VLN datasets into multi-turn dialog and creates cost-efficient and high-quality dataset. Then, we introduce two additional complementary advances to unlock the data's full potential: (1) Dual-Strategy Training, a navigation training scheme to align the navigation training with the dynamic dialog-navigation loop, and (2) a localization model that leverages VLN knowledge. By combining these complementary solutions, our model substantially outperforms the baseline in success rate on both Val Seen (58.24, +89%) and Val Unseen (29.05, +100%) splits, establishing a new state of the art.