mlsanityData to 7 Oct 2026

Scaling Data Generation in Vision-and-Language Navigation

arXiv 2307.15644 ↗First posted 28 Jul 2023cs.CV cs.AI cs.CLcited by 115

Not on the map. Why?

Abstract (arXiv)

Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents. To tackle the common data scarcity issue in existing vision-and-language navigation datasets, we propose an effective paradigm for generating large-scale data for learning, which applies 1200+ photo-realistic environments from HM3D and Gibson datasets and synthesizes 4.9 million instruction trajectory pairs using fully-accessible resources on the web. Importantly, we investigate the influence of each component in this paradigm on the agent's performance and study how to adequately apply the augmented data to pre-train and fine-tune an agent. Thanks to our large-scale dataset, the performance of an existing agent can be pushed up (+11% absolute with regard to previous SoTA) to a significantly new best of 80% single-run success rate on the R2R test split by simple imitation learning. The long-lasting generalization gap between navigating in seen and unseen environments is also reduced to less than 1% (versus 8% in the previous best method). Moreover, our paradigm also facilitates different models to achieve new state-of-the-art navigation results on CVDN, REVERIE, and R2R in continuous environments.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 79 of its 97 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

79 of 97 references.

References · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  8. 2018On Evaluation of Embodied Navigation Agentscited by 526
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  10. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  12. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
38 more
  1. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  3. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  6. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  7. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  8. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  9. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  10. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  13. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  14. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  15. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  16. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  17. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  18. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  19. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  20. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  21. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  22. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  23. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  24. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  25. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  26. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  27. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  28. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  29. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  30. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  31. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  32. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  33. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  34. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  35. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  36. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  37. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  38. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 32; 2025: 39; 2026: 42.

115 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  3. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  4. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  5. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  6. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  7. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  8. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  9. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  10. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  11. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  12. 2024Vision-and-Language Navigation via Causal Learningcited by 51
38 more
  1. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  2. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  3. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  4. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  5. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  6. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  7. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  8. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  9. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  10. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  11. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  12. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18
  13. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  14. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  15. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  16. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  17. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  18. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  19. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  20. 2025Towards Physically Executable 3D Gaussian for Embodied Navigationcited by 16
  21. 2025Do Visual Imaginations Improve Vision-and-Language Navigation Agents?cited by 14
  22. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  23. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  24. 2026When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoningcited by 12
  25. 2024Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigationcited by 11
  26. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 11
  27. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  28. 2025Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environmentscited by 10
  29. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  30. 2024Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Promptscited by 9
  31. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  32. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  33. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  34. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  35. 2025PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigationcited by 6
  36. 2025HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactionscited by 6
  37. 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
  38. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Multimodal Large Language Model for Visual Navigation
  2. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
  3. 2026GN0: Toward a Unified Paradigm for Generation, Evaluation, and Policy Learning in Visual-Language Navigation
  4. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains
  5. 2026OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
  6. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  7. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
  8. 2024Cog-GA: A Large Language Models-based Generative Agent for Vision-Language Navigation in Continuous Environments