mlsanityData to 7 Oct 2026

Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation

arXiv 2203.02764 ↗First posted 5 Mar 2022cs.CV cs.CL cs.ROcited by 127

Not on the map. Why?

Abstract (arXiv)

Most existing works in vision-and-language navigation (VLN) focus on either discrete or continuous environments, training agents that cannot generalize across the two. The fundamental difference between the two setups is that discrete navigation assumes prior knowledge of the connectivity graph of the environment, so that the agent can effectively transfer the problem of navigation with low-level controls to jumping from node to node with high-level actions by grounding to an image of a navigable direction. To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments. We refine the connectivity graph of Matterport3D to fit the continuous Habitat-Matterport3D, and train the waypoints predictor with the refined graphs to produce accessible waypoints at each time step. Moreover, we demonstrate that the predicted waypoints can be augmented during training to diversify the views and paths, and therefore enhance agent's generalization ability. Through extensive experiments we show that agents navigating in continuous environments with predicted waypoints perform significantly better than agents using low-level actions, which reduces the absolute discrete-to-continuous gap by 11.76% Success Weighted by Path Length (SPL) for the Cross-Modal Matching Agent and 18.24% SPL for the Recurrent VLN-BERT. Our agents, trained with a simple imitation learning objective, outperform previous methods by a large margin, achieving new state-of-the-art results on the testing environments of the R2R-CE and the RxR-CE datasets.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 5 Mar 2022.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
38 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  7. 2017Embodied Question Answeringcited by 335
  8. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  9. 2020Learning to Explore using Active Neural SLAMcited by 308
  10. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  11. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  12. 2019Vision-and-Dialog Navigationcited by 246
  13. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  14. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  15. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  16. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  17. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  18. 2018Semi-parametric Topological Memory for Navigationcited by 203
  19. 2020Neural Topological SLAM for Visual Navigationcited by 193
  20. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  21. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  22. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  23. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  24. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  25. 2019Learning Exploration Policies for Navigationcited by 113
  26. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  27. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  28. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  29. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  30. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  31. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  32. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  33. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  34. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  35. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  36. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  37. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  38. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 15; 2024: 21; 2025: 36; 2026: 46.

127 papers.

Sort:

By area:

  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  2. 2022Visual Language Maps for Robot Navigationcited by 362
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  5. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  6. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  7. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  8. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  9. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  10. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  11. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  12. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
38 more
  1. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  2. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  5. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  6. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  7. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  8. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  9. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  10. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  11. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  12. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  13. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  14. 2022Iterative Vision-and-Language Navigationcited by 34
  15. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  16. 20221st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)cited by 30
  17. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  18. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  19. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  20. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  21. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  22. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  23. 2023VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigationcited by 22
  24. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  25. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  26. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  27. 2024Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigationcited by 18
  28. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  29. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  30. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  31. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  32. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  33. 2025DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigationcited by 15
  34. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  35. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  36. 2025Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Constructioncited by 12
  37. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  38. 2023MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigationcited by 11

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Narrowing the Gap between Vision and Action in Navigation
  2. 2026Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
  3. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigation
  4. 2025ETP-R1: Evolving Topological Planning with Reinforcement Fine-tuning for Vision-Language Navigation in Continuous Environments
  5. 2026Language-Conditioned World Modeling for Visual Navigation
  6. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
  7. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
  8. 2026Revisiting Topological Graphs for Macro Action based Closed-loop Reinforcement Learning of Vision Language Navigation in Continuous Environment ↗