mlsanityData to 7 Oct 2026

Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments

arXiv 2204.09667 ↗First posted 20 Apr 2022cs.CV cs.CL cs.ROcited by 80

Not on the map. Why?

Abstract (arXiv)

Recent work in Vision-and-Language Navigation (VLN) has presented two environmental paradigms with differing realism -- the standard VLN setting built on topological environments where navigation is abstracted away, and the VLN-CE setting where agents must navigate continuous 3D environments using low-level actions. Despite sharing the high-level task and even the underlying instruction-path data, performance on VLN-CE lags behind VLN significantly. In this work, we explore this gap by transferring an agent from the abstract environment of VLN to the continuous environment of VLN-CE. We find that this sim-2-sim transfer is highly effective, improving over the prior state of the art in VLN-CE by +12% success rate. While this demonstrates the potential for this direction, the transfer does not fully retain the original performance of the agent in the abstract setting. We present a sequence of experiments to identify what differences result in performance degradation, providing clear directions for further improvement.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

26 of 28 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2018On Evaluation of Embodied Navigation Agentscited by 526
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  8. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  9. 2020Learning to Explore using Active Neural SLAMcited by 308
  10. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  11. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  12. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
14 more
  1. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  2. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  3. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  4. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  5. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  6. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  7. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  8. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  9. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  10. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  11. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  12. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  13. 2019SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigationcited by 29
  14. 2019Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?cited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 14; 2025: 25; 2026: 30.

80 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  5. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  6. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  7. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  8. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  9. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  10. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  11. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  12. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
38 more
  1. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  2. 2023Navigating to Objects Specified by Imagescited by 50
  3. 2024GaussNav: Gaussian Splatting for Visual Navigationcited by 49
  4. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  5. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  6. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  7. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  8. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  9. 20221st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)cited by 30
  10. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  11. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 26
  12. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  13. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  14. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  15. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  16. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  17. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  18. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  19. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  20. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  21. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  22. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  23. 2025Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Constructioncited by 12
  24. 2023MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigationcited by 11
  25. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  26. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  27. 2025VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigationcited by 8
  28. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigationcited by 7
  29. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  30. 2025Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigationcited by 6
  31. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 6
  32. 2024Narrowing the Gap between Vision and Action in Navigationcited by 5
  33. 2026textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentationcited by 5
  34. 2026PanoWorld: Towards Spatial Supersensing in 360^circ Panorama Worldcited by 5
  35. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 5
  36. 2025VPN: Visual Prompt Navigationcited by 4
  37. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigationcited by 4
  38. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Promptingcited by 4

50 most cited of 80 All 80 (JSON)