mlsanityData to 13 Oct 2026

Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout

arXiv 1904.04195 ↗First posted 8 Apr 2019cs.CL cs.CV cs.LGcited by 241

Not on the map. Why?

Abstract (arXiv)

A grand goal in AI is to build a robot that can accurately navigate based on natural language instructions, which requires the agent to perceive the scene, understand and ground language, and act in the real-world environment. One key challenge here is to learn to navigate in new environments that are unseen during training. Most of the existing approaches perform dramatically worse in unseen environments as compared to seen ones. In this paper, we present a generalizable navigational agent. Our agent is trained in two stages. The first stage is training via mixed imitation and reinforcement learning, combining the benefits from both off-policy and on-policy optimization. The second stage is fine-tuning via newly-introduced 'unseen' triplets (environment, path, instruction). To generate these unseen triplets, we propose a simple but effective 'environmental dropout' method to mimic unseen environments, which overcomes the problem of limited seen environment variability. Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions. Empirically, we show that our agent is substantially better at generalizability when fine-tuned with these triplets, outperforming the state-of-art approaches by a large margin on the private unseen test set of the Room-to-Room task, and achieving the top rank on the leaderboard.

Key facts. Cited by 241 papers in the map (data through 13 Oct 2026). First posted on arXiv 8 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 1; 2021: 42; 2022: 37; 2023: 41; 2024: 53; 2025: 37; 2026: 30.

241 papers.

Sort:

By area:

  1. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 276
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 236
  5. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 226
  6. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 205
  7. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 188
  8. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  9. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  10. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  11. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 129
  12. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 126
38 more
  1. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 119
  2. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  3. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  4. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  5. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  6. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  7. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 111
  8. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 109
  9. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  10. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  11. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  12. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 89
  13. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  14. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  15. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  16. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 83
  17. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 82
  18. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  19. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  20. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  21. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  22. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  23. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  24. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  25. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  26. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  27. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  28. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 62
  29. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  30. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 61
  31. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  32. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  33. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  34. 2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
  35. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  36. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  37. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  38. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50

50 most cited of 241 All 241 (JSON)