mlsanityData to 7 Oct 2026

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training

arXiv 2002.10638 ↗First posted 25 Feb 2020cs.CV cs.CL cs.LGcited by 228

Not on the map. Why?

Abstract (arXiv)

Learning to navigate in a visual environment following natural-language instructions is a challenging task, because the multimodal inputs to the agent are highly variable, and the training data on a new task is often limited. In this paper, we present the first pre-training and fine-tuning paradigm for vision-and-language navigation (VLN) tasks. By training on a large amount of image-text-action triplets in a self-supervised learning manner, the pre-trained model provides generic representations of visual environments and language instructions. It can be easily used as a drop-in for existing VLN frameworks, leading to the proposed agent called Prevalent. It learns more effectively in new tasks and generalizes better in a previously unseen environment. The performance is validated on three VLN tasks. On the Room-to-Room benchmark, our model improves the state-of-the-art from 47% to 51% on success rate weighted by path length. Further, the learned representation is transferable to other VLN tasks. On two recent tasks, vision-and-dialog navigation and "Help, Anna!" the proposed Prevalent leads to significant improvement over existing methods, achieving a new state of the art.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 49; 2023: 43; 2024: 46; 2025: 32; 2026: 29.

228 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  5. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  6. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  7. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  8. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  9. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  10. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  11. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  12. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
38 more
  1. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  2. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  3. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  4. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  5. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  8. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  9. 2023AerialVLN: Vision-and-Language Navigation for UAVscited by 117
  10. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  11. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  12. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  13. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  14. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  15. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  16. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  17. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  18. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  19. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  20. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  21. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  22. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  23. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  24. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  25. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  26. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  27. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  28. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  29. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  30. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  31. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  32. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  33. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  34. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  35. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  36. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  37. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  38. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53

50 most cited of 228 All 228 (JSON)