mlsanityData to 7 Oct 2026

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

arXiv 2004.14973 ↗First posted 30 Apr 2020cs.CV cs.AI cs.CLcited by 177

Not on the map. Why?

Abstract (arXiv)

Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e.g. 'stairs') to visual content in the environment (pixels corresponding to 'stairs'). We ask the following question -- can we leverage abundant 'disembodied' web-scraped vision-and-language corpora (e.g. Conceptual Captions) to learn visual groundings (what do 'stairs' look like?) that improve performance on a relatively data-starved embodied perception task (Vision-and-Language Navigation)? Specifically, we develop VLN-BERT, a visiolinguistic transformer-based model for scoring the compatibility between an instruction ('...stop at the brown sofa') and a sequence of panoramic RGB images captured by the agent. We demonstrate that pretraining VLN-BERT on image-text pairs from the web before fine-tuning on embodied path-instruction data significantly improves performance on VLN -- outperforming the prior state-of-the-art in the fully-observed setting by 4 absolute percentage points on success rate. Ablations of our pretraining curriculum show each stage to be impactful -- with their combination resulting in further positive synergistic effects.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 43; 2023: 34; 2024: 35; 2025: 16; 2026: 8.

177 papers.

Sort:

By area:

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  5. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  6. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  7. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  8. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  9. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  10. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  11. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  12. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
38 more
  1. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  2. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  3. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  4. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  5. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  6. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  7. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  8. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  9. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  10. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  11. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  12. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  13. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  14. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  15. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  16. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  17. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  18. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  19. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  20. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  21. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  22. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  23. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  24. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  25. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  26. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  27. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  28. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  29. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  30. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  31. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  32. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  33. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  34. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  35. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  36. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  37. 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 50
  38. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50

50 most cited of 177 All 177 (JSON)