mlsanityData to 7 Oct 2026

Airbert: In-domain Pretraining for Vision-and-Language Navigation

arXiv 2108.09105 ↗First posted 20 Aug 2021cs.CV cs.AI cs.CLcited by 129

Not on the map. Why?

Abstract (arXiv)

Vision-and-language navigation (VLN) aims to enable embodied agents to navigate in realistic environments using natural language instructions. Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging. Recent methods explore pretraining to improve generalization, however, the use of generic image-caption datasets or existing small-scale VLN environments is suboptimal and results in limited improvements. In this work, we introduce BnB, a large-scale and diverse in-domain VLN dataset. We first collect image-caption (IC) pairs from hundreds of thousands of listings from online rental marketplaces. Using IC pairs we next propose automatic strategies to generate millions of VLN path-instruction (PI) pairs. We further propose a shuffling loss that improves the learning of temporal order inside PI pairs. We use BnB pretrain our Airbert model that can be adapted to discriminative and generative settings and show that it outperforms state of the art for Room-to-Room (R2R) navigation and Remote Referring Expression (REVERIE) benchmarks. Moreover, our in-domain pretraining significantly increases performance on a challenging few-shot VLN evaluation, where we train the model only on VLN instructions from a few houses.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 45 of its 53 references that are in the map. First posted on arXiv 20 Aug 2021.

Builds on

45 of 53 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
33 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2015Pointer Networkscited by 882
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2018On Evaluation of Embodied Navigation Agentscited by 526
  5. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  6. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  7. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  8. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  10. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  11. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  12. 2019Vision-and-Dialog Navigationcited by 246
  13. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  14. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  15. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  16. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  17. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  18. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  19. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  20. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  21. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  22. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  23. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  24. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  25. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  26. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  27. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  28. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  29. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  30. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
  31. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  32. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37
  33. 2018Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examplescited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 22; 2023: 35; 2024: 32; 2025: 24; 2026: 14.

129 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2022Visual Language Maps for Robot Navigationcited by 362
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  5. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  6. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  7. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  8. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  9. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  10. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  11. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  12. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
38 more
  1. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  2. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  3. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  4. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  5. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  6. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  7. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  8. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  9. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  10. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  11. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  12. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  13. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  14. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  15. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  16. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  17. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  18. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  19. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  20. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  21. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  22. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  23. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  24. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  25. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  26. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
  27. 2022Reinforced Structured State-Evolution for Vision-Language Navigationcited by 36
  28. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  29. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  30. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  31. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  32. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  33. 2022Towards Versatile Embodied Navigationcited by 26
  34. 2022Visual-Language Navigation Pretraining via Prompt-based Environmental Self-explorationcited by 25
  35. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  36. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  37. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  38. 2023GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigationcited by 24

50 most cited of 129 All 129 (JSON)