mlsanityData to 7 Oct 2026

BEVBert: Multimodal Map Pre-training for Language-guided Navigation

arXiv 2212.04385 ↗First posted 8 Dec 2022cs.CV cs.AI cs.CLcited by 117

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-training has shown promising results on the vision-and-language navigation (VLN) task. However, most existing pre-training methods employ discrete panoramas to learn visual-textual associations. This requires the model to implicitly correlate incomplete, duplicate observations within the panoramas, which may impair an agent's spatial understanding. Thus, we propose a new map-based pre-training paradigm that is spatial-aware for use in VLN. Concretely, we build a local metric map to explicitly aggregate incomplete observations and remove duplicates, while modeling navigation dependency in a global topological map. This hybrid design can balance the demand of VLN for both short-term reasoning and long-term planning. Then, based on the hybrid map, we devise a pre-training framework to learn a multimodal map representation, which enhances spatial-aware cross-modal reasoning thereby facilitating the language-guided navigation goal. Extensive experiments demonstrate the effectiveness of the map-based pre-training route for VLN, and the proposed method achieves state-of-the-art on four VLN benchmarks.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 70 of its 84 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

70 of 84 references.

References · 70

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2018On Evaluation of Embodied Navigation Agentscited by 526
38 more
  1. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  2. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  3. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  4. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  5. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  6. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  7. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  8. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  9. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  10. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  11. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  12. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  13. 2020Neural Topological SLAM for Visual Navigationcited by 193
  14. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  15. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  16. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  17. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  18. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  19. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  20. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  21. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  22. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  23. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  24. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  25. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  26. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  27. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  28. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  29. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  30. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  31. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  32. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  33. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  34. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  35. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  36. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  37. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  38. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 31; 2025: 38; 2026: 43.

117 papers.

Sort:

By area:

  1. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  2. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  3. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  4. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  5. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  6. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  7. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  8. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  9. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  10. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  11. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  12. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
38 more
  1. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  2. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  3. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  4. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  5. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 26
  6. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  7. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  8. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  9. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  10. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  11. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  12. 2024NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigationcited by 18
  13. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  14. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  15. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  16. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  17. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  18. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  19. 2025Do Visual Imaginations Improve Vision-and-Language Navigation Agents?cited by 14
  20. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  21. 2024Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigationcited by 11
  22. 2025Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environmentscited by 10
  23. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  24. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  25. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  26. 2026HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory Systemcited by 8
  27. 2024Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigationcited by 7
  28. 2025Active Test-time Vision-Language Navigationcited by 7
  29. 2024Semantic Map-based Generation of Navigation Instructionscited by 6
  30. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  31. 2024PRET: Planning with Directed Fidelity Trajectory for Vision and Language Navigationcited by 6
  32. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  33. 2025HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactionscited by 6
  34. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  35. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  36. 2025STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimizationcited by 6
  37. 2025Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigationcited by 6
  38. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 6

50 most cited of 117 All 117 (JSON)