mlsanityData to 7 Oct 2026

Cross-modal Map Learning for Vision and Language Navigation

arXiv 2203.05137 ↗First posted 10 Mar 2022cs.CV cs.ROcited by 91

Not on the map. Why?

Abstract (arXiv)

We consider the problem of Vision-and-Language Navigation (VLN). The majority of current methods for VLN are trained end-to-end using either unstructured memory such as LSTM, or using cross-modal attention over the egocentric observations of the agent. In contrast to other works, our key insight is that the association between language and vision is stronger when it occurs in explicit spatial representations. In this work, we propose a cross-modal map learning model for vision-and-language navigation that first learns to predict the top-down semantics on an egocentric map for both observed and unobserved regions, and then predicts a path towards the goal as a set of waypoints. In both cases, the prediction is informed by the language through cross-modal attention mechanisms. We experimentally test the basic hypothesis that language-driven navigation can be solved given a map, and then show competitive results on the full VLN-CE benchmark.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

49 of 64 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
37 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  3. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  7. 2020Learning to Explore using Active Neural SLAMcited by 308
  8. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  9. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  10. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  11. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  12. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  13. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  14. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  15. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  16. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  17. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  18. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  19. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  20. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  21. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  22. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  23. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  24. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  25. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  26. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 57
  27. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  28. 2021Learning to Map for Active Semantic Goal Navigationcited by 47
  29. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 43
  30. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
  31. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  32. 2020Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigationcited by 34
  33. 2022Uncertainty-driven Planner for Exploration and Navigationcited by 31
  34. 2021The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigationcited by 27
  35. 2019Multi-modal Discriminative Model for Vision-and-Language Navigationcited by 17
  36. 2019Simultaneous Mapping and Target Driven Navigationcited by 11
  37. 2020Cross-Modality Relevance for Reasoning on Language and Visioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 16; 2024: 12; 2025: 20; 2026: 39.

91 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  5. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  6. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  7. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  8. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  9. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  10. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  11. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  12. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
38 more
  1. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  2. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  3. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  4. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  5. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  6. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 40
  7. 2022Iterative Vision-and-Language Navigationcited by 34
  8. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  9. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  10. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  11. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  12. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  13. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  14. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  15. 2023Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigationcited by 18
  16. 2024NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigationcited by 18
  17. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  18. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  19. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  20. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  21. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  22. 2024VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigationcited by 14
  23. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  24. 2023MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigationcited by 11
  25. 2023Vision and Language Navigation in the Real World via Online Visual Language Mappingcited by 11
  26. 2023ESceme: Vision-and-Language Navigation with Episodic Scene Memorycited by 9
  27. 2025ForesightNav: Learning Scene Imagination for Efficient Explorationcited by 9
  28. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  29. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigationcited by 7
  30. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  31. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 6
  32. 2023Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural languagecited by 5
  33. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  34. 2026textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentationcited by 5
  35. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  36. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Promptingcited by 4
  37. 2026MapDream: Task-Driven Map Learning for Vision-Language Navigationcited by 4
  38. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4

50 most cited of 91 All 91 (JSON)