mlsanityData to 7 Oct 2026

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

arXiv 2210.07506 ↗First posted 14 Oct 2022cs.CVcited by 86

Not on the map. Why?

Abstract (arXiv)

We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (e.g., color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. Code is available at https://github.com/PeihaoChen/WS-MGMap.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

46 of 65 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  10. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  11. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
34 more
  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  2. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  3. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  4. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  5. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  6. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  7. 2017Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localizationcited by 207
  8. 2020Neural Topological SLAM for Visual Navigationcited by 193
  9. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  10. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  11. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  12. 2018Adversarial Complementary Learning for Weakly Supervised Object Localizationcited by 166
  13. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  14. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  15. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  16. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  17. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  18. 2017Interpreting Deep Visual Representations via Network Dissectioncited by 103
  19. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  20. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  21. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  22. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  23. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 57
  24. 2021Shape or Texture: Understanding Discriminative Features in CNNscited by 52
  25. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  26. 2021Learning to Map for Active Semantic Goal Navigationcited by 47
  27. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 47
  28. 2019Scaling Local Control to Large-Scale Topological Navigationcited by 37
  29. 2020Learning to plan with uncertain topological mapscited by 29
  30. 2017Soft Proposal Networks for Weakly Supervised Object Localizationcited by 21
  31. 2018Geometry-Aware Recurrent Neural Networks for Active Visual Recognitioncited by 20
  32. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 13
  33. 2020High-Speed Robot Navigation using Predicted Occupancy Mapscited by 10
  34. 2017Progressive Representation Adaptation for Weakly Supervised Object Localizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 19; 2025: 19; 2026: 30.

86 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  5. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  6. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  7. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  8. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  9. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  10. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  11. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  12. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
38 more
  1. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  2. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  3. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  4. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  5. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  6. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  7. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  8. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  9. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  10. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  11. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  12. 2023CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detectioncited by 28
  13. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  14. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  15. 2023FGPrompt: Fine-grained Goal Prompting for Image-goal Navigationcited by 21
  16. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  17. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  18. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  19. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  20. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  21. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 18
  22. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  23. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  24. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  25. 2023Hard Sample Matters a Lot in Zero-Shot Quantizationcited by 15
  26. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  27. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  28. 2024VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigationcited by 14
  29. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  30. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  31. 2023Vision and Language Navigation in the Real World via Online Visual Language Mappingcited by 11
  32. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  33. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  34. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigationcited by 7
  35. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  36. 2025STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimizationcited by 6
  37. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 6
  38. 2024Narrowing the Gap between Vision and Action in Navigationcited by 5

50 most cited of 86 All 86 (JSON)