mlsanityData to 7 Oct 2026

KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigation

arXiv 2303.15796 ↗First posted 28 Mar 2023cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

Vision-and-language navigation (VLN) is the task to enable an embodied agent to navigate to a remote location following the natural language instruction in real scenes. Most of the previous approaches utilize the entire features or object-centric features to represent navigable candidates. However, these representations are not efficient enough for an agent to perform actions to arrive the target location. As knowledge provides crucial information which is complementary to visible content, in this paper, we propose a Knowledge Enhanced Reasoning Model (KERM) to leverage knowledge to improve agent navigation ability. Specifically, we first retrieve facts (i.e., knowledge described by language descriptions) for the navigation views based on local regions from the constructed knowledge base. The retrieved facts range from properties of a single object (e.g., color, shape) to relationships between objects (e.g., action, spatial position), providing crucial information for VLN. We further present the KERM which contains the purification, fact-aware interaction, and instruction-guided aggregation modules to integrate visual, history, instruction, and fact features. The proposed KERM can automatically select and gather crucial and relevant cues, obtaining more accurate action prediction. Experimental results on the REVERIE, R2R, and SOON datasets demonstrate the effectiveness of the proposed method.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

31 of 40 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  9. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  10. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  11. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  12. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
19 more
  1. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  2. 2019Vision-and-Dialog Navigationcited by 246
  3. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  4. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  5. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  6. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  7. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  8. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  9. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  10. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  11. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  12. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  13. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  14. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  15. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  16. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  17. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  18. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  19. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 14; 2026: 5.

38 papers.

Sort:

By area:

  1. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  2. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  3. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  4. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  5. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  6. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  7. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  8. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  9. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  10. 2024OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representationcited by 17
  11. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  12. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
26 more
  1. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  2. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  3. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  4. 2024Augmented Commonsense Knowledge for Remote Object Groundingcited by 7
  5. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  6. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  7. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  8. 2025STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimizationcited by 6
  9. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 4
  10. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  11. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigationcited by 4
  12. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  13. 2025Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments with a Hierarchical Spatial-Cognition Long-Short Memory Systemcited by 3
  14. 2026LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigationcited by 3
  15. 2024Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigationcited by 2
  16. 2024AIGeN: An Adversarial Approach for Instruction Generation in VLNcited by 2
  17. 2024NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired Peoplecited by 2
  18. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2
  19. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigationcited by 1
  20. 2025Efficient and Generalizable Environmental Understanding for Visual Navigationcited by 1
  21. 2024Temporal-Spatial Object Relations Modeling for Vision-and-Language Navigationcited by 0
  22. 2025Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Modelscited by 0
  23. 2025Landmark-Guided Knowledge for Vision-and-Language Navigationcited by 0
  24. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0
  25. 2026ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Groundingcited by 0
  26. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
  2. 2025Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation
  3. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
  4. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
  5. 2026Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos
  6. 2025A Navigation Framework Utilizing Vision-Language Models
  7. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  8. 2026VISOR: VIsual Spatial Object Reasoning for Language-driven Object Navigation