mlsanityData to 7 Oct 2026

Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigation

arXiv 2603.26859 ↗First posted 27 Mar 2026cs.CV cs.AI eess.IVcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) requires an agent to navigate through complex unseen environments based on natural language instructions. However, existing methods often struggle to effectively capture key semantic cues and accurately align them with visual observations. To address this limitation, we propose Beyond Textual Knowledge (BTK), a VLN framework that synergistically integrates environment-specific textual knowledge with generative image knowledge bases. BTK employs Qwen3-4B to extract goal-related phrases and utilizes Flux-Schnell to construct two large-scale image knowledge bases: R2R-GP and REVERIE-GP. Additionally, we leverage BLIP-2 to construct a large-scale textual knowledge base derived from panoramic views, providing environment-specific semantic cues. These multimodal knowledge bases are effectively integrated via the Goal-Aware Augmentor and Knowledge Augmentor, significantly enhancing semantic grounding and cross-modal alignment. Extensive experiments on the R2R dataset with 7,189 trajectories and the REVERIE dataset with 21,702 instructions demonstrate that BTK significantly outperforms existing baselines. On the test unseen splits of R2R and REVERIE, SR increased by 5% and 2.07% respectively, and SPL increased by 4% and 3.69% respectively. The source code is available at https://github.com/yds3/IPM-BTK/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

28 of 43 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  7. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  8. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  9. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  10. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  11. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  12. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
16 more
  1. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  2. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  3. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  4. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  5. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  6. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  7. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  8. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  9. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  10. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  11. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  12. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  13. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  14. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  15. 2023ESceme: Vision-and-Language Navigation with Episodic Scene Memorycited by 9
  16. 2024Augmented Commonsense Knowledge for Remote Object Groundingcited by 7

Built on by

Not cited yet.