mlsanityData to 7 Oct 2026

NavHint: Vision and Language Navigation Agent with a Hint Generator

arXiv 2402.02559 ↗First posted 4 Feb 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Existing work on vision and language navigation mainly relies on navigation-related losses to establish the connection between vision and language modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment. In our work, we provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions. The hint generator assists the navigation agent in developing a global understanding of the visual environment. It directs the agent's attention toward related navigation details, including the relevant sub-instruction, potential challenges in recognition and ambiguities in grounding, and the targeted viewpoint description. To train the hint generator, we construct a synthetic dataset based on landmarks in the instructions and visible and distinctive objects in the visual environment. We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics. The experimental results demonstrate that generating hints not only enhances the navigation performance but also helps improve the interpretability of the agent's actions.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 36 of its 40 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

36 of 40 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  6. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  7. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  8. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  9. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  10. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  11. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  12. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
24 more
  1. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  2. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  5. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  6. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  7. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  8. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  9. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  10. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  11. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  12. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  13. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  14. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  15. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  16. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  17. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  18. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  19. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  20. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  21. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  22. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  23. 2021Contrastive Instruction-Trajectory Learning for Vision-Language Navigationcited by 16
  24. 2021Towards Navigation by Reasoning over Spatial Configurationscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 2; 2026: 3.

12 papers.

  1. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  2. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 38
  3. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  4. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  5. 2024Language-guided Hierarchical Fine-grained Image Forgery Detection and Localizationcited by 16
  6. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 13
  7. 2024Narrowing the Gap between Vision and Action in Navigationcited by 5
  8. 2026On the Holistic Approach for Detecting Human Image Forgerycited by 5
  9. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMscited by 3
  10. 2026VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructionscited by 2
  11. 2024To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigationcited by 1
  12. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1