mlsanityData to 7 Oct 2026

Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation

arXiv 2409.17313 ↗First posted 25 Sep 2024cs.CV cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

19 of 28 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  6. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  7. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  8. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  10. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  11. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  12. 2019Vision-and-Dialog Navigationcited by 246
7 more
  1. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  2. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  3. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  4. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  5. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  6. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  7. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  2. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  3. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  4. 2025Fine-Tuning Vision-Language Models for Visual Navigation Assistancecited by 1
  5. 2026Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigationcited by 1
  6. 2026OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agentscited by 1
  7. 2026SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigationcited by 0