mlsanityData to 7 Oct 2026

Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation

arXiv 1905.12255 ↗First posted 29 May 2019cs.AI cs.CLcited by 129

Not on the map. Why?

Abstract (arXiv)

Advances in learning and representations have reinvigorated work that connects language to other modalities. A particularly exciting direction is Vision-and-Language Navigation(VLN), in which agents interpret natural language instructions and visual scenes to move through environments and reach goals. Despite recent progress, current research leaves unclear how much of a role language understanding plays in this task, especially because dominant evaluation metrics have focused on goal completion rather than the sequence of actions corresponding to the instructions. Here, we highlight shortcomings of current metrics for the Room-to-Room dataset (Anderson et al.,2018b) and propose a new metric, Coverage weighted by Length Score (CLS). We also show that the existing paths in the dataset are not ideal for evaluating instruction following because they are direct-to-goal shortest paths. We join existing short paths to form more challenging extended paths to create a new data set, Room-for-Room (R4R). Using R4R and CLS, we show that agents that receive rewards for instruction fidelity outperform agents that focus on goal completion.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 18; 2023: 22; 2024: 28; 2025: 21; 2026: 17.

129 papers.

Sort:

By area:

  1. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  5. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  6. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  7. 2023AerialVLN: Vision-and-Language Navigation for UAVscited by 117
  8. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  9. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  10. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  11. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  12. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
38 more
  1. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  2. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  5. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  6. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  7. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  8. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  9. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  10. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  11. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  12. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  13. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  14. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  15. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  16. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  17. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  18. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  19. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  20. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  21. 2022Reinforced Structured State-Evolution for Vision-Language Navigationcited by 36
  22. 2022Iterative Vision-and-Language Navigationcited by 34
  23. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  24. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  25. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  26. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  27. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  28. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  29. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  30. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  31. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
  32. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  33. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  34. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20
  35. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  36. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  37. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  38. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18

50 most cited of 129 All 129 (JSON)