mlsanityData to 7 Oct 2026

On the Evaluation of Vision-and-Language Navigation Instructions

arXiv 2101.10504 ↗First posted 26 Jan 2021cs.AI cs.CL cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automatically generated navigation instructions. However, existing instruction generators have not been comprehensively evaluated, and the automatic evaluation metrics used to develop them have not been validated. Using human wayfinders, we show that these generators perform on par with or only slightly better than a template-based generator and far worse than human instructors. Furthermore, we discover that BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions. To improve instruction evaluation, we propose an instruction-trajectory compatibility model that operates without reference instructions. Our model shows the highest correlation with human wayfinding outcomes when scoring individual instructions. For ranking instruction generation systems, if reference instructions are available we recommend using SPICE.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 26 Jan 2021.

Builds on

26 of 40 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2018On Evaluation of Embodied Navigation Agentscited by 526
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
14 more
  1. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  3. 2019Vision-and-Dialog Navigationcited by 246
  4. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  5. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  6. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  7. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  8. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  9. 2017Towards Diverse and Natural Image Descriptions via a Conditional GANcited by 59
  10. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  11. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
  12. 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22
  13. 2020Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street Viewcited by 18
  14. 2019VALAN: Vision and Language Agent Navigationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 10; 2023: 8; 2024: 10; 2025: 3; 2026: 4.

45 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  3. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  4. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  5. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  6. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  7. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  8. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  9. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  10. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  11. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  12. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
33 more
  1. 2022FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigationcited by 22
  2. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  3. 2021Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigationcited by 20
  4. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  5. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  6. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  7. 2024VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigationcited by 14
  8. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  9. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 11
  10. 2024Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigationcited by 11
  11. 2021CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigationcited by 8
  12. 2023PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigationcited by 8
  13. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 8
  14. 2022Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Modelscited by 7
  15. 2023Hallucination Detection for Grounded Instruction Generationcited by 7
  16. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  17. 2024Semantic Map-based Generation of Navigation Instructionscited by 6
  18. 2021Natural Language for Human-Robot Collaboration: Problems Beyond Language Groundingcited by 5
  19. 2021Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructionscited by 4
  20. 2021PanGEA: The Panoramic Graph Environment Annotation Toolkitcited by 3
  21. 2023Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation modelcited by 3
  22. 2024Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generationcited by 3
  23. 2025Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generationcited by 3
  24. 2022Bridging the visual gap in VLN via semantically richer instructionscited by 2
  25. 2024Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigationcited by 2
  26. 2024Embodied AI in Mobile Robots: Coverage Path Planning with Large Language Modelscited by 2
  27. 2024NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired Peoplecited by 2
  28. 2023Core Challenges in Embodied Vision-Language Planningcited by 0
  29. 2025Can Hallucination Correction Improve Video-Language Alignment?cited by 0
  30. 2026Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluationcited by 0
  31. 2026GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topologycited by 0
  32. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  33. 2026Time-Aware Assistive Navigationcited by 0