mlsanityData to 7 Oct 2026

Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation model

arXiv 2312.00151 ↗First posted 30 Nov 2023cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

The challenging task of Vision-and-Language Navigation (VLN) requires embodied agents to follow natural language instructions to reach a goal location or object (e.g. `walk down the hallway and turn left at the piano'). For agents to complete this task successfully, they must be able to ground objects referenced into the instruction (e.g.`piano') into the visual scene as well as ground directional phrases (e.g.`turn left') into actions. In this work we ask the following question -- to what degree are spatial and directional language cues informing the navigation model's decisions? We propose a series of simple masking experiments to inspect the model's reliance on different parts of the instruction. Surprisingly we uncover that certain top performing models rely only on the noun tokens of the instructions. We propose two training methods to alleviate this concerning limitation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

33 of 41 references.

References · 33

  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  7. 2018On Evaluation of Embodied Navigation Agentscited by 526
  8. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2017Embodied Question Answeringcited by 335
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
21 more
  1. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  2. 2019Vision-and-Dialog Navigationcited by 246
  3. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  4. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  5. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  6. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  7. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  8. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  9. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  10. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  11. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  12. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  13. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 60
  14. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  15. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  16. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  17. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  18. 2018Shifting the Baseline: Single Modality Performance on Visual Navigation & QAcited by 30
  19. 2020Where Are You? Localization from Embodied Dialogcited by 24
  20. 2019Multi-modal Discriminative Model for Vision-and-Language Navigationcited by 17
  21. 2022Transformer-based Localization from Embodied Dialog with Large-scale Pre-trainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigationcited by 11
  2. 2026VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructionscited by 2
  3. 2024I2EDL: Interactive Instruction Error Detection and Localizationcited by 1