To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigation
Not on the map. Why?
Abstract (arXiv)
Recent research in Vision Language Navigation (VLN) has overlooked the development of agents' inquisitive abilities, which allow them to ask clarifying questions when instructions are incomplete. This paper addresses how agents can recognize "when" they lack sufficient information, without focusing on "what" is missing, particularly in VLN tasks with vague instructions. Equipping agents with this ability enhances efficiency by reducing potential digressions and seeking timely assistance. The challenge in identifying such uncertain points is balancing between being overly cautious (high recall) and overly confident (high precision). We propose an attention-based instruction-vagueness estimation module that learns associations between instructions and the agent's trajectory. By leveraging instruction-to-path alignment information during training, the module's vagueness estimation performance improves by around 52% in terms of precision-recall balance. In our ablative experiments, we also demonstrate the effectiveness of incorporating this additional instruction-to-path attention network alongside the cross-modal attention networks within the navigator module. Our results show that the attention scores from the instruction-to-path attention network serve as better indicators for estimating vagueness.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 6 Nov 2024.
Builds on
24 of 27 references.
References · 24
- 2017Attention Is All You Needcited by 60,361
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
- 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2018On Evaluation of Embodied Navigation Agentscited by 547
- 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
- 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
- 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
- 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
- 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
- 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
12 more
- 2019Vision-and-Dialog Navigationcited by 250
- 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 66
- 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
- 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 40
- 2021Vision-Language Navigation: A Survey and Taxonomycited by 34
- 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
- 2023CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agentscited by 30
- 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
- 2024DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learningcited by 7
- 2022ULN: Towards Underspecified Vision-and-Language Navigationcited by 5
- 2024Language to Map: Topological map generation from natural language path instructionscited by 3
- 2024AIGeN: An Adversarial Approach for Instruction Generation in VLNcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.