# ULN: Towards Underspecified Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/2210.10020 · HTML: https://mlsanity.com/paper/2210.10020/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 59 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 30 of 52)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2202.11742/) (2022)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation](https://mlsanity.com/paper/1905.12255/) (2019)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)

## Built on by (most cited first, 5 of 5)

- [Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2404.01943/) (2024)
- [Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning](https://mlsanity.com/paper/2410.03321/) (2024)
- [Semantic Map-based Generation of Navigation Instructions](https://mlsanity.com/paper/2403.19603/) (2024)
- [To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigation](https://mlsanity.com/paper/2411.05831/) (2024)
- [IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instruction](https://mlsanity.com/paper/2605.23187/) (2026)

## Abstract (arXiv)

Vision-and-Language Navigation (VLN) is a task to guide an embodied agent moving to a target position using language instructions. Despite the significant performance improvement, the wide use of fine-grained instructions fails to characterize more practical linguistic variations in reality. To fill in this gap, we introduce a new setting, namely Underspecified vision-and-Language Navigation (ULN), and associated evaluation datasets. ULN evaluates agents using multi-level underspecified instructions instead of purely fine-grained or coarse-grained, which is a more realistic and general setting. As a primary step toward ULN, we propose a VLN framework that consists of a classification module, a navigation agent, and an Exploitation-to-Exploration (E2E) module. Specifically, we propose to learn Granularity Specific Sub-networks (GSS) for the agent to ground multi-level instructions with minimal additional parameters. Then, our E2E module estimates grounding uncertainty and conducts multi-step lookahead exploration to improve the success rate further. Experimental results show that existing VLN models are still brittle to multi-level language underspecification. Our framework is more robust and outperforms the baselines on ULN by ~10% relative success rate across all levels.