NAVCON: A Cognitively Inspired and Linguistically Grounded Corpus for Vision and Language Navigation
Not on the map. Why?
Abstract (arXiv)
We present NAVCON, a large-scale annotated Vision-Language Navigation (VLN) corpus built on top of two popular datasets (R2R and RxR). The paper introduces four core, cognitively motivated and linguistically grounded, navigation concepts and an algorithm for generating large-scale silver annotations of naturally occurring linguistic realizations of these concepts in navigation instructions. We pair the annotated instructions with video clips of an agent acting on these instructions. NAVCON contains 236, 316 concept annotations for approximately 30, 0000 instructions and 2.7 million aligned images (from approximately 19, 000 instructions) showing what the agent sees when executing an instruction. To our knowledge, this is the first comprehensive resource of navigation concepts. We evaluated the quality of the silver annotations by conducting human evaluation studies on NAVCON samples. As further validation of the quality and usefulness of the resource, we trained a model for detecting navigation concepts and their linguistic realizations in unseen instructions. Additionally, we show that few-shot learning with GPT-4o performs well on this task using large-scale silver annotations of NAVCON.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 48 references that are in the map. First posted on arXiv 17 Dec 2024.
Builds on
11 of 48 references.
References · 11
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
- 2021FILM: Following Instructions in Language with Modular Methodscited by 119
- 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
- 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
- 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
- 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
- 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
- 2021Towards Navigation by Reasoning over Spatial Configurationscited by 14
- 2024Semantic Map-based Generation of Navigation Instructionscited by 6
Built on by
Not cited yet.