Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?
Not on the map. Why?
Abstract (arXiv)
Data augmentation via back-translation is common when pretraining Vision-and-Language Navigation (VLN) models, even though the generated instructions are noisy. But: does that noise matter? We find that nonsensical or irrelevant language instructions during pretraining can have little effect on downstream performance for both HAMT and VLN-BERT on R2R, and is still better than only using clean, human data. To underscore these results, we concoct an efficient augmentation method, Unigram + Object, which generates nonsensical instructions that nonetheless improve downstream performance. Our findings suggest that what matters for VLN R2R pretraining is the quantity of visual trajectories, not the quality of instructions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 23 references that are in the map. First posted on arXiv 28 Nov 2023.
Builds on
21 of 23 references.
References · 21
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
- 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
- 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
- 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
- 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
- 2019Vision-and-Dialog Navigationcited by 250
- 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
- 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
- 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
9 more
- 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
- 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
- 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
- 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
- 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 64
- 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
- 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 48
- 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
- 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.