mlsanityData to 7 Oct 2026

Learning Vision-and-Language Navigation from YouTube Videos

arXiv 2307.11984 ↗First posted 22 Jul 2023cs.CV cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Vision-and-language navigation (VLN) requires an embodied agent to navigate in realistic 3D environments using natural language instructions. Existing VLN methods suffer from training on small-scale environments or unreasonable path-instruction datasets, limiting the generalization to unseen environments. There are massive house tour videos on YouTube, providing abundant real navigation experiences and layout information. However, these videos have not been explored for VLN before. In this paper, we propose to learn an agent from these videos by creating a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it. To achieve this, we have to tackle the challenges of automatically constructing path-instruction pairs and exploiting real layout knowledge from raw and unlabeled videos. To address these, we first leverage an entropy-based method to construct the nodes of a path trajectory. Then, we propose an action-aware generator for generating instructions from unlabeled trajectories. Last, we devise a trajectory judgment pretext task to encourage the agent to mine the layout knowledge. Experimental results show that our method achieves state-of-the-art performance on two popular benchmarks (R2R and REVERIE). Code is available at https://github.com/JeremyLinky/YouTube-VLN

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 22 Jul 2023.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2017Mask R-CNNcited by 7,259
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
38 more
  1. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  4. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  5. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  6. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  7. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  8. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  10. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  11. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  12. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  13. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  14. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  15. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  16. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  17. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  18. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  19. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  20. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  21. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  22. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  23. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  24. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  25. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  26. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  27. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  28. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  29. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  30. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  31. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  32. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
  33. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  34. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  35. 2023CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detectioncited by 28
  36. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  37. 2022Visual-Language Navigation Pretraining via Prompt-based Environmental Self-explorationcited by 25
  38. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 17; 2026: 12.

47 papers.

Sort:

By area:

  1. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  2. 2023Large Language Models for Robotics: A Surveycited by 132
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  5. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  6. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  7. 2023FGPrompt: Fine-grained Goal Prompting for Image-goal Navigationcited by 21
  8. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  9. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18
  10. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  11. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  12. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
35 more
  1. 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 13
  2. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 11
  3. 2024Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environmentscited by 10
  4. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  5. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  6. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  7. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  8. 2025HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactionscited by 6
  9. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  10. 2025NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularizationcited by 5
  11. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 4
  12. 2025VPN: Visual Prompt Navigationcited by 4
  13. 2025AgriVLN: Vision-and-Language Navigation for Agricultural Robotscited by 4
  14. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  15. 2023Contrastive Vision-Language Alignment Makes Efficient Instruction Learnercited by 3
  16. 2025Image-Goal Navigation Using Refined Feature Guidance and Scene Graph Enhancementcited by 3
  17. 2024NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired Peoplecited by 2
  18. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2
  19. 2026SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigationcited by 2
  20. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigationcited by 1
  21. 2025YouTube-Occ: Learning Indoor 3D Semantic Occupancy Prediction from YouTube Videoscited by 1
  22. 2025User-Feedback-Driven Adaptation for Vision-and-Language Navigationcited by 1
  23. 2026All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptationcited by 1
  24. 2026Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memorycited by 1
  25. 2024MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modelingcited by 0
  26. 2025Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Modelscited by 0
  27. 2025TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Makingcited by 0
  28. 2026Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videoscited by 0
  29. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0
  30. 2026Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigationcited by 0
  31. 2026Lifting Unlabeled Internet-level Data for 3D Scene Understandingcited by 0
  32. 2026Advancing DialNav through Automatic Embodied Dialog Augmentationcited by 0
  33. 2026FlowDec: Temporal Conditional Flow Decorruptor for Robust Continuous Vision-Language Navigationcited by 0
  34. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  35. 2026Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Modelscited by 0