mlsanityData to 7 Oct 2026

Semantic Visual Navigation by Watching YouTube Videos

arXiv 2006.10034 ↗First posted 17 Jun 2020cs.CV cs.AI cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos don't come with labels for actions or goals, and may not even showcase optimal behavior. Our method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning from passive data is able to learn meaningful semantic cues for navigation. These cues, when used in a hierarchical navigation policy, lead to improved efficiency at the ObjectGoal task in visually realistic simulations. We observe a relative improvement of 15-83% over end-to-end RL, behavior cloning, and classical methods, while using minimal direct interaction.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 11; 2023: 12; 2024: 10; 2025: 11; 2026: 4.

60 papers.

Sort:

By area:

  1. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  2. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  3. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  4. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  5. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  6. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  7. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  8. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  9. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  10. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  11. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  12. 2024GaussNav: Gaussian Splatting for Visual Navigationcited by 49
38 more
  1. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  2. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  3. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  4. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  7. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  8. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  9. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  10. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28
  11. 2021VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigationcited by 27
  12. 2021A Survey on Human-aware Robot Navigationcited by 23
  13. 2022EgoEnv: Human-centric environment representations from egocentric videocited by 22
  14. 2022PEANUT: Predicting and Navigating to Unseen Targetscited by 22
  15. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  16. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20
  17. 2021Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agentscited by 19
  18. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 18
  19. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  20. 2022SelfD: Self-Learning Large-Scale Driving Policies From the Webcited by 16
  21. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 13
  22. 2023FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicingcited by 13
  23. 2021Environment Predictive Coding for Embodied Agentscited by 12
  24. 2025From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learningcited by 11
  25. 2023Exploitation-Guided Exploration for Semantic Embodied Navigationcited by 9
  26. 2025ForesightNav: Learning Scene Imagination for Efficient Explorationcited by 9
  27. 2025Mobi-π: Mobilizing Your Robot Learning Policycited by 9
  28. 2022Learning Value Functions from Undirected State-only Experiencecited by 8
  29. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  30. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 8
  31. 2022Offline Policy Optimization with Eligible Actionscited by 7
  32. 2023E(2)-Equivariant Graph Planning for Navigationcited by 6
  33. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  34. 2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
  35. 2026LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigationcited by 5
  36. 2024Cognitive Planning for Object Goal Navigation using Generative AI Modelscited by 4
  37. 2026AirHunt: Bridging VLM Semantics and Continuous Planning for Efficient Aerial Object Navigationcited by 4
  38. 2024NOLO: Navigate Only Look Oncecited by 3

50 most cited of 60 All 60 (JSON)