mlsanityData to 14 Oct 2026

Playing hard exploration games by watching YouTube

arXiv 1805.11592 ↗First posted 29 May 2018cs.LG cs.AI cs.CVcited by 99

Not on the map. Why?

Cited by 45 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games Montezuma's Revenge, Pitfall! and Private Eye for the first time, even if the agent is not presented with any environment rewards.

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 4; 2019: 6; 2020: 5; 2021: 25; 2022: 17; 2023: 18; 2024: 13; 2025: 7; 2026: 4.

99 papers.

Sort:

By area:

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  3. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  4. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  5. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  7. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
  8. 2020Primal Wasserstein Imitation Learningcited by 121
  9. 2023Learning to Act without Actionscited by 121
  10. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
  11. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  12. 2019Recent Advances in Imitation Learning from Observationcited by 93
38 more
  1. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  2. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 82
  3. 2019State Alignment-based Imitation Learningcited by 73
  4. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  5. 2021Off-Policy Imitation Learning from Observationscited by 70
  6. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  7. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 58
  8. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 51
  9. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 51
  10. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  11. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 49
  12. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  13. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 46
  14. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  15. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  16. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  17. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 44
  18. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 43
  19. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  20. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  21. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  22. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  23. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 31
  24. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  25. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 28
  26. 2018Inverse reinforcement learning for video gamescited by 26
  27. 2018Learning from Demonstration in the Wildcited by 25
  28. 2022Large-Scale Retrieval for Reinforcement Learningcited by 24
  29. 2019Imitation Learning from Video by Leveraging Proprioceptioncited by 23
  30. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  31. 2021Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learningcited by 23
  32. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 22
  33. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  34. 2024Diffusion-Reward Adversarial Imitation Learningcited by 20
  35. 2021There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learningcited by 18
  36. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  37. 2021Object-Aware Regularization for Addressing Causal Confusion in Imitation Learningcited by 17
  38. 2022Learning Reward Functions for Robotic Manipulation by Observing Humanscited by 17

50 most cited of 99 All 99 (JSON)