mlsanityData to 7 Oct 2026

Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

arXiv 2206.11795 ↗First posted 23 Jun 2022cs.LG cs.AIcited by 338

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision domains such as robotics, video games, and computer use, publicly available data does not contain the labels required to train behavioral priors in the same way. We extend the internet-scale pretraining paradigm to sequential decision domains through semi-supervised imitation learning wherein agents learn to act by watching online unlabeled videos. Specifically, we show that with a small amount of labeled data we can train an inverse dynamics model accurate enough to label a huge unlabeled source of online data -- here, online videos of people playing Minecraft -- from which we can then train a general behavioral prior. Despite using the native human interface (mouse and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-shot capabilities and that it can be fine-tuned, with both imitation learning and reinforcement learning, to hard-exploration tasks that are impossible to learn from scratch via reinforcement learning. For many tasks our models exhibit human-level performance, and we are the first to report computer agents that can craft diamond tools, which can take proficient humans upwards of 20 minutes (24,000 environment actions) of gameplay to accomplish.

Key facts. Cited by 338 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 66 of its 94 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

66 of 94 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning7 Dec
  2. 2019Recent Advances in Imitation Learning from Observation30 May
  3. 2018Imitating Latent Policies from Observation21 May
  4. 2018Behavioral Cloning from Observation4 May

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2017Proximal Policy Optimization Algorithmscited by 14,441
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2014How transferable are features in deep neural networks?cited by 1,216
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2016End to End Learning for Self-Driving Carscited by 1,000
  12. 2021Scaling Vision Transformerscited by 993
  13. 2018Exploration by Random Network Distillationcited by 966
  14. 2022A Generalist Agentcited by 802
  15. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  16. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  17. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  18. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  19. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  20. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  21. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  22. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  23. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  24. 2020First return, then explorecited by 256
  25. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  26. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  27. 2017Teacher-Student Curriculum Learningcited by 197
  28. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  29. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  30. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  31. 2020Phasic Policy Gradientcited by 121
  32. 2018Zero-Shot Visual Imitationcited by 114
  33. 2018Playing hard exploration games by watching YouTubecited by 98
  34. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  35. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  36. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  37. 2022A data-driven approach for learning to control computerscited by 87
  38. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 89; 2024: 91; 2025: 72; 2026: 67.

338 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023Mastering Diverse Domains through World Modelscited by 1,193
  4. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  5. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  8. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2023Language Models can Solve Computer Taskscited by 371
38 more
  1. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 2023Tool Learning with Foundation Modelscited by 302
  4. 2024Latent Action Pretraining from Videoscited by 302
  5. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  6. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  9. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  10. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  11. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  12. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  13. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  14. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  15. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  16. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  17. 2023Getting aligned on representational alignmentcited by 142
  18. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  19. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  20. 2025Training Agents Inside of Scalable World Modelscited by 122
  21. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  22. 2023Learning to Act without Actionscited by 116
  23. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  24. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  25. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  26. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  27. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 93
  28. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  29. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  30. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  31. 2023MindAgent: Emergent Gaming Interactioncited by 87
  32. 2024Video as the New Language for Real-World Decision Makingcited by 86
  33. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  34. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  35. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  36. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 79
  37. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  38. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75

50 most cited of 338 All 338 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets
  2. 2026Learning Latent Action World Models In The Wild
  3. 2024Lifelong Learning of Video Diffusion Models From a Single Video Stream
  4. 2025NIL: No-data Imitation Learning by Leveraging Pre-trained Video Diffusion Models
  5. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
  6. 2022Reinforcement Learning with Action-Free Pre-Training from Videos
  7. 2025Learning from Online Videos at Inference Time for Computer-Use Agents
  8. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets