mlsanityData to 7 Oct 2026

Reinforcement Learning with Action-Free Pre-Training from Videos

arXiv 2203.13880 ↗First posted 25 Mar 2022cs.CV cs.AIcited by 107

Not on the map. Why?

Cited by 30 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent unsupervised pre-training methods have shown to be effective on language and vision domains by learning useful representations for multiple downstream tasks. In this paper, we investigate if such unsupervised pre-training methods can also be effective for vision-based reinforcement learning (RL). To this end, we introduce a framework that learns representations useful for understanding the dynamics via generative pre-training on videos. Our framework consists of two phases: we pre-train an action-free latent video prediction model, and then utilize the pre-trained representations for efficiently learning action-conditional world models on unseen environments. To incorporate additional action inputs during fine-tuning, we introduce a new architecture that stacks an action-conditional latent prediction model on top of the pre-trained action-free prediction model. Moreover, for better exploration, we propose a video-based intrinsic bonus that leverages pre-trained representations. We demonstrate that our framework significantly improves both final performances and sample-efficiency of vision-based RL in a variety of manipulation and locomotion tasks. Code is available at https://github.com/younggyoseo/apv.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 94 of its 119 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

94 of 119 references.

References · 94

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
38 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2018Exploration by Random Network Distillationcited by 966
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  13. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  14. 2019Solving Rubik's Cube with a Robot Handcited by 699
  15. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  16. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  17. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  18. 2019Model-Based Reinforcement Learning for Ataricited by 495
  19. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  20. 2020Reinforcement Learning with Augmented Datacited by 452
  21. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  22. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  23. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  24. 2016Generating Videos with Scene Dynamicscited by 376
  25. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  26. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  27. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  28. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  29. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  30. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  31. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  32. 2019Self-Supervised Exploration via Disagreementcited by 263
  33. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  34. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  35. 2017Count-Based Exploration with Neural Density Modelscited by 255
  36. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  37. 2022Masked Visual Pre-training for Motor Controlcited by 246
  38. 2018Stochastic Video Generation with a Learned Priorcited by 244

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 33; 2024: 29; 2025: 20; 2026: 12.

107 papers.

Sort:

By area:

  1. 2023Learning Interactive Real-World Simulatorscited by 440
  2. 2022Transformers are Sample-Efficient World Modelscited by 317
  3. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  4. 2022Masked World Models for Visual Controlcited by 169
  5. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  6. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  7. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  8. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  9. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  10. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  11. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  12. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
38 more
  1. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  2. 2024Foundation Policies with Hilbert Representationscited by 89
  3. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  4. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  5. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  6. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  7. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  8. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  9. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  10. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  11. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  12. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  13. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40
  14. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  15. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  16. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  17. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31
  18. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  19. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  20. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  21. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  22. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 25
  23. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  24. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  25. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  26. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  27. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  28. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  29. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  30. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  31. 2023Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Trainingcited by 18
  32. 2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15
  33. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  34. 2023Learning Video-Conditioned Policies for Unseen Manipulation Taskscited by 15
  35. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  36. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamicscited by 14
  37. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  38. 2024Diffusion Imitation from Observationcited by 12

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Latent Action Pretraining from Videos
  2. 2026ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
  3. 2026Native Action-Prior Learning from Videos for World Action Models
  4. 2026Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs
  5. 2026Native Video-Action Pretraining for Generalizable Robot Control
  6. 2026DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning
  7. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
  8. 2026JEPA-VLA: Video Predictive Embedding is Needed for VLA Models