mlsanityData to 14 Oct 2026

Video Prediction Models as Rewards for Reinforcement Learning

arXiv 2305.14343 ↗First posted 23 May 2023cs.LG cs.AI cs.CVcited by 78

Not on the map. Why?

Cited by 21 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https://escontrela.me/viper

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

38 of 50 references.

References · 38

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2022Video Diffusion Modelscited by 2,305
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  10. 2023Mastering Diverse Domains through World Modelscited by 1,253
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  12. 2018Exploration by Random Network Distillationcited by 993
26 more
  1. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  4. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  5. 2018Behavioral Cloning from Observationcited by 492
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  7. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  8. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  11. 2022MAGVIT: Masked Generative Video Transformercited by 346
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  13. 2018Generative Adversarial Imitation from Observationcited by 184
  14. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  15. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  16. 2018Zero-Shot Visual Imitationcited by 117
  17. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  18. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  19. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  20. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  21. 2018Probabilistic Recurrent State-Space Modelscited by 64
  22. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  23. 2020Action and Perception as Divergence Minimizationcited by 32
  24. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  25. 2021AMP: Adversarial Motion Priors for Stylized Physics-Based Character Controlcited by 18
  26. 2021Imitation by Predicting Observationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 35; 2025: 22; 2026: 15.

78 papers.

Sort:

By area:

  1. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  2. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  4. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  5. 2024Navigation World Modelscited by 272
  6. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  7. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  8. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  9. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  10. 2024Video as the New Language for Real-World Decision Makingcited by 87
  11. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  12. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
38 more
  1. 2023Vision-Language Models as a Source of Rewardscited by 66
  2. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 57
  3. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  4. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 31
  7. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  8. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  9. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  10. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 26
  11. 2024VideoAgent: Self-Improving Video Generationcited by 25
  12. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  13. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  14. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  15. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  16. 2025Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulationcited by 15
  17. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  18. 2024Diffusion Imitation from Observationcited by 13
  19. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 13
  20. 2025Solving New Tasks by Adapting Internet Video Knowledgecited by 13
  21. 2025The Serial Scaling Hypothesiscited by 13
  22. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  23. 2024Towards Principled Representation Learning from Videos for Reinforcement Learningcited by 10
  24. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 10
  25. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 9
  26. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learningcited by 9
  27. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  28. 2024Text-Aware Diffusion for Policy Learningcited by 8
  29. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  30. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 8
  31. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  32. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learningcited by 7
  33. 2025Self-Improving Loops for Visual Robotic Planningcited by 7
  34. 2025SPACeR: Self-Play Anchoring with Centralized Reference Modelscited by 7
  35. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  36. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 6
  37. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  38. 2025AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoningcited by 5

50 most cited of 78 All 78 (JSON)