mlsanityData to 14 Oct 2026

Diffusion Reward: Learning Rewards via Conditional Video Diffusion

arXiv 2312.14134 ↗First posted 21 Dec 2023cs.LG cs.CV cs.ROcited by 57

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Learning rewards from expert videos offers an affordable and effective solution to specify the intended behaviors for reinforcement learning (RL) tasks. In this work, we propose Diffusion Reward, a novel framework that learns rewards from expert videos via conditional video diffusion models for solving complex visual RL problems. Our key insight is that lower generative diversity is exhibited when conditioning diffusion on expert trajectories. Diffusion Reward is accordingly formalized by the negative of conditional entropy that encourages productive exploration of expert behaviors. We show the efficacy of our method over robotic manipulation tasks in both simulation platforms and the real world with visual input. Moreover, Diffusion Reward can even solve unseen tasks successfully and effectively, largely surpassing baseline methods. Project page and code: https://diffusion-reward.github.io.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 59 of its 65 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

59 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning8 Jun
  2. 2023Extracting Reward Functions from Diffusion Models1 Jun
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 53

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  11. 2018Exploration by Random Network Distillationcited by 993
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
38 more
  1. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  7. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  8. 2018Behavioral Cloning from Observationcited by 492
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  11. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  12. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  13. 2017Toward Multimodal Image-to-Image Translationcited by 296
  14. 2022Masked Visual Pre-training for Motor Controlcited by 248
  15. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  16. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  17. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  18. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
  19. 2018Generative Adversarial Imitation from Observationcited by 184
  20. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  21. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  22. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  23. 2019Planning with Goal-Conditioned Policiescited by 149
  24. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  25. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  26. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  27. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  28. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  29. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  30. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  31. 2021Off-Policy Imitation Learning from Observationscited by 70
  32. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  33. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  34. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  35. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 57
  36. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  37. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 52
  38. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 22; 2026: 19.

57 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  2. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 116
  3. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  4. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  7. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  8. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 26
  9. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  10. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  11. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 19
  12. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 19
38 more
  1. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  2. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  3. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15
  4. 2024Diffusion Imitation from Observationcited by 13
  5. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 13
  6. 2025Solving New Tasks by Adapting Internet Video Knowledgecited by 13
  7. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 13
  8. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  9. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 10
  10. 2024Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Modelscited by 9
  11. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 9
  12. 2025Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Modelscited by 9
  13. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learningcited by 9
  14. 2026Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goalscited by 9
  15. 2024Text-Aware Diffusion for Policy Learningcited by 8
  16. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 8
  17. 20254D Visual Pre-training for Robot Learningcited by 8
  18. 2025Self-Improving Loops for Visual Robotic Planningcited by 7
  19. 2026Cross-Hand Latent Representation for Vision-Language-Action Modelscited by 7
  20. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  21. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 6
  22. 2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 6
  23. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 5
  24. 2024Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videoscited by 4
  25. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  26. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  27. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  28. 2025Integrating Diffusion-based Multi-task Learning with Online Reinforcement Learning for Robust Quadruped Robot Controlcited by 3
  29. 2025Goal-Driven Reward by Video Diffusion Models for Reinforcement Learningcited by 3
  30. 2026Dreaming the Unseen: World Model-regularized Diffusion Policy for Out-of-Distribution Robustnesscited by 3
  31. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  32. 2024Lifelong Learning of Video Diffusion Models From a Single Video Streamcited by 2
  33. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  34. 2025A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Roboticscited by 2
  35. 2026Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RLcited by 2
  36. 2025ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulationcited by 1
  37. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learningcited by 1
  38. 2026Constraint-Aware Diffusion Priors for High-Fidelity and Versatile Quadruped Locomotioncited by 1

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unpacking the Individual Components of Diffusion Policy
  2. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward
  3. 2024Diffusion-Reward Adversarial Imitation Learning
  4. 2026DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
  5. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
  6. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewards
  7. 2025Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
  8. 2026Guided Discovery of New Behaviors using Diffusion Policies