# DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models

arXiv: https://arxiv.org/abs/2605.15055 · HTML: https://mlsanity.com/paper/2605.15055/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 44 of its 52 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 30 of 44)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation](https://mlsanity.com/paper/2404.04057/) (2024)
- [Unified Reward Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.05236/) (2025)

## Built on by (most cited first, 30 of 40)

- [DanceOPD: On-Policy Generative Field Distillation](https://mlsanity.com/paper/2606.27377/) (2026)
- [OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators](https://mlsanity.com/paper/2607.08766/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model](https://mlsanity.com/paper/2606.17800/) (2026)
- [Qwen-Image-2.0-RL Technical Report](https://mlsanity.com/paper/2606.27608/) (2026)
- [Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging](https://mlsanity.com/paper/2608.03316/) (2026)
- [Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation](https://mlsanity.com/paper/2607.24731/) (2026)
- [Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models](https://mlsanity.com/paper/2608.04349/) (2026)
- [WAM-OPD: On-Policy Distillation for World Action Models](https://mlsanity.com/paper/2608.22364/) (2026)
- [STEP-OPD: Rethinking Output Targets and Internal Dynamics in On-Policy Distillation for Diffusion Models](https://mlsanity.com/paper/2608.04887/) (2026)
- [DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models](https://mlsanity.com/paper/2608.09233/) (2026)
- [Exploring the Performance Frontier of Compact Unified Image Generation Models](https://mlsanity.com/paper/2608.20334/) (2026)
- [ReNFT: Repairing Mode Collapse in Reward Post-Training via Internal Probability-Mass Recalibration](https://mlsanity.com/paper/2609.00061/) (2026)
- [FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models](https://mlsanity.com/paper/2607.24522/) (2026)
- [Latent Reward Registers for Diffusion Preference Alignment](https://mlsanity.com/paper/2608.03929/) (2026)
- [FlowErase-OPD: Multi-Concept Erasure via Anchored On-Policy Distillation in Flow Matching Models](https://mlsanity.com/paper/2608.07620/) (2026)
- [HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models](https://mlsanity.com/paper/2608.13205/) (2026)
- [Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher](https://mlsanity.com/paper/2608.26872/) (2026)
- [Flow3D-OPD: Multi-Teacher On-Policy Distillation for 3D Geometry Generation with Flow-Matching Diffusion Transformer](https://mlsanity.com/paper/2609.07137/) (2026)
- [On-Policy Self-Distillation for Multi-Turn Image Editing](https://mlsanity.com/paper/2609.35611/) (2026)
- [Qwen-Image-Flash: Rethinking the Training Recipe for Few-Step Distillation](https://mlsanity.com/paper/2606.03746/) (2026)
- [ABOPD: Antibody CDR Design via On-Policy Distillation](https://mlsanity.com/paper/2607.18835/) (2026)
- [SelfLift: Accelerating Few-Step Diffusion via Self-Recovering Resolution Transition](https://mlsanity.com/paper/2609.02036/) (2026)
- [LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL](https://mlsanity.com/paper/2609.03528/) (2026)
- [KwaiMind Technical Report](https://mlsanity.com/paper/2609.26375/) (2026)
- [WAM-OPD: Sharpening World Action Models via On-Policy Distillation](https://mlsanity.com/paper/2609.34250/) (2026)
- [From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Models](https://mlsanity.com/paper/2609.34371/) (2026)
- [CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models](https://mlsanity.com/paper/2609.34658/) (2026)
- [PE-OPSD: Internalizing Prompt Enhancement into Flow-matching Models via On-Policy Self-Distillation](https://mlsanity.com/paper/2609.36638/) (2026)
- [FlowMap-OPD: Rollout--Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators](https://mlsanity.com/paper/2609.37851/) (2026)

## Abstract (arXiv)

Reinforcement learning has emerged as a powerful tool for improving diffusion-based text-to-image models, but existing methods are largely limited to single-task optimization. Extending RL to multiple tasks is challenging: joint optimization suffers from cross-task interference and imbalance, while cascade RL is cumbersome and prone to catastrophic forgetting. We propose DiffusionOPD, a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD). DiffusionOPD first trains task-specific teachers independently, then distills their capabilities into a unified student along the student own rollout trajectories. This decouples single-task exploration from multi-task integration and avoids the optimization burden of solving all tasks jointly from scratch. Theoretically, we lift the OPD framework from discrete tokens to continuous-state Markov processes, deriving a closed-form per-step KL objective that unifies both stochastic SDE and deterministic ODE refinement via mean-matching. We formally and empirically demonstrate that this analytic gradient provides lower variance and better generality compared to conventional PPO-style policy gradients. Extensive experiments show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance, while achieving state-of-the-art results on all evaluated benchmarks.