# Flow-OPD: On-Policy Distillation for Flow Matching Models

arXiv: https://arxiv.org/abs/2605.08063 · HTML: https://mlsanity.com/paper/2605.08063/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 37 of its 41 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 37)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Unified Reward Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.05236/) (2025)
- [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization](https://mlsanity.com/paper/2601.05242/) (2026)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution](https://mlsanity.com/paper/2501.11561/) (2025)
- [DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs](https://mlsanity.com/paper/2503.07067/) (2025)
- [Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning](https://mlsanity.com/paper/2506.04207/) (2025)
- [Fast and Effective On-policy Distillation from Reasoning Prefixes](https://mlsanity.com/paper/2602.15260/) (2026)
- [Interleaving Reasoning for Better Text-to-Image Generation](https://mlsanity.com/paper/2509.06945/) (2025)
- [Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2601.22060/) (2026)
- [Gen-Searcher: Reinforcing Agentic Search for Image Generation](https://mlsanity.com/paper/2603.28767/) (2026)
- [UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervision](https://mlsanity.com/paper/2601.03193/) (2026)
- [AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning](https://mlsanity.com/paper/2508.06924/) (2025)

## Built on by (most cited first, 30 of 34)

- [DanceOPD: On-Policy Generative Field Distillation](https://mlsanity.com/paper/2606.27377/) (2026)
- [OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators](https://mlsanity.com/paper/2607.08766/) (2026)
- [Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training](https://mlsanity.com/paper/2605.12483/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [Qwen-Image-2.0-RL Technical Report](https://mlsanity.com/paper/2606.27608/) (2026)
- [Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging](https://mlsanity.com/paper/2608.03316/) (2026)
- [Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation](https://mlsanity.com/paper/2607.24731/) (2026)
- [Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models](https://mlsanity.com/paper/2608.04349/) (2026)
- [WAM-OPD: On-Policy Distillation for World Action Models](https://mlsanity.com/paper/2608.22364/) (2026)
- [CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2605.25378/) (2026)
- [DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models](https://mlsanity.com/paper/2608.09233/) (2026)
- [D3-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation](https://mlsanity.com/paper/2608.24987/) (2026)
- [FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models](https://mlsanity.com/paper/2607.24522/) (2026)
- [HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models](https://mlsanity.com/paper/2608.13205/) (2026)
- [Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher](https://mlsanity.com/paper/2608.26872/) (2026)
- [Flow3D-OPD: Multi-Teacher On-Policy Distillation for 3D Geometry Generation with Flow-Matching Diffusion Transformer](https://mlsanity.com/paper/2609.07137/) (2026)
- [On-Policy Self-Distillation for Multi-Turn Image Editing](https://mlsanity.com/paper/2609.35611/) (2026)
- [Qwen-Image-Flash: Rethinking the Training Recipe for Few-Step Distillation](https://mlsanity.com/paper/2606.03746/) (2026)
- [ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering](https://mlsanity.com/paper/2607.16060/) (2026)
- [SelfLift: Accelerating Few-Step Diffusion via Self-Recovering Resolution Transition](https://mlsanity.com/paper/2609.02036/) (2026)
- [NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech](https://mlsanity.com/paper/2609.31892/) (2026)
- [WAM-OPD: Sharpening World Action Models via On-Policy Distillation](https://mlsanity.com/paper/2609.34250/) (2026)
- [From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Models](https://mlsanity.com/paper/2609.34371/) (2026)
- [CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models](https://mlsanity.com/paper/2609.34658/) (2026)
- [G3-LoRA: Organizing Reward-Weighted Video Data with Gradient-Guided Grouped LoRA](https://mlsanity.com/paper/2609.35189/) (2026)
- [FlowMap-OPD: Rollout--Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators](https://mlsanity.com/paper/2609.37851/) (2026)
- [UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement](https://mlsanity.com/paper/2609.38721/) (2026)
- [GFD-OPD: Guidance-Folded On-Policy Distillation of Diffusion Models Across Scales](https://mlsanity.com/paper/2609.39692/) (2026)
- [DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies](https://mlsanity.com/paper/2610.00317/) (2026)
- [FastOPD: On-Policy Distillation for Lightweight VLA Deployment](https://mlsanity.com/paper/2610.02832/) (2026)

## Abstract (arXiv)

Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a 'seesaw effect' of competing metrics and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in the large language model community, we propose Flow-OPD, the first unified post-training framework that integrates on-policy distillation into Flow Matching models. Flow-OPD adopts a two-stage alignment strategy: it first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; it then establishes a robust initial policy through a Flow-based Cold-Start scheme and seamlessly consolidates heterogeneous expertise into a single student via a three-step orchestration of on-policy sampling, task-routing labeling, and dense trajectory-level supervision. We further introduce Manifold Anchor Regularization (MAR), which leverages a task-agnostic teacher to provide full-data supervision that anchors generation to a high-quality manifold, effectively mitigating the aesthetic degradation commonly observed in purely RL-driven alignment. Built upon Stable Diffusion 3.5 Medium, Flow-OPD raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference alignment and exhibiting an emergent 'teacher-surpassing' effect. These results establish Flow-OPD as a scalable alignment paradigm for building generalist text-to-image models. The codes and weights will be released in: https://github.com/CostaliyA/Flow-OPD .