mlsanityData to 14 Oct 2026

Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models

arXiv 2509.25050 ↗First posted 29 Sep 2025cs.LGcited by 67

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Reinforcement Learning (RL) has emerged as a central paradigm for advancing Large Language Models (LLMs), where both pre-training and RL post-training stages are grounded in the same log-likelihood formulation. In contrast, recent RL approaches for diffusion models, most notably Denoising Diffusion Policy Optimization (DDPO), optimize an objective different from the pretraining objectives--score/flow matching loss. In this work, we establish a novel theoretical analysis: DDPO is an implicit form of score/flow matching with noisy targets, which increases variance and slows convergence. Building on this analysis, we introduce Advantage Weighted Matching (AWM), a policy-gradient method for diffusion. It uses the score/flow-matching loss and reweights each sample by its advantage. In effect, AWM raises the influence of high-reward samples and suppresses low-reward ones while keeping the modeling objective identical to pretraining. This simple yet effective design yields substantial benefits: on the GenEval, OCR, and PickScore benchmarks, AWM delivers up to a 34× speedup over Flow-GRPO (which builds on DDPO), when applied to Stable Diffusion 3.5 Medium and FLUX, without compromising generation quality. Code is available at https://github.com/scxue/advantageweightedmatching

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 58 of its 70 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

58 of 70 references.

References · 58

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022Flow Matching for Generative Modelingcited by 5,702
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  13. 2021Variational Diffusion Modelscited by 1,153
  14. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  15. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  16. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  17. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  18. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  19. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  20. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  21. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  22. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  23. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  24. 2021Score-based Generative Modeling in Latent Spacecited by 636
  25. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  26. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  27. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  28. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  29. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  30. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  31. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  32. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  33. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  34. 2021Learning to Efficiently Sample from Diffusion Probabilistic Modelscited by 129
  35. 2025Flow Matching Policy Gradientscited by 99
  36. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 97
  37. 2023Non-Denoising Forward-Time Diffusionscited by 77
  38. 2023SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Modelscited by 72

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 64.

67 papers.

Sort:

By area:

  1. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  2. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 19
  3. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  4. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  5. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  6. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 12
  7. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
  8. 2026Reinforcing Few-step Generators via Reward-Tilted Distribution Matchingcited by 8
  9. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Rewardcited by 7
  10. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  11. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  12. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
38 more
  1. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 6
  2. 2026Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Modelscited by 6
  3. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  4. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 5
  5. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
  6. 2026A Systematic Post-Train Framework for Video Generationcited by 5
  7. 2026Qwen-Image-2.0-RL Technical Reportcited by 5
  8. 2026MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generatorscited by 5
  9. 2025Diffusion Alignment as Variational Expectation-Maximizationcited by 4
  10. 2026Faster and Better Alignment for Flow Matching Models via Step-aware Advantagescited by 4
  11. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Modelscited by 4
  12. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  13. 2026FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scalingcited by 3
  14. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  15. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space Viewcited by 3
  16. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matchingcited by 3
  17. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  18. 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learningcited by 2
  19. 2026Guiding Distribution Matching Distillation with Gradient-Based Reinforcement Learningcited by 2
  20. 2026DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Modelscited by 2
  21. 2026ReNFT: Repairing Mode Collapse in Reward Post-Training via Internal Probability-Mass Recalibrationcited by 2
  22. 2026FAIL: Flow Matching Adversarial Imitation Learning for Image Generationcited by 1
  23. 2026AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Modelscited by 1
  24. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Modelscited by 1
  25. 2026Exploring the Design Space of Reward Backpropagation for Flow Matchingcited by 1
  26. 2026Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomescited by 1
  27. 2026Evaluation-Verification Reward for Consistent Multi-Reference Image Editingcited by 1
  28. 2026GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech modelcited by 1
  29. 2026CounterAlign: Counterfactual Supervision for Vision-Language-Action Modelscited by 1
  30. 2026Can We Perform Online RL for Image Editing without Editing Rewards?cited by 1
  31. 2026Self-OPD: On-Policy Distillation for Flow Matching Models without Teachercited by 1
  32. 2026When Can Conditional Flow Matching Replace Pointwise Negative Log-Likelihood?cited by 1
  33. 2026FlowCPO: A Unified Divergence View of Preference Alignment for Flow Modelscited by 1
  34. 2026DOHF: Online Diffusion Fine-tuning with Doob's h-transform Guidancecited by 1
  35. 2026TIGFlow-GRPO: Trajectory Forecasting via Interaction-Aware Flow Matching and Reward-Guided Optimizationcited by 0
  36. 2026Explicit Critic Guidance for Aligning Diffusion Modelscited by 0
  37. 2026World Model Self-Distillation: Training World Models to Solve General Taskscited by 0
  38. 2026NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learningcited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
  2. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
  4. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
  5. 2026Relative Score Policy Optimization for Diffusion Language Models
  6. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
  7. 2026Reinforcement Learning from Denoising Feedback
  8. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective