mlsanityData to 7 Oct 2026

AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Models

arXiv 2605.26013 ↗First posted 25 May 2026cs.LG cs.AI cs.CVcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We present AdvantageFlow, a forward-process reinforcement learning (RL) algorithm for rectified flow models. The algorithm minimizes an advantage-weighted prediction loss, which maximizes reward, regularized by the rollout policy, which convexifies the objective and makes its optimization stable. Our objective can be viewed as fitting a local reward-improving target distribution. The rollout regularization arises as a variance reduction step. We evaluate AdvantageFlow empirically on text-to-image generation with Stable Diffusion 3.5 Medium and FLUX.1, and compare it to both forward- and reverse-process RL algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 30 of its 39 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

30 of 39 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Stepwise Credit Assignment for GRPO on Flow-Matching Models30 Mar
  2. 2026Diffusion Reinforcement Learning via Centered Reward Distillation14 Mar
  3. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models13 Feb
  4. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedback20 Oct
  5. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Process19 Sep
  6. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models7 Sep
  7. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Models6 Aug
  8. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE29 Jul
  9. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  10. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  11. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  12. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
5 more
  1. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  3. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  4. 2023Training Diffusion Models with Reinforcement Learning22 May
  5. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  10. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  11. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  12. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
1 more
  1. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO
  2. 2025SuperFlow: Training Flow Matching Models with RL on the Fly
  3. 2026Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models
  4. 2026CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models
  5. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
  6. 2025Flow Matching Policy Gradients
  7. 2026MEND: RL For Flow Models via Proximal Velocity Matching
  8. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design