mlsanityData to 14 Oct 2026

Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching

arXiv 2509.05952 ↗First posted 7 Sep 2025cs.CVcited by 56

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) has recently emerged as a powerful technique for improving image and video generation in Diffusion and Flow Matching models, specifically for enhancing output quality and alignment with prompts. A critical step for applying online RL methods on Flow Matching is the introduction of stochasticity into the deterministic framework, commonly realized by Stochastic Differential Equation (SDE). Our investigation reveals a significant drawback to this approach: SDE-based sampling introduces pronounced noise artifacts in the generated images, which we found to be detrimental to the reward learning process. A rigorous theoretical analysis traces the origin of this noise to an excess of stochasticity injected during inference. To address this, we draw inspiration from Denoising Diffusion Implicit Models (DDIM) to reformulate the sampling process. Our proposed method, Coefficients-Preserving Sampling (CPS), eliminates these noise artifacts. This leads to more accurate reward modeling, ultimately enabling faster and more stable convergence for reinforcement learning-based optimizers like Flow-GRPO and Dance-GRPO. Code will be released at https://github.com/IamCreateAI/FlowCPS

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 34 of its 37 references that are in the map. First posted on arXiv 7 Sep 2025.

Builds on

34 of 37 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  3. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning3 Feb
  4. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  5. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  6. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2022Flow Matching for Generative Modelingcited by 5,702
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
16 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  4. 2023Consistency Modelscited by 1,822
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  7. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  10. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  11. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  13. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 450
  14. 2025PaddleOCR 3.0 Technical Reportcited by 143
  15. 2024AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Eracited by 8
  16. 2025Rethinking Diffusion Model in High Dimensioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 43.

56 papers.

Sort:

By area:

  1. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  2. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  3. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  4. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  5. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  6. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  7. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15
  8. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  9. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 14
  10. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editingcited by 14
  11. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  12. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 12
38 more
  1. 2025SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigationcited by 12
  2. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 12
  3. 2026Stepwise Credit Assignment for GRPO on Flow-Matching Modelscited by 10
  4. 2025Ovis-Image Technical Reportcited by 8
  5. 2026Reinforcing Few-step Generators via Reward-Tilted Distribution Matchingcited by 8
  6. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modelingcited by 7
  7. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  8. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  9. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  10. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 7
  11. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 7
  12. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 6
  13. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  14. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 6
  15. 2026Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Modelscited by 6
  16. 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 6
  17. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
  18. 2026A Systematic Post-Train Framework for Video Generationcited by 5
  19. 2026Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridgingcited by 5
  20. 2025PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Modelscited by 4
  21. 2026TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignmentcited by 4
  22. 2026Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimizationcited by 4
  23. 2026TempAct: Advancing Temporal Plausibility in Autoregressive Video Generation via Planner-Executor RLcited by 4
  24. 2026Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformerscited by 3
  25. 2026When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropycited by 3
  26. 2026Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Modelscited by 3
  27. 2026Reward as An Agent for Embodied World Modelscited by 3
  28. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  29. 2026AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPOcited by 2
  30. 2026FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generationcited by 1
  31. 2026Correlation-Weighted Multi-Reward Optimization for Compositional Generationcited by 1
  32. 2026ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matchingcited by 1
  33. 2026AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generationcited by 1
  34. 2026Optimizing Visual Generative Models via Distribution-wise Rewardscited by 1
  35. 2026GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech modelcited by 1
  36. 2026StructRL: Structured Action-Space Exploration for Flow-Based VLAscited by 1
  37. 2026LithoGRPO: Fast Inverse Lithography via GRPO Reinforced Flow Matchingcited by 0
  38. 2026LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correctioncited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models
  2. 2026A Survey of Flow Matching in Reinforcement Learning
  3. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHF
  4. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  5. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Models
  6. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learning
  7. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Models
  8. 2026E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Models