# Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching

arXiv: https://arxiv.org/abs/2509.05952 · HTML: https://mlsanity.com/paper/2509.05952/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 34 of its 37 references that are in the map.
- First posted on arXiv 7 Sep 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2302.04867/) (2023)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://mlsanity.com/paper/2311.13231/) (2023)

## Built on by (most cited first, 30 of 56)

- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping](https://mlsanity.com/paper/2510.22319/) (2025)
- [DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO](https://mlsanity.com/paper/2512.21514/) (2025)
- [GARDO: Reinforcing Diffusion Models without Reward Hacking](https://mlsanity.com/paper/2512.24138/) (2025)
- [π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models](https://mlsanity.com/paper/2510.25889/) (2025)
- [VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization](https://mlsanity.com/paper/2512.16906/) (2025)
- [Manifold-Aware Exploration for Reinforcement Learning in Video Generation](https://mlsanity.com/paper/2603.21872/) (2026)
- [Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence](https://mlsanity.com/paper/2607.07675/) (2026)
- [Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization](https://mlsanity.com/paper/2510.21583/) (2025)
- [DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing](https://mlsanity.com/paper/2602.12205/) (2026)
- [UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation](https://mlsanity.com/paper/2603.23500/) (2026)
- [The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation](https://mlsanity.com/paper/2511.20256/) (2025)
- [SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation](https://mlsanity.com/paper/2511.21135/) (2025)
- [Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models](https://mlsanity.com/paper/2602.12529/) (2026)
- [Stepwise Credit Assignment for GRPO on Flow-Matching Models](https://mlsanity.com/paper/2603.28718/) (2026)
- [Ovis-Image Technical Report](https://mlsanity.com/paper/2511.22982/) (2025)
- [Reinforcing Few-step Generators via Reward-Tilted Distribution Matching](https://mlsanity.com/paper/2605.26108/) (2026)
- [PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling](https://mlsanity.com/paper/2512.04784/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models](https://mlsanity.com/paper/2604.17415/) (2026)
- [V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think](https://mlsanity.com/paper/2604.23380/) (2026)
- [RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO](https://mlsanity.com/paper/2605.15190/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment](https://mlsanity.com/paper/2512.12387/) (2025)
- [ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuning](https://mlsanity.com/paper/2602.03425/) (2026)
- [π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs](https://mlsanity.com/paper/2603.02083/) (2026)
- [Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models](https://mlsanity.com/paper/2606.11025/) (2026)
- [VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning](https://mlsanity.com/paper/2608.26105/) (2026)
- [Diffusion Reinforcement Learning via Centered Reward Distillation](https://mlsanity.com/paper/2603.14128/) (2026)
- [A Systematic Post-Train Framework for Video Generation](https://mlsanity.com/paper/2604.25427/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has recently emerged as a powerful technique for improving image and video generation in Diffusion and Flow Matching models, specifically for enhancing output quality and alignment with prompts. A critical step for applying online RL methods on Flow Matching is the introduction of stochasticity into the deterministic framework, commonly realized by Stochastic Differential Equation (SDE). Our investigation reveals a significant drawback to this approach: SDE-based sampling introduces pronounced noise artifacts in the generated images, which we found to be detrimental to the reward learning process. A rigorous theoretical analysis traces the origin of this noise to an excess of stochasticity injected during inference. To address this, we draw inspiration from Denoising Diffusion Implicit Models (DDIM) to reformulate the sampling process. Our proposed method, Coefficients-Preserving Sampling (CPS), eliminates these noise artifacts. This leads to more accurate reward modeling, ultimately enabling faster and more stable convergence for reinforcement learning-based optimizers like Flow-GRPO and Dance-GRPO. Code will be released at https://github.com/IamCreateAI/FlowCPS