mlsanityData to 14 Oct 2026

DiffusionNFT: Online Diffusion Reinforcement with Forward Process

arXiv 2509.16117 ↗First posted 19 Sep 2025cs.LG cs.AI cs.CVcited by 230

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Online reinforcement learning (RL) has been central to post-training language models, but its extension to diffusion models remains challenging due to intractable likelihoods. Recent works discretize the reverse sampling process to enable GRPO-style training, yet they inherit fundamental drawbacks, including solver restrictions, forward-reverse inconsistency, and complicated integration with classifier-free guidance (CFG). We introduce Diffusion Negative-aware FineTuning (DiffusionNFT), a new online RL paradigm that optimizes diffusion models directly on the forward process via flow matching. DiffusionNFT contrasts positive and negative generations to define an implicit policy improvement direction, naturally incorporating reinforcement signals into the supervised learning objective. This formulation enables training with arbitrary black-box solvers, eliminates the need for likelihood estimation, and requires only clean images rather than sampling trajectories for policy optimization. DiffusionNFT is up to 25× more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free. For instance, DiffusionNFT improves the GenEval score from 0.24 to 0.98 within 1k steps, while FlowGRPO achieves 0.95 with over 5k steps and additional CFG employment. By leveraging multiple reward models, DiffusionNFT significantly boosts the performance of SD3.5-Medium in every benchmark tested.

Key facts. Cited by 230 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 60 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

53 of 60 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching7 Sep
  2. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance28 Aug
  3. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE29 Jul
  4. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignment10 Jul
  5. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  6. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  7. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  8. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  9. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  10. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  11. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  12. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
3 more
  1. 2023Training Diffusion Models with Reinforcement Learning22 May
  2. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  3. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2022Flow Matching for Generative Modelingcited by 5,702
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
26 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  2. 2021Variational Diffusion Modelscited by 1,153
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  5. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  7. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  9. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  10. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  12. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  13. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  14. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matchingcited by 183
  15. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  16. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  17. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  18. 2024Video Diffusion Alignment via Reward Gradientscited by 107
  19. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  20. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  21. 2024Diffusion Bridge Implicit Modelscited by 49
  22. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  23. 2023SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Modelscited by 36
  24. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 23
  25. 2025Visual Generation Without Guidancecited by 17
  26. 2024Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignmentcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 215.

230 papers.

Sort:

By area:

  1. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 86
  2. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  3. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  4. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 40
  5. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 40
  6. 2026FireRed-Image-Edit-1.0 Technical Reportcited by 36
  7. 2026Qwen-Image-2.0 Technical Reportcited by 36
  8. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 34
  9. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Modelscited by 25
  10. 2026WorldCompass: Reinforcement Learning for Long-Horizon World Modelscited by 24
  11. 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 24
  12. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
38 more
  1. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 22
  2. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  3. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 19
  4. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  5. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  6. 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoderscited by 15
  7. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 14
  8. 2026RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Timecited by 14
  9. 2025Data-regularized Reinforcement Learning for Diffusion Models at Scalecited by 13
  10. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  11. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generationcited by 12
  12. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 12
  13. 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 12
  14. 2026Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learningcited by 12
  15. 2026JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generationcited by 12
  16. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  17. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
  18. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generationcited by 9
  19. 2026Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPOcited by 9
  20. 2026PosterOmni: Generalized Artistic Poster Creation via Task Distillation and Unified Reward Feedbackcited by 9
  21. 2026Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generationcited by 9
  22. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 9
  23. 2026Motus2: A Self-Evolving General World Model for Dexterous Manipulationcited by 9
  24. 2026Reinforcing Few-step Generators via Reward-Tilted Distribution Matchingcited by 8
  25. 2025RubricRL: Simple Generalizable Rewards for Text-to-Image Generationcited by 7
  26. 2025MotionEdit: Benchmarking and Learning Motion-Centric Image Editingcited by 7
  27. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  28. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Rewardcited by 7
  29. 2026WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editingcited by 7
  30. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  31. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  32. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 7
  33. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 7
  34. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 6
  35. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  36. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 6
  37. 2026SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Modelscited by 6
  38. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6

50 most cited of 230 All 230 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  2. 2025One-Step Flow Policy Mirror Descent
  3. 2026Reinforcement Learning from Denoising Feedback
  4. 2026CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
  5. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  6. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  7. 2026Explicit Critic Guidance for Aligning Diffusion Models
  8. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development