mlsanityData to 14 Oct 2026

Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

arXiv 2603.12893 ↗First posted 13 Mar 2026cs.CV cs.AI cs.LGcited by 5

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models, as it enables learning from reward signals to explicitly improve desirable aspects such as image quality and prompt alignment. In this paper, we propose an online RL variant that reduces the variance in the model updates by sampling paired trajectories and pulling the flow velocity in the direction of the more favorable image. Unlike existing methods that treat each sampling step as a separate policy action, we consider the entire sampling process as a single action. We experiment with both high-quality vision language models and off-the-shelf quality metrics for rewards, and evaluate the outputs using a broad set of metrics. Our method converges faster and yields higher output quality and prompt alignment than previous approaches.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 59 of its 67 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

59 of 67 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Flow Matching Policy Gradients28 Jul
  2. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  4. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  5. 2025Flow Q-Learning4 Feb
  6. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets10 Dec
  7. 2024Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward22 Nov
  8. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  9. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review18 Jul
  10. 2024Improving GFlowNets for Text-to-Image Diffusion Alignment2 Jun
  11. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models1 May
  12. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
3 more
  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  3. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2022Flow Matching for Generative Modelingcited by 5,702
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
32 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  9. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  10. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  12. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  13. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  14. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  15. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  16. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  17. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  18. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  19. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  20. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  21. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  22. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  23. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 165
  24. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 87
  25. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  26. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 61
  27. 2023Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Modelcited by 46
  28. 2022Understanding DDPM Latent Codes Through Optimal Transportcited by 45
  29. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  30. 2024Simple Policy Optimizationcited by 32
  31. 2024TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editingcited by 16
  32. 2025Dual-Process Image Generationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 7
  2. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  3. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space Viewcited by 3
  4. 2026LeanGRPO: Eliminating Redundant Recomputation in Diffusion RLcited by 0
  5. 2026MEND: RL For Flow Models via Proximal Velocity Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
  2. 2024RL for Consistency Models: Faster Reward Guided Text-to-Image Generation
  3. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  4. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design
  5. 2025Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learning
  6. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generation
  7. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training
  8. 2026Unified Trajectory Matching Policy Optimization: Diverse T2I Generation and VLA Generalization