# Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

arXiv: https://arxiv.org/abs/2605.10759 · HTML: https://mlsanity.com/paper/2605.10759/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 40 of its 47 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Stochastic Interpolants: A Unifying Framework for Flows and Diffusions](https://mlsanity.com/paper/2303.08797/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://mlsanity.com/paper/2309.17400/) (2023)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://mlsanity.com/paper/2311.13231/) (2023)
- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control](https://mlsanity.com/paper/2402.15194/) (2024)
- [Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution](https://mlsanity.com/paper/2501.11561/) (2025)
- [Flow map matching with stochastic interpolants: A mathematical framework for consistency models](https://mlsanity.com/paper/2406.07507/) (2024)
- [Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matching](https://mlsanity.com/paper/2504.11713/) (2025)
- [Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models](https://mlsanity.com/paper/2509.25050/) (2025)
- [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.00760/) (2024)
- [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://mlsanity.com/paper/2402.08265/) (2024)
- [PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models](https://mlsanity.com/paper/2402.08714/) (2024)
- [Meta Flow Maps enable scalable reward alignment](https://mlsanity.com/paper/2601.14430/) (2026)

## Built on by (most cited first, 19 of 19)

- [MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generators](https://mlsanity.com/paper/2607.15273/) (2026)
- [Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View](https://mlsanity.com/paper/2608.14430/) (2026)
- [Scaling Reinforcement Learning for Diffusion Models via Velocity Matching](https://mlsanity.com/paper/2608.23664/) (2026)
- [Non-Asymptotic Error Bounds for SMC with Biased Proposals: Application to Conditional Diffusion Sampling](https://mlsanity.com/paper/2607.04780/) (2026)
- [When Can Conditional Flow Matching Replace Pointwise Negative Log-Likelihood?](https://mlsanity.com/paper/2608.28010/) (2026)
- [FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models](https://mlsanity.com/paper/2609.09905/) (2026)
- [OMatG-flash: An All-Atom Flow Map with Reinforce Adjoint Matching for Scalable Materials Discovery](https://mlsanity.com/paper/2609.26402/) (2026)
- [WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps](https://mlsanity.com/paper/2609.27033/) (2026)
- [Tilted Schr"odinger Bridge Matching](https://mlsanity.com/paper/2609.34642/) (2026)
- [VINE: Taming Generative Control Policies for Reinforcement Learning](https://mlsanity.com/paper/2607.10369/) (2026)
- [RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation](https://mlsanity.com/paper/2608.09226/) (2026)
- [LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL](https://mlsanity.com/paper/2609.03528/) (2026)
- [Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling](https://mlsanity.com/paper/2609.05727/) (2026)
- [Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching](https://mlsanity.com/paper/2609.29405/) (2026)
- [Let CSP Be Your ANCHOR: Adaptive Crystal Search over Frozen Structure Priors](https://mlsanity.com/paper/2609.33407/) (2026)
- [Elucidating the Design Space of Regression-based Diffusion Reinforcement Learning](https://mlsanity.com/paper/2609.33444/) (2026)
- [Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows](https://mlsanity.com/paper/2609.36812/) (2026)
- [Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models](https://mlsanity.com/paper/2609.39091/) (2026)
- [GuidanceNFT: NFT-Based Reward Fine-Tuning of Guided Diffusion Models](https://mlsanity.com/paper/2610.15856/) (2026)

## Abstract (arXiv)

Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses against a closed-form target. RL post-training aligns the model with a reward. In image generation, this makes samples compose objects correctly, render text legibly, and match human preferences. Existing methods rely on costly SDE rollouts, reward gradients, or surrogate losses, sacrificing pretraining's regression structure. We show that the structure extends to RL post-training. Under KL-regularized reward maximization, the optimal generative process tilts the clean-endpoint distribution towards samples with higher reward and leaves the noising law unchanged. Combining this with the adjoint-matching optimality condition and a REINFORCE identity, we derive Reinforce Adjoint Matching (RAM): a consistency loss that corrects the pretraining target with the reward. At each step, we draw a clean endpoint from the current model, evaluate its reward, noise it as in pretraining, and regress. No SDE rollouts, backward adjoint sweeps, or reward gradients are required. Like the pretraining objective, RAM is simple and scales. On Stable Diffusion 3.5M, RAM achieves the highest reward on composability, text rendering, and human preference, reaching Flow-GRPO's peak reward in up to 50× fewer training steps.