mlsanityData to 7 Oct 2026

Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control

arXiv 2409.08861 ↗First posted 13 Sep 2024cs.LG math.OC stat.MLcited by 222

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Dynamical generative models that produce samples through an iterative process, such as Flow Matching and denoising diffusion models, have seen widespread use, but there have not been many theoretically-sound methods for improving these models with reward fine-tuning. In this work, we cast reward fine-tuning as stochastic optimal control (SOC). Critically, we prove that a very specific memoryless noise schedule must be enforced during fine-tuning, in order to account for the dependency between the noise variable and the generated samples. We also propose a new algorithm named Adjoint Matching which outperforms existing SOC algorithms, by casting SOC problems as a regression problem. We find that our approach significantly improves over existing methods for reward fine-tuning, achieving better consistency, realism, and generalization to unseen human preference reward models, while retaining sample diversity.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 38 of its 98 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

38 of 98 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review18 Jul
  2. 2024Improving GFlowNets for Text-to-Image Diffusion Alignment2 Jun
  3. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  4. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May

Other references · 33

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  9. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  12. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
21 more
  1. 2022Rectified Flow: A Marginal Preserving Approach to Optimal Transportcited by 277
  2. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  3. 2022An optimal control perspective on diffusion-based generative modelingcited by 145
  4. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  5. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 115
  6. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  7. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  8. 2019Theoretical guarantees for sampling and inference in generative models with latent diffusionscited by 97
  9. 2023Training-free Linear Image Inverses via Flowscited by 70
  10. 2024Symbolic Music Generation with Non-Differentiable Rule Guided Diffusioncited by 52
  11. 2016Deep Learning Approximation for Stochastic Control Problemscited by 51
  12. 2024RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Controlcited by 46
  13. 2024DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised h-transformcited by 44
  14. 2024Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modellingcited by 43
  15. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  16. 2023Stochastic Optimal Control Matchingcited by 39
  17. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  18. 2023Neural Diffusion Modelscited by 20
  19. 2024Neural Optimal Transport with Lagrangian Costscited by 16
  20. 2024Posterior Sampling with Denoising Oracles via Tilted Transportcited by 12
  21. 2021Differentiable Gaussianization Layers for Inverse Problems Regularized by Deep Generative Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 95; 2026: 118.

222 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  2. 2025Improving Video Generation with Human Feedbackcited by 267
  3. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  4. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  5. 2025Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matchingcited by 78
  6. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  7. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
  8. 2024Sequential Controlled Langevin Diffusionscited by 47
  9. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  10. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularizationcited by 40
  11. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  12. 2025Scalable Equilibrium Sampling with Sequential Boltzmann Generatorscited by 36
38 more
  1. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  2. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  3. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  4. 2026Q-learning with Adjoint Matchingcited by 33
  5. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  6. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
  7. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  8. 2025Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlocited by 29
  9. 2025Test-Time Scaling of Diffusion Models via Noise Trajectory Searchcited by 28
  10. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Modelscited by 28
  11. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 26
  12. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 25
  13. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 23
  14. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
  15. 2025DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Modelscited by 21
  16. 2025Adjoint Schr"odinger Bridge Samplercited by 20
  17. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 20
  18. 2025Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generationcited by 19
  19. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 18
  20. 2025From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster trainingcited by 17
  21. 2025GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Modelscited by 17
  22. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  23. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 17
  24. 2025Outsourced diffusion sampling: Efficient posterior inference in latent spaces of generative modelscited by 16
  25. 2025RNE: plug-and-play diffusion inference-time control and energy-based trainingcited by 16
  26. 2025Proximal Diffusion Neural Samplercited by 16
  27. 2025Conditioning Diffusions Using Malliavin Calculuscited by 15
  28. 2025Science-T2I: Addressing Scientific Illusions in Image Synthesiscited by 15
  29. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  30. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 15
  31. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 15
  32. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  33. 2025Reinforcement Learning for Flow-Matching Policiescited by 14
  34. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  35. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  36. 2025Posterior Inference with Diffusion Models for High-dimensional Black-box Optimizationcited by 12
  37. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  38. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12

50 most cited of 222 All 222 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching
  2. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  3. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards
  4. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
  5. 2026A Unified Measure-Theoretic View of Diffusion, Score-Based, and Flow Matching Generative Models
  6. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Models
  7. 2026ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
  8. 2026Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models