mlsanityData to 7 Oct 2026

Value Gradient Guidance for Flow Matching Alignment

arXiv 2512.05116 ↗First posted 4 Dec 2025cs.LG cs.CVcited by 14

On the map: Diffusion models → Flow matching, time series and science

also in 1 more
Abstract (arXiv)

While methods exist for aligning flow matching models--a popular and effective class of generative models--with human preferences, existing approaches fail to achieve both adaptation efficiency and probabilistically sound prior preservation. In this work, we leverage the theory of optimal control and propose VGG-Flow, a gradient-matching-based method for finetuning pretrained flow matching models. The key idea behind this algorithm is that the optimal difference between the finetuned velocity field and the pretrained one should be matched with the gradient field of a value function. This method not only incorporates first-order information from the reward model but also benefits from heuristic initialization of the value function to enable fast adaptation. Empirically, we show on a popular text-to-image flow matching model, Stable Diffusion 3, that our method can finetune flow matching models under limited computational budgets while achieving effective and prior-preserving alignment.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Flow matching, time series and science, one of 14 directions in Diffusion models. Builds on 56 of its 79 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

56 of 79 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matching16 Apr
  2. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Models30 Jun
  3. 2022Flow Matching for Generative Modeling6 Oct
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow7 Sep

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 939
38 more
  1. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  2. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  3. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  4. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  6. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  8. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  9. 2021GFlowNet Foundationscited by 286
  10. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  11. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  12. 2021How to Train Your Energy-Based Modelscited by 271
  13. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  14. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  15. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  16. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  17. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  18. 2021Path Integral Sampler: a stochastic control approach for samplingcited by 175
  19. 2022An optimal control perspective on diffusion-based generative modelingcited by 145
  20. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  21. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 113
  22. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  23. 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 80
  24. 2024NETS: A Non-Equilibrium Transport Samplercited by 67
  25. 2023Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimizationcited by 57
  26. 2024Sequential Controlled Langevin Diffusionscited by 47
  27. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularizationcited by 40
  28. 2023Stochastic Optimal Control Matchingcited by 39
  29. 2022Stochastic Optimal Control for Collective Variable Free Sampling of Molecular Transition Pathscited by 36
  30. 2023Stochastic Generative Flow Networkscited by 32
  31. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
  32. 2025Underdamped Diffusion Bridges with Applications to Samplingcited by 28
  33. 2017Deep Relaxation: partial differential equations for optimizing deep neural networkscited by 27
  34. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  35. 2023Ghost on the Shell: An Expressive Representation of General 3D Shapescited by 24
  36. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
  37. 2025Conditioning Diffusions Using Malliavin Calculuscited by 15
  38. 2016High-Dimensional Stochastic Optimal Control using Continuous Tensor Decompositionscited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 5
  2. 2026Discrete Adjoint Schr"odinger Bridge Samplercited by 4
  3. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 4
  4. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 3
  5. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  6. 2026Inference-Time Attribute Distribution Alignment for Unconditional Diffusioncited by 2
  7. 2026Drifting Preference Optimization for One-Step Generative Modelscited by 2
  8. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  9. 2026Solving Inverse Problems with Flow-based Models via Model Predictive Controlcited by 1
  10. 2026Euphonium: Steering Video Flow Matching via Process Reward Gradient Guided Stochastic Dynamicscited by 1
  11. 2026Gradient-Free Noise Optimization for Reward Alignment in Generative Modelscited by 1
  12. 2026Conflict-Aware Additive Guidance for Flow Models under Compositional Rewardscited by 0
2 more
  1. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  2. 2026WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Mapscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories
  2. 2026Faster and Better Alignment for Flow Matching Models via Step-aware Advantages
  3. 2026Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation
  4. 2023Flow Matching in Latent Space
  5. 2025On the Guidance of Flow Matching
  6. 2025Flow Matching in the Low-Noise Regime: Pathologies and a Contrastive Remedy
  7. 2026Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching
  8. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedback