mlsanityData to 7 Oct 2026

A Taxonomy of Loss Functions for Stochastic Optimal Control

arXiv 2410.00345 ↗First posted 1 Oct 2024cs.LG math.OC stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Stochastic optimal control (SOC) aims to direct the behavior of noisy systems and has widespread applications in science, engineering, and artificial intelligence. In particular, reward fine-tuning of diffusion and flow matching models and sampling from unnormalized methods can be recast as SOC problems. A recent work has introduced Adjoint Matching (Domingo-Enrich et al., 2024), a loss function for SOC problems that vastly outperforms existing loss functions in the reward fine-tuning setup. The goal of this work is to clarify the connections between all the existing (and some new) SOC loss functions. Namely, we show that SOC loss functions can be grouped into classes that share the same gradient in expectation, which means that their optimization landscape is the same; they only differ in their gradient variance. We perform simple SOC experiments to understand the strengths and weaknesses of different loss functions.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

26 of 55 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  9. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  10. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  11. 2020Scalable Gradients for Stochastic Differential Equationscited by 264
  12. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
14 more
  1. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 188
  2. 2021Neural SDEs as Infinite-Dimensional GANscited by 159
  3. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  4. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
  5. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 83
  6. 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 80
  7. 2023Stochastic Optimal Control Matchingcited by 39
  8. 2022Stochastic Optimal Control for Collective Variable Free Sampling of Molecular Transition Pathscited by 36
  9. 2017Deep Relaxation: partial differential equations for optimizing deep neural networkscited by 27
  10. 2021Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networkscited by 27
  11. 2015Adaptive importance sampling for control and inferencecited by 23
  12. 2016High-Dimensional Stochastic Optimal Control using Continuous Tensor Decompositionscited by 13
  13. 2019Variational approach to rare event simulation using least-squares regressioncited by 12
  14. 2022Improving control based importance sampling strategies for metastable diffusions via adapted metadynamicscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2024Sequential Controlled Langevin Diffusionscited by 47
  2. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
  3. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 20
  4. 2025Conditioning Diffusions Using Malliavin Calculuscited by 15
  5. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  6. 2025Control Consistency Losses for Diffusion Bridgescited by 6
  7. 2026How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidancecited by 5
  8. 2025End-To-End Learning of Gaussian Mixture Priors for Diffusion Samplercited by 4
  9. 2024An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Controlcited by 1
  10. 2026A Schr"odinger Eigenfunction Method for Long-Horizon Stochastic Optimal Controlcited by 0
  11. 2026Path Integral Value Matching for Linear Quadratic Stochastic Optimal Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A unified perspective on fine-tuning and sampling with diffusion and flow models
  2. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Models
  3. 2024Stochastic Optimal Control for Diffusion Bridges in Function Spaces
  4. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence
  5. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
  6. 2023Target-based Surrogates for Stochastic Optimization
  7. 2026Unsupervised Diffusion Solver for Combinatorial Optimization via Combinatorial Adjoint Matching
  8. 2026CADO: From Imitation to Cost Minimization for Heatmap-based Solvers in Combinatorial Optimization