# A Taxonomy of Loss Functions for Stochastic Optimal Control

arXiv: https://arxiv.org/abs/2410.00345 · HTML: https://mlsanity.com/paper/2410.00345/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://mlsanity.com/paper/2309.17400/) (2023)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Neural SDEs as Infinite-Dimensional GANs](https://mlsanity.com/paper/2102.03657/) (2021)
- [Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control](https://mlsanity.com/paper/2402.15194/) (2024)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [Do Differentiable Simulators Give Better Policy Gradients?](https://mlsanity.com/paper/2202.00817/) (2022)
- [Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space](https://mlsanity.com/paper/2005.05409/) (2020)
- [Stochastic Optimal Control Matching](https://mlsanity.com/paper/2312.02027/) (2023)
- [Stochastic Optimal Control for Collective Variable Free Sampling of Molecular Transition Paths](https://mlsanity.com/paper/2207.02149/) (2022)
- [Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks](https://mlsanity.com/paper/2102.11379/) (2021)
- [Deep Relaxation: partial differential equations for optimizing deep neural networks](https://mlsanity.com/paper/1704.04932/) (2017)
- [Adaptive importance sampling for control and inference](https://mlsanity.com/paper/1505.01874/) (2015)
- [High-Dimensional Stochastic Optimal Control using Continuous Tensor Decompositions](https://mlsanity.com/paper/1611.04706/) (2016)
- [Variational approach to rare event simulation using least-squares regression](https://mlsanity.com/paper/1901.09195/) (2019)
- [Improving control based importance sampling strategies for metastable diffusions via adapted metadynamics](https://mlsanity.com/paper/2206.06628/) (2022)

## Built on by (most cited first, 13 of 13)

- [Sequential Controlled Langevin Diffusions](https://mlsanity.com/paper/2412.07081/) (2024)
- [Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference](https://mlsanity.com/paper/2508.12511/) (2025)
- [MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Control](https://mlsanity.com/paper/2508.10684/) (2025)
- [Conditioning Diffusions Using Malliavin Calculus](https://mlsanity.com/paper/2504.03461/) (2025)
- [Value Gradient Guidance for Flow Matching Alignment](https://mlsanity.com/paper/2512.05116/) (2025)
- [Control Consistency Losses for Diffusion Bridges](https://mlsanity.com/paper/2512.05070/) (2025)
- [How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance](https://mlsanity.com/paper/2604.27147/) (2026)
- [End-To-End Learning of Gaussian Mixture Priors for Diffusion Sampler](https://mlsanity.com/paper/2503.00524/) (2025)
- [Rare Event Analysis via Stochastic Optimal Control](https://mlsanity.com/paper/2604.13213/) (2026)
- [An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control](https://mlsanity.com/paper/2410.05163/) (2024)
- [A Schr"odinger Eigenfunction Method for Long-Horizon Stochastic Optimal Control](https://mlsanity.com/paper/2603.23173/) (2026)
- [Path Integral Value Matching for Linear Quadratic Stochastic Optimal Control](https://mlsanity.com/paper/2608.10777/) (2026)
- [One for All, All for One: Coordinated Multi-Agent Diffusion Steering via Stochastic Optimal Control](https://mlsanity.com/paper/2610.08595/) (2026)

## Abstract (arXiv)

Stochastic optimal control (SOC) aims to direct the behavior of noisy systems and has widespread applications in science, engineering, and artificial intelligence. In particular, reward fine-tuning of diffusion and flow matching models and sampling from unnormalized methods can be recast as SOC problems. A recent work has introduced Adjoint Matching (Domingo-Enrich et al., 2024), a loss function for SOC problems that vastly outperforms existing loss functions in the reward fine-tuning setup. The goal of this work is to clarify the connections between all the existing (and some new) SOC loss functions. Namely, we show that SOC loss functions can be grouped into classes that share the same gradient in expectation, which means that their optimization landscape is the same; they only differ in their gradient variance. We perform simple SOC experiments to understand the strengths and weaknesses of different loss functions.