# Learning Stochastic Optimal Policies via Gradient Descent

arXiv: https://arxiv.org/abs/2106.03780 · HTML: https://mlsanity.com/paper/2106.03780/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Variance Reduction for Faster Non-Convex Optimization](https://mlsanity.com/paper/1603.05643/) (2016)
- [LQR through the Lens of First Order Methods: Discrete-time Case](https://mlsanity.com/paper/1907.08921/) (2019)
- [Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems](https://mlsanity.com/paper/1411.1134/) (2014)
- [Stable Neural Flows](https://mlsanity.com/paper/2003.08063/) (2020)
- [Infinitely deep neural networks as diffusion processes](https://mlsanity.com/paper/1905.11065/) (2019)
- [Learning the Globally Optimal Distributed LQ Regulator](https://mlsanity.com/paper/1912.08774/) (2019)
- [Learning Deep Stochastic Optimal Control Policies using Forward-Backward SDEs](https://mlsanity.com/paper/1902.03986/) (2019)
- [Deep Forward-Backward SDEs for Min-max Control](https://mlsanity.com/paper/1906.04771/) (2019)

## Built on by (most cited first, 3 of 3)

- [How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.06335/) (2023)
- [Continuous-Depth Neural Models for Dynamic Graph Prediction](https://mlsanity.com/paper/2106.11581/) (2021)
- [An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations](https://mlsanity.com/paper/2407.10065/) (2024)

## Abstract (arXiv)

We systematically develop a learning-based treatment of stochastic optimal control (SOC), relying on direct optimization of parametric control policies. We propose a derivation of adjoint sensitivity results for stochastic differential equations through direct application of variational calculus. Then, given an objective function for a predetermined task specifying the desiderata for the controller, we optimize their parameters via iterative gradient descent methods. In doing so, we extend the range of applicability of classical SOC techniques, often requiring strict assumptions on the functional form of system and control. We verify the performance of the proposed approach on a continuous-time, finite horizon portfolio optimization with proportional transaction costs.