# From Reinforcement Learning to Optimal Control: A unified framework for sequential decisions

arXiv: https://arxiv.org/abs/1912.03513 · HTML: https://mlsanity.com/paper/1912.03513/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2019.

## Built on by (most cited first, 12 of 12)

- [Bayesian Learning via Neural Schr"odinger-F"ollmer Flows](https://mlsanity.com/paper/2111.10510/) (2021)
- [Model-predictive control and reinforcement learning in multi-energy system case studies](https://mlsanity.com/paper/2104.09785/) (2021)
- [On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation](https://mlsanity.com/paper/2301.09709/) (2023)
- [Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems](https://mlsanity.com/paper/2107.07788/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Stochastic Optimization for Vaccine and Testing Kit Allocation for the COVID-19 Pandemic](https://mlsanity.com/paper/2101.01204/) (2021)
- [A sequential transit network design algorithm with optimal learning under correlated beliefs](https://mlsanity.com/paper/2305.09452/) (2023)
- [Stein Diffusion Guidance: Training-Free Posterior Correction for Sampling Beyond High-Density Regions](https://mlsanity.com/paper/2507.05482/) (2025)
- [A deep real options policy for sequential service region design and timing](https://mlsanity.com/paper/2212.14800/) (2022)
- [Learning State-Dependent Policy Parametrizations for Dynamic Technician Routing with Rework](https://mlsanity.com/paper/2409.01815/) (2024)
- [Reinforcement Learning-Driven Plant-Wide Refinery Planning Using Model Decomposition](https://mlsanity.com/paper/2504.08642/) (2025)
- [Tackling Heterogeneity in Federated Learning via Variance-Reduced Boltzmann Sampling within Homogeneous Social Coalitions](https://mlsanity.com/paper/2506.02897/) (2025)

## Abstract (arXiv)

There are over 15 distinct communities that work in the general area of sequential decisions and information, often referred to as decisions under uncertainty or stochastic optimization. We focus on two of the most important fields: stochastic optimal control, with its roots in deterministic optimal control, and reinforcement learning, with its roots in Markov decision processes. Building on prior work, we describe a unified framework that covers all 15 different communities, and note the strong parallels with the modeling framework of stochastic optimal control. By contrast, we make the case that the modeling framework of reinforcement learning, inherited from discrete Markov decision processes, is quite limited. Our framework (and that of stochastic control) is based on the core problem of optimizing over policies. We describe four classes of policies that we claim are universal, and show that each of these two fields have, in their own way, evolved to include examples of each of these four classes.