# Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

arXiv: https://arxiv.org/abs/2609.14972 · HTML: https://mlsanity.com/paper/2609.14972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations](https://mlsanity.com/paper/2406.00612/) (2024)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Regret of exploratory policy improvement and q-learning](https://mlsanity.com/paper/2411.01302/) (2024)
- [Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approach](https://mlsanity.com/paper/2602.05533/) (2026)
- [Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems](https://mlsanity.com/paper/2209.07059/) (2022)
- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study](https://mlsanity.com/paper/2412.16175/) (2024)
- [Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State](https://mlsanity.com/paper/2509.23711/) (2025)
- [Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach](https://mlsanity.com/paper/2406.10959/) (2024)
- [Policy Gradient for Continuous-Time Mean-Field Control](https://mlsanity.com/paper/2605.20718/) (2026)
- [Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2507.00358/) (2025)
- [A Two-fold Randomization Framework for Impulse Control Problems](https://mlsanity.com/paper/2509.12018/) (2025)
- [Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems](https://mlsanity.com/paper/2606.11798/) (2026)
- [Policy Iteration Achieves Regularized Equilibrium under Time Inconsistency](https://mlsanity.com/paper/2603.06145/) (2026)
- [Discretization error from regularized Reinforcement Learning to continuous-time stochastic control](https://mlsanity.com/paper/2604.21179/) (2026)
- [A Zeroth-Order Deep Learning Method for Fully Nonlinear Parabolic Partial Differential Equations with Unknown Coefficients](https://mlsanity.com/paper/2606.24999/) (2026)

## Abstract (arXiv)

We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.