# Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach

arXiv: https://arxiv.org/abs/2406.10959 · HTML: https://mlsanity.com/paper/2406.10959/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Regularity and stability of feedback relaxed controls](https://mlsanity.com/paper/2001.03148/) (2020)
- [Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations](https://mlsanity.com/paper/2406.00612/) (2024)
- [A modified MSA for stochastic control problems](https://mlsanity.com/paper/2007.05209/) (2020)
- [Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems](https://mlsanity.com/paper/2209.07059/) (2022)
- [A neural network based policy iteration algorithm with global H2-superlinear convergence for stochastic games on domains](https://mlsanity.com/paper/1906.02304/) (2019)
- [Linear convergence of a policy gradient method for some finite horizon continuous time control problems](https://mlsanity.com/paper/2203.11758/) (2022)

## Built on by (most cited first, 6 of 6)

- [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025)
- [Regret of exploratory policy improvement and q-learning](https://mlsanity.com/paper/2411.01302/) (2024)
- [A Reinforcement Learning Framework for Some Singular Stochastic Control Problems](https://mlsanity.com/paper/2506.22203/) (2025)
- [DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problems](https://mlsanity.com/paper/2511.04309/) (2025)
- [Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence](https://mlsanity.com/paper/2609.14972/) (2026)
- [Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems](https://mlsanity.com/paper/2610.04285/) (2026)

## Abstract (arXiv)

In this paper we investigate the convergence of the Policy Iteration Algorithm (PIA) for a class of general continuous-time entropy-regularized stochastic control problems. In particular, instead of employing sophisticated PDE estimates for the iterative PDEs involved in the algorithm (see, e.g., Huang-Wang-Zhou(2025)), we shall provide a simple proof from scratch for the convergence of the PIA. Our approach builds on probabilistic representation formulae for solutions of PDEs and their derivatives. Moreover, in the finite horizon model and in the infinite horizon model with large discount factor, the similar arguments lead to a super-exponential rate of convergence without tear. Finally, with some extra efforts we show that our approach can be extended to the diffusion control case in the one dimensional setting, also with a super-exponential rate of convergence.