# Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems

arXiv: https://arxiv.org/abs/2209.07059 · HTML: https://mlsanity.com/paper/2209.07059/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2022.

## Built on by (most cited first, 10 of 10)

- [Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach](https://mlsanity.com/paper/2406.10959/) (2024)
- [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025)
- [A Reinforcement Learning Framework for Some Singular Stochastic Control Problems](https://mlsanity.com/paper/2506.22203/) (2025)
- [Discretization error from regularized Reinforcement Learning to continuous-time stochastic control](https://mlsanity.com/paper/2604.21179/) (2026)
- [Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process](https://mlsanity.com/paper/2606.28669/) (2026)
- [Robust Exploratory Stopping under Ambiguity in Reinforcement Learning](https://mlsanity.com/paper/2510.10260/) (2025)
- [DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problems](https://mlsanity.com/paper/2511.04309/) (2025)
- [Dynamic mean-variance portfolio selection with no-shorting constraints and unknown investment opportunity sets](https://mlsanity.com/paper/2607.16625/) (2026)
- [Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence](https://mlsanity.com/paper/2609.14972/) (2026)
- [Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems](https://mlsanity.com/paper/2610.04285/) (2026)

## Abstract (arXiv)

For a general entropy-regularized stochastic control problem on an infinite horizon, we prove that a policy iteration algorithm (PIA) converges to an optimal relaxed control. Contrary to the standard stochastic control literature, classical H"older estimates of value functions do not ensure the convergence of the PIA, due to the added entropy-regularizing term. To circumvent this, we carry out a delicate estimation by moving back and forth between appropriate H"older and Sobolev spaces. This requires new Sobolev estimates designed specifically for the purpose of policy iteration and a nontrivial technique to contain the entropy growth. Ultimately, we obtain a uniform H"older bound for the sequence of value functions generated by the PIA, thereby achieving the desired convergence result. Characterization of the optimal value function as the unique solution to an exploratory Hamilton-Jacobi-Bellman equation comes as a by-product. The PIA is numerically implemented in an example of optimal consumption.