# A Fully Data-Driven Value Iteration for Stochastic LQR: Convergence, Robustness and Stability

arXiv: https://arxiv.org/abs/2505.02970 · HTML: https://mlsanity.com/paper/2505.02970/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 66 references that are in the map.
- First posted on arXiv 5 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018)
- [Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator](https://mlsanity.com/paper/1712.08642/) (2017)
- [Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulator](https://mlsanity.com/paper/1905.12842/) (2019)
- [Model-Free Linear Quadratic Control via Reduction to Expert Prediction](https://mlsanity.com/paper/1804.06021/) (2018)
- [Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradient](https://mlsanity.com/paper/2302.13144/) (2023)
- [Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems](https://mlsanity.com/paper/2107.07788/) (2021)
- [Robust Reinforcement Learning: A Case Study in Linear Quadratic Regulation](https://mlsanity.com/paper/2008.11592/) (2020)
- [Continuous-Time Robust Dynamic Programming](https://mlsanity.com/paper/1809.05867/) (2018)
- [Small-Disturbance Input-to-State Stability of Perturbed Gradient Flows: Applications to LQR Problem](https://mlsanity.com/paper/2310.02930/) (2023)
- [Convergence and Robustness of Value and Policy Iteration for the Linear Quadratic Regulator](https://mlsanity.com/paper/2411.04548/) (2024)
- [Convergence of Dynamic Programming on the Semidefinite Cone](https://mlsanity.com/paper/2106.09391/) (2021)

## Abstract (arXiv)

Unlike traditional model-based reinforcement learning approaches that estimate system parameters from data, non-model-based data-driven control learns the optimal policy directly from input-state data without any intermediate model identification. Although this direct reinforcement learning approach offers increased adaptability and resilience to model misspecification, its reliance on raw data leaves it vulnerable to system noise and disturbances that may undermine convergence, robustness, and stability. In this article, we establish the convergence, robustness, and stability of value iteration (VI) for data-driven control of stochastic linear quadratic (LQ) systems in discrete-time with entirely unknown dynamics and cost. Our contributions are three-fold. First, we prove that VI is globally exponentially stable for any positive semidefinite initial value matrix in noise-free settings, thereby significantly relaxing restrictive assumptions on initial value functions in existing literature. Second, we extend our analysis to settings with external disturbances, proving that VI maintains small-disturbance input-to-state stability (ISS) and converges within a small neighborhood of the optimal solution when disturbances are sufficiently small. Third, we propose a new non-model-based robust adaptive dynamic programming (ADP) algorithm for adaptive optimal controller design, which, unlike existing procedures, requires no prior knowledge of an initial admissible control policy. Numerical experiments on a ``data center cooling'' problem demonstrate the convergence and stability of the algorithm compared to established methods, highlighting its robustness and adaptability for data-driven control in noisy environments. Finally, we apply the method to dynamic portfolio allocation, demonstrating its practical relevance outside traditional control tasks.