# The Role of Identification in Data-driven Policy Iteration: A System Theoretic Study

arXiv: https://arxiv.org/abs/2401.06721 · HTML: https://mlsanity.com/paper/2401.06721/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2024.

## Built on by (most cited first, 3 of 3)

- [Towards a Systems Theory of Algorithms](https://mlsanity.com/paper/2401.14029/) (2024)
- [Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data](https://mlsanity.com/paper/2502.19977/) (2025)
- [Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression](https://mlsanity.com/paper/2512.03764/) (2025)

## Abstract (arXiv)

The goal of this article is to study fundamental mechanisms behind so-called indirect and direct data-driven control for unknown systems. Specifically, we consider policy iteration applied to the linear quadratic regulator problem. Two iterative procedures, where data collected from the system are repeatedly used to compute new estimates of the desired optimal controller, are considered. In indirect policy iteration, data are used to obtain an updated model estimate through a recursive identification scheme, which is used in a certainty-equivalent fashion to perform the classic policy iteration update. By casting the concurrent model identification and control design as a feedback interconnection between two algorithmic systems, we provide a closed-loop analysis that shows convergence and robustness properties for arbitrary levels of excitation in the data. In direct policy iteration, data are used to approximate the value function and design the associated controller without requiring the intermediate identification step. After proposing an extension to a recently proposed scheme that overcomes potential identifiability issues, we establish under which conditions this procedure is guaranteed to deliver the optimal controller. Based on these analyses we are able to compare the strengths and limitations of the two approaches, highlighting aspects such as the required samples, convergence properties, and excitation requirement. Simulations are also provided to illustrate the results.