# Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problems

arXiv: https://arxiv.org/abs/2105.07761 · HTML: https://mlsanity.com/paper/2105.07761/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2021.

## Built on by (most cited first, 9 of 9)

- [An Efficient Off-Policy Reinforcement Learning Algorithm for the Continuous-Time LQR Problem](https://mlsanity.com/paper/2303.17819/) (2023)
- [Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Survey](https://mlsanity.com/paper/2508.20315/) (2025)
- [Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient](https://mlsanity.com/paper/2403.05367/) (2024)
- [MR-ARL: Model Reference Adaptive Reinforcement Learning for Robustly Stable On-Policy Data-Driven LQR](https://mlsanity.com/paper/2402.14483/) (2024)
- [Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint](https://mlsanity.com/paper/2403.16767/) (2024)
- [Data-Based Efficient Off-Policy Stabilizing Optimal Control Algorithms for Discrete-Time Linear Systems via Damping Coefficients](https://mlsanity.com/paper/2412.20845/) (2024)
- [Adaptive Inverse Reinforcement Learning with Online Off-Policy Data Collection](https://mlsanity.com/paper/2511.15171/) (2025)
- [An Output Feedback Q-learning Algorithm for Optimal Control of Nonlinear Systems with Koopman Linear Embedding](https://mlsanity.com/paper/2603.29858/) (2026)
- [Model-free LQG Control with Chance Constraints](https://mlsanity.com/paper/2605.31310/) (2026)

## Abstract (arXiv)

This paper introduces and analyzes an improved Q-learning algorithm for discrete-time linear time-invariant systems. The proposed method does not require any knowledge of the system dynamics, and it enjoys significant efficiency advantages over other data-based optimal control methods in the literature. This algorithm can be fully executed off-line, as it does not require to apply the current estimate of the optimal input to the system as in on-policy algorithms. It is shown that a persistently exciting input, defined from an easily tested matrix rank condition, guarantees the convergence of the algorithm. A data-based method is proposed to design the initial stabilizing feedback gain that the algorithm requires. Robustness of the algorithm in the presence of noisy measurements is analyzed. We compare the proposed algorithm in simulation to different direct and indirect data-based control design methods.