mlsanityData to 14 Oct 2026

The Confusing Instance Principle for Online Linear Quadratic Control

arXiv 2510.19531 ↗First posted 22 Oct 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

We revisit the problem of controlling linear systems with quadratic cost under unknown dynamics with model-based reinforcement learning. Traditional methods like Optimism in the Face of Uncertainty and Thompson Sampling, rooted in multi-armed bandits (MABs), face practical limitations. In contrast, we propose an alternative based on the Confusing Instance (CI) principle, which underpins regret lower bounds in MABs and discrete Markov Decision Processes (MDPs) and is central to the Minimum Empirical Divergence (MED) family of algorithms, known for their asymptotic optimality in various settings. By leveraging the structure of LQR policies along with sensitivity and stability analysis, we develop MED-LQ. This novel control strategy extends the principles of CI and MED beyond small-scale settings. Our benchmarks on a comprehensive control suite demonstrate that MED-LQ achieves competitive performance in various scenarios while highlighting its potential for broader applications in large-scale MDPs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 57 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

27 of 57 references.

References · 27

  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  5. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  6. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  7. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  8. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  9. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  10. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  11. 2018Online Linear Quadratic Controlcited by 56
  12. 2016The End of Optimism? An Asymptotic Analysis of Finite-Armed Linear Banditscited by 49
15 more
  1. 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 40
  2. 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 30
  3. 2020Tightening Exploration in Upper Confidence Reinforcement Learningcited by 25
  4. 2020Reinforcement Learning with Fast Stabilization in Linear Dynamical Systemscited by 20
  5. 2020Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxationcited by 18
  6. 2017Thompson Sampling for Linear-Quadratic Control Problemscited by 14
  7. 2021Maillard Sampling: Boltzmann Exploration Done Optimallycited by 13
  8. 2022Thompson Sampling Achieves tilde O(√T) Regret in Linear Quadratic Controlcited by 8
  9. 2022Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systemscited by 5
  10. 2021Indexed Minimum Empirical Divergence for Unimodal Banditscited by 4
  11. 2023A General Recipe for the Analysis of Randomized Multi-Armed Bandit Algorithmscited by 4
  12. 2024Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPscited by 4
  13. 2025The regret lower bound for communicating Markov Decision Processescited by 4
  14. 2024Minimum Empirical Divergence for Sub-Gaussian Linear Banditscited by 3
  15. 2024Indexed Minimum Empirical Divergence-Based Algorithms for Linear Banditscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025How Hard is it to Confuse a World Model?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimistic Online LQR via Intrinsic Rewards
  2. 2024Foundations of Safe Online Reinforcement Learning in the Linear Quadratic Regulator: Generalized Baselines
  3. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems
  4. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
  5. 2024Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lens
  6. 2024Scalable Reinforcement Learning for Linear-Quadratic Control of Networks
  7. 2026Model-Free Q-Learning for Infinite-Horizon Stochastic Linear Quadratic Problems with Regime Switching
  8. 2025Foundations of Safe Online Reinforcement Learning in the Linear Quadratic Regulator: √T-Regret