# Deep Q-learning: a robust control approach

arXiv: https://arxiv.org/abs/2201.08610 · HTML: https://mlsanity.com/paper/2201.08610/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 21 Jan 2022.

## Builds on (most cited first, 11 of 11)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [Observe and Look Further: Achieving Consistent Performance on Atari](https://mlsanity.com/paper/1805.11593/) (2018)
- [Towards Characterizing Divergence in Deep Q-Learning](https://mlsanity.com/paper/1903.08894/) (2019)
- [Enforcing robust control guarantees within neural network policies](https://mlsanity.com/paper/2011.08105/) (2020)
- [Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.10056/) (2018)

## Built on by (most cited first, 1 of 1)

- [Controlled Descent Training](https://mlsanity.com/paper/2303.09216/) (2023)

## Abstract (arXiv)

In this paper, we place deep Q-learning into a control-oriented perspective and study its learning dynamics with well-established techniques from robust control. We formulate an uncertain linear time-invariant model by means of the neural tangent kernel to describe learning. We show the instability of learning and analyze the agent's behavior in frequency-domain. Then, we ensure convergence via robust controllers acting as dynamical rewards in the loss function. We synthesize three controllers: state-feedback gain scheduling H2, dynamic Hinf, and constant gain Hinf controllers. Setting up the learning agent with a control-oriented tuning methodology is more transparent and has well-established literature compared to the heuristics in reinforcement learning. In addition, our approach does not use a target network and randomized replay memory. The role of the target network is overtaken by the control input, which also exploits the temporal dependency of samples (opposed to a randomized memory buffer). Numerical simulations in different OpenAI Gym environments suggest that the Hinf controlled learning performs slightly better than Double deep Q-learning.