# Natural Gradient Deep Q-learning

arXiv: https://arxiv.org/abs/1803.07482 · HTML: https://mlsanity.com/paper/1803.07482/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2018.

## Built on by (most cited first, 5 of 5)

- [Bridging the Gap Between Target Networks and Functional Regularization](https://mlsanity.com/paper/2106.02613/) (2021)
- [Faster Deep Reinforcement Learning with Slower Online Network](https://mlsanity.com/paper/2112.05848/) (2021)
- [Toward Efficient Gradient-Based Value Estimation](https://mlsanity.com/paper/2301.13757/) (2023)
- [Learning the Target Network in Function Space](https://mlsanity.com/paper/2406.01838/) (2024)
- [Bridging the Gap Between Target Networks and Functional Regularization](https://mlsanity.com/paper/2210.12282/) (2022)

## Abstract (arXiv)

We present a novel algorithm to train a deep Q-learning agent using natural-gradient techniques. We compare the original deep Q-network (DQN) algorithm to its natural-gradient counterpart, which we refer to as NGDQN, on a collection of classic control domains. Without employing target networks, NGDQN significantly outperforms DQN without target networks, and performs no worse than DQN with target networks, suggesting that NGDQN stabilizes training and can help reduce the need for additional hyperparameter tuning. We also find that NGDQN is less sensitive to hyperparameter optimization relative to DQN. Together these results suggest that natural-gradient techniques can improve value-function optimization in deep reinforcement learning.