# An Information-Theoretic Optimality Principle for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1708.01867 · HTML: https://mlsanity.com/paper/1708.01867/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2017.

## Built on by (most cited first, 3 of 3)

- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Variational Inference for Model-Free and Model-Based Reinforcement Learning](https://mlsanity.com/paper/2209.01693/) (2022)
- [Beyond Bayes-optimality: meta-learning what you know you don't know](https://mlsanity.com/paper/2209.15618/) (2022)

## Abstract (arXiv)

We methodologically address the problem of Q-value overestimation in deep reinforcement learning to handle high-dimensional state spaces efficiently. By adapting concepts from information theory, we introduce an intrinsic penalty signal encouraging reduced Q-value estimates. The resultant algorithm encompasses a wide range of learning outcomes containing deep Q-networks as a special case. Different learning outcomes can be demonstrated by tuning a Lagrange multiplier accordingly. We furthermore propose a novel scheduling scheme for this Lagrange multiplier to ensure efficient and robust learning. In experiments on Atari, our algorithm outperforms other algorithms (e.g. deep and double deep Q-networks) in terms of both game-play performance and sample complexity. These results remain valid under the recently proposed dueling architecture.