# Entropy Regularized Reinforcement Learning Using Large Deviation Theory

arXiv: https://arxiv.org/abs/2106.03931 · HTML: https://mlsanity.com/paper/2106.03931/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 47 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Why does deep and cheap learning work so well?](https://mlsanity.com/paper/1608.08225/) (2016)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [An exact mapping between the Variational Renormalization Group and Deep Learning](https://mlsanity.com/paper/1410.3831/) (2014)
- [Topological Quantum Compiling with Reinforcement Learning](https://mlsanity.com/paper/2004.04743/) (2020)
- [A reinforcement learning approach to rare trajectory sampling](https://mlsanity.com/paper/2005.12890/) (2020)
- [Reinforcement learning of rare diffusive dynamics](https://mlsanity.com/paper/2105.04321/) (2021)
- [Evolutionary reinforcement learning of dynamical large deviations](https://mlsanity.com/paper/1909.00835/) (2019)

## Built on by (most cited first, 5 of 5)

- [Average-Reward Soft Actor-Critic](https://mlsanity.com/paper/2501.09080/) (2025)
- [The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective](https://mlsanity.com/paper/2408.09974/) (2024)
- [Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning](https://mlsanity.com/paper/2510.13832/) (2025)
- [Maximum Entropy Exploration Without the Rollouts](https://mlsanity.com/paper/2603.12325/) (2026)
- [Thermodynamics of Reinforcement Learning Curricula](https://mlsanity.com/paper/2603.12324/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov Decision Process (MDP) models of reinforcement learning. The results obtained lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and non-equilibrium statistical mechanics, thereby opening new avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.