# Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning

arXiv: https://arxiv.org/abs/1909.05950 · HTML: https://mlsanity.com/paper/1909.05950/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2019.

## Built on by (most cited first, 18 of 18)

- [Reinforcement Learning with Simple Sequence Priors](https://mlsanity.com/paper/2305.17109/) (2023)
- [Hierarchically Structured Task-Agnostic Continual Learning](https://mlsanity.com/paper/2211.07725/) (2022)
- [Mixture-of-Variational-Experts for Continual Learning](https://mlsanity.com/paper/2110.12667/) (2021)
- [Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow](https://mlsanity.com/paper/2103.14407/) (2021)
- [Planning Not to Talk: Multiagent Systems that are Robust to Communication Loss](https://mlsanity.com/paper/2201.06619/) (2022)
- [Solving Dynamic Principal-Agent Problems with a Rationally Inattentive Principal](https://mlsanity.com/paper/2202.01691/) (2022)
- [Mutual Information Optimal Control of Discrete-Time Linear Systems](https://mlsanity.com/paper/2507.04712/) (2025)
- [Temporal Difference Learning with Constrained Initial Representations](https://mlsanity.com/paper/2602.11800/) (2026)
- [A space of goals: the cognitive geometry of informationally bounded agents](https://mlsanity.com/paper/2111.03699/) (2021)
- [Variational Inference for Model-Free and Model-Based Reinforcement Learning](https://mlsanity.com/paper/2209.01693/) (2022)
- [On Policy Stochasticity in Mutual Information Optimal Control of Linear Systems](https://mlsanity.com/paper/2507.21543/) (2025)
- [Behavior-Consistent Deep Reinforcement Learning](https://mlsanity.com/paper/2605.21214/) (2026)
- [Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning](https://mlsanity.com/paper/2311.03711/) (2023)
- [Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learning](https://mlsanity.com/paper/2505.04193/) (2025)
- [Maximum Total Correlation Reinforcement Learning](https://mlsanity.com/paper/2505.16734/) (2025)
- [Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light](https://mlsanity.com/paper/2507.11482/) (2025)
- [Mutual Information Optimal Density Control of Linear Systems and Generalized Schr"odinger Bridges with Reference Refinement](https://mlsanity.com/paper/2605.09349/) (2026)
- [Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning](https://mlsanity.com/paper/2609.28737/) (2026)

## Abstract (arXiv)

Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it tackles the value overestimation problem. It also leads to a significant performance increase in tabular and high-dimensional settings, as demonstrated via algorithms such as soft Q-learning (SQL) and soft actor-critic (SAC). Cumulative entropy regularization has been extended to optimize over the reference marginal policy instead of keeping it fixed, yielding a regularization that minimizes the mutual information between states and actions. While this has been initially proposed for Markov Decision Processes (MDPs) in tabular settings, it was recently shown that a similar principle leads to significant improvements over vanilla SQL in RL for high-dimensional domains with discrete actions and function approximators. Here, we follow the motivation of mutual-information regularization from an inference perspective and theoretically analyze the corresponding Bellman operator. Inspired by this Bellman operator, we devise a novel mutual-information regularized actor-critic learning (MIRACLE) algorithm for continuous action spaces that optimizes over the reference marginal policy. We empirically validate MIRACLE in the Mujoco robotics simulator, where we demonstrate that it can compete with contemporary RL methods. Most notably, it can improve over the model-free state-of-the-art SAC algorithm which implicitly assumes a fixed reference policy.