# If MaxEnt RL is the Answer, What is the Question?

arXiv: https://arxiv.org/abs/1910.01913 · HTML: https://mlsanity.com/paper/1910.01913/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2019.

## Built on by (most cited first, 27 of 27)

- [Extreme Q-Learning: MaxEnt RL without Entropy](https://mlsanity.com/paper/2301.02328/) (2023)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [Offline RL Policies Should be Trained to be Adaptive](https://mlsanity.com/paper/2207.02200/) (2022)
- [Same State, Different Task: Continual Reinforcement Learning without Interference](https://mlsanity.com/paper/2106.02940/) (2021)
- [DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning](https://mlsanity.com/paper/2203.06662/) (2022)
- [Towards Efficient Exact Optimization of Language Model Alignment](https://mlsanity.com/paper/2402.00856/) (2024)
- [Fast Rates for Maximum Entropy Exploration](https://mlsanity.com/paper/2303.08059/) (2023)
- [Regularized Policies are Reward Robust](https://mlsanity.com/paper/2101.07012/) (2021)
- [Model-Based Reinforcement Learning via Latent-Space Collocation](https://mlsanity.com/paper/2106.13229/) (2021)
- [Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularization](https://mlsanity.com/paper/2110.10117/) (2021)
- [Divergence-Regularized Multi-Agent Actor-Critic](https://mlsanity.com/paper/2110.00304/) (2021)
- [Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games](https://mlsanity.com/paper/2402.13399/) (2024)
- [Efficient Exploration via Epistemic-Risk-Seeking Policy Optimization](https://mlsanity.com/paper/2302.09339/) (2023)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Model-Free Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.02907/) (2021)
- [SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Critics](https://mlsanity.com/paper/2204.09424/) (2022)
- [State Entropy Regularization for Robust Reinforcement Learning](https://mlsanity.com/paper/2506.07085/) (2025)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [Robust Entropy-regularized Markov Decision Processes](https://mlsanity.com/paper/2112.15364/) (2021)
- [Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation](https://mlsanity.com/paper/2407.18143/) (2024)
- [Planning and Learning Using Adaptive Entropy Tree Search](https://mlsanity.com/paper/2102.06808/) (2021)
- [Off-policy Reinforcement Learning with Model-based Exploration Augmentation](https://mlsanity.com/paper/2510.25529/) (2025)
- [Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality](https://mlsanity.com/paper/2303.12785/) (2023)
- [Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.00787/) (2024)
- [Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning](https://mlsanity.com/paper/2510.08526/) (2025)
- [Concurrent Credit Assignment for Data-efficient Reinforcement Learning](https://mlsanity.com/paper/2205.12020/) (2022)
- [StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm](https://mlsanity.com/paper/2512.16444/) (2025)

## Abstract (arXiv)

Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.