mlsanityData to 7 Oct 2026

If MaxEnt RL is the Answer, What is the Question?

arXiv 1910.01913 ↗First posted 4 Oct 2019cs.LG cs.AI stat.MLcited by 27

Not on the map. Why?

Abstract (arXiv)

Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 4; 2023: 5; 2024: 4; 2025: 4.

27 papers.

Sort:

By area:

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  2. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  3. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  4. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  5. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  6. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  7. 2021Regularized Policies are Reward Robustcited by 32
  8. 2023Fast Rates for Maximum Entropy Explorationcited by 32
  9. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27
  10. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  11. 2021Divergence-Regularized Multi-Agent Actor-Criticcited by 15
  12. 2024Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Gamescited by 11
15 more
  1. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  2. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  3. 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7
  4. 2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6
  5. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  6. 2021Towards Automatic Actor-Critic Solutions to Continuous Controlcited by 4
  7. 2021Robust Entropy-regularized Markov Decision Processescited by 4
  8. 2021Planning and Learning Using Adaptive Entropy Tree Searchcited by 3
  9. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  10. 2023Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimalitycited by 2
  11. 2024Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learningcited by 2
  12. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimationcited by 2
  13. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  14. 2022Concurrent Credit Assignment for Data-efficient Reinforcement Learningcited by 0
  15. 2025StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigmcited by 0