Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies

arXiv 2607.17316 ↗First posted 19 Jul 2026cs.LG econ.THcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The softmax policy π(a mid s) propto exp(β Q(s,a)) is the default model of stochastic choice in reinforcement learning (RL). Various justifications based on robustness, exploration, and optimization have been offered in the RL literature, but none uniquely derives the softmax form from first principles. This leaves a basic tension unresolved: the entropy bonus in the soft Bellman equation violates the Independence axiom that underwrites the Markov decision process (MDP) reward structure. We dissolve this tension by distinguishing two kinds of randomness: chance and choice. By restricting von Neumann-Morgenstern (VNM) Independence to environmental lotteries over base prospects, we show that imposing independence of irrelevant alternatives (IIA) and monotonicity on the policy and value functions at choice nodes uniquely determines the Boltzmann policy, the entropy-regularized representation, and the soft Bellman equation. The choice between the soft and hard Bellman equations thus reduces to a design decision: whether the agent values its own ability to choose. We develop RL-specific consequences, including return monotonicity and convergence under generalized discounting, and synthesize the independent lines from economics and information theory that arrive at the same structure, offering a normative assessment of when IIA is appropriate for agent design.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 46 references that are in the map. First posted on arXiv 19 Jul 2026.

Builds on

19 of 46 references.

References · 19

2022Training language models to follow instructions with human feedbackcited by 15,971
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
2019A Theory of Regularized Markov Decision Processescited by 286
2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
2018Understanding the impact of entropy on policy optimizationcited by 176
2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
2023A Definition of Continual Reinforcement Learningcited by 101
2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
2016Unifying task specification in reinforcement learningcited by 41

7 more

2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 37
2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
2025Jackpot! Alignment as a Maximal Lotterycited by 17
2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
2023RLHF and IIA: Perverse Incentivescited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way