mlsanityData to 7 Oct 2026

q-Munchausen Reinforcement Learning

arXiv 2205.07467 ↗First posted 16 May 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The recently successful Munchausen Reinforcement Learning (M-RL) features implicit Kullback-Leibler (KL) regularization by augmenting the reward function with logarithm of the current stochastic policy. Though significant improvement has been shown with the Boltzmann softmax policy, when the Tsallis sparsemax policy is considered, the augmentation leads to a flat learning curve for almost every problem considered. We show that it is due to the mismatch between the conventional logarithm and the non-logarithmic (generalized) nature of Tsallis entropy. Drawing inspiration from the Tsallis statistics literature, we propose to correct the mismatch of M-RL with the help of q-logarithm/exponential functions. The proposed formulation leads to implicit Tsallis KL regularization under the maximum Tsallis entropy framework. We show such formulation of M-RL again achieves superior performance on benchmark problems and sheds light on more general M-RL with various entropic indices q.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 36 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

17 of 36 references.

References · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  5. 2019A Theory of Regularized Markov Decision Processescited by 279
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  7. 2018Understanding the impact of entropy on policy optimizationcited by 171
  8. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  9. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  10. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  11. 2020Munchausen Reinforcement Learningcited by 68
  12. 2021Adversarially Guided Actor-Criticcited by 47
5 more
  1. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  2. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  3. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  4. 2019Momentum in Reinforcement Learningcited by 19
  5. 2018Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularizationcited by 8

Built on by

Not cited yet.