mlsanityData to 7 Oct 2026

Munchausen Reinforcement Learning

arXiv 2007.14430 ↗First posted 28 Jul 2020cs.LG stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 11; 2023: 11; 2024: 11; 2025: 11; 2026: 6.

68 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  3. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  4. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  5. 2021Evolving Reinforcement Learning Algorithmscited by 48
  6. 2021Adversarially Guided Actor-Criticcited by 47
  7. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  8. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  9. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  10. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 33
  11. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  12. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
38 more
  1. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  2. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  3. 2021There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learningcited by 17
  4. 2022Learning to Solve Combinatorial Graph Partitioning Problems via Efficient Explorationcited by 14
  5. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  6. 2022KL-Entropy-Regularized RL with a Generative Model is Minimax Optimalcited by 13
  7. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  8. 2024Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximizationcited by 12
  9. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  10. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  11. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  12. 2023Conversational Tree Search: A New Hybrid Dialog Taskcited by 7
  13. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  14. 2021Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processescited by 6
  15. 2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
  16. 2023Regularization of the policy updates for stabilizing Mean Field Gamescited by 5
  17. 2023Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Rewardcited by 5
  18. 2024Random Policy Evaluation Uncovers Policies of Generative Flow Networkscited by 5
  19. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  20. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  21. 2023Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practicecited by 4
  22. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 4
  23. 2025Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learningcited by 4
  24. 2023A Kernel Perspective on Behavioural Metrics for Markov Decision Processescited by 3
  25. 2025SR-Reward: Taking The Path More Traveledcited by 3
  26. 2025ShiQ: Bringing back Bellman to LLMscited by 3
  27. 2021A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learningcited by 2
  28. 2024Reflective Policy Optimizationcited by 2
  29. 2025Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learningcited by 2
  30. 2021Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learningcited by 1
  31. 2022Smoothing Advantage Learningcited by 1
  32. 2022Robust Action Gap Increasing with Clipped Advantage Learningcited by 1
  33. 2023Generalized Munchausen Reinforcement Learning using Tsallis KL Divergencecited by 1
  34. 2025Revealing Human Attention Patterns from Gameplay Analysis for Reinforcement Learningcited by 1
  35. 2025Multi-Agent Reinforcement Learning for Inverse Design in Photonic Integrated Circuitscited by 1
  36. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Gamescited by 1
  37. 2026Perspectives on Tsallis Statistics for Artificial Intelligencecited by 1
  38. 2021Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learningcited by 0

50 most cited of 68 All 68 (JSON)