mlsanityData to 14 Oct 2026

Mirror Descent Actor Critic via Bounded Advantage Learning

arXiv 2502.03854 ↗First posted 6 Feb 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates. Despite its empirical success in discrete action domains and strong theoretical guarantees, the performance of KL-entropy-regularized methods does not surpass that of a strong entropy-only-regularized method in continuous action domains. In this study, we propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains, and show that its empirical performance is significantly boosted by bounding the actor's log-density terms in the critic's loss function, compared to a non-bounded naive instantiation. Further, we relate MDAC to Advantage Learning by recalling that the actor's log-probability is equal to the regularized advantage function in tabular cases, and theoretically discuss when and why bounding the advantage terms is validated and beneficial. We also empirically explore effective choices for the bounding functions, and show that MDAC performs better than strong non-regularized and entropy-only-regularized methods with an appropriate choice of the bounding functions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 48 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

26 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Divergence-Augmented Policy Optimization25 Jan

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  10. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2019A Theory of Regularized Markov Decision Processescited by 286
13 more
  1. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  3. 2020Mirror Descent Policy Optimizationcited by 106
  4. 2020Munchausen Reinforcement Learningcited by 69
  5. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  6. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  7. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  8. 2019Policy Optimization with Stochastic Mirror Descentcited by 18
  9. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 18
  10. 2022KL-Entropy-Regularized RL with a Generative Model is Minimax Optimalcited by 13
  11. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  12. 2022Robust Action Gap Increasing with Clipped Advantage Learningcited by 1
  13. 2023Generalized Munchausen Reinforcement Learning using Tsallis KL Divergencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence
  2. 2026An analysis of Mirror-Descent Soft Actor-Critic
  3. 2025StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
  4. 2025On the Effect of Regularization in Policy Mirror Descent
  5. 2025Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces
  6. 2026Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
  7. 2026Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity
  8. 2026Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces