mlsanityData to 7 Oct 2026

Vanishing L2 regularization for the softmax Multi Armed Bandit

arXiv 2605.03752 ↗First posted 5 May 2026cs.LG math.ST stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi Armed Bandit (MAB) algorithms are a cornerstone of reinforcement learning and have been studied both theoretically and numerically. One of the most commonly used implementation uses a softmax mapping to prescribe the optimal policy and served as the foundation for downstream algorithms, including REINFORCE. Distinct from vanilla approaches, we consider here the L2 regularized softmax policy gradient where a quadratic term is subtracted from the mean reward. Previous studies exploiting convexity failed to identify a suitable theoretical framework to analyze its convergence when the regularization parameter vanishes. We prove here theoretical convergence results and confirm empirically that this regime makes the L2 regularization numerically advantageous on standard benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 37 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

19 of 37 references.

References · 19

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2016End to End Learning for Self-Driving Carscited by 1,000
  3. 2019Introduction to Multi-Armed Banditscited by 475
  4. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  6. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
  7. 2021Reinforcement learning based recommender systems: A surveycited by 187
  8. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  9. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  10. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  11. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  12. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
7 more
  1. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  2. 2020On the Convergence of SGD with Biased Gradientscited by 79
  3. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  4. 2021The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learningcited by 29
  5. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  6. 2025Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularizationcited by 3
  7. 2024Convergence of a L2 regularized Policy Gradient Algorithm for the Multi Armed Banditcited by 2

Built on by

Not cited yet.