mlsanityData to 14 Oct 2026

How Log-Barrier Helps Exploration in Policy Optimization

arXiv 2603.15001 ↗First posted 16 Mar 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

28 of 42 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2021Convex optimizationcited by 2,515
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  8. 2018Understanding the impact of entropy on policy optimizationcited by 176
  9. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  10. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  11. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  12. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
16 more
  1. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  2. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  3. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  4. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  5. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  6. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  7. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  8. 2023The Role of Baselines in Policy Gradient Optimizationcited by 25
  9. 2021Understanding the Effect of Stochasticity in Policy Optimizationcited by 18
  10. 2021The Fragility of Optimized Bandit Algorithmscited by 17
  11. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradientscited by 12
  12. 2024Stochastic Gradient Succeeds for Banditscited by 10
  13. 2024Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPscited by 7
  14. 2025Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning ratescited by 4
  15. 2019Policy Space Identification in Configurable Environmentscited by 3
  16. 2024Behind the Myth of Exploration in Policy Gradientscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Model-Based Reinforcement Learning with Double Oracle Efficiency in Policy Optimization and Offline Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Diffusion Analysis of Policy Gradient for Stochastic Bandits
  2. 2024Towards Provable Log Density Policy Gradient
  3. 2025Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
  4. 2025Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead
  5. 2026Reinforcement Learning in Real Option Models
  6. 2026Convergence and Regret of the Policy Gradient for Multi-Armed Bandits in Diffusion Environment
  7. 2026Delightful Gradients Accelerate Corner Escape
  8. 2026Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds