mlsanityData to 14 Oct 2026

Delightful Gradients Accelerate Corner Escape

arXiv 2605.11908 ↗First posted 12 May 2026cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Softmax policy gradient converges at O(1/t), but its transient behavior near sub-optimal corners of the simplex can be exponentially slow. The bottleneck is self-trapping: negative-advantage actions reinforce the corner policy and can initially push the optimal action backward. We study Delightful Policy Gradient (DG), which gates each policy-gradient term by the product of advantage and action surprisal. For K-armed bandits, we prove that the zero-temperature limit of DG removes this corner-trapping mechanism on a quantitative sector near any sub-optimal corner, yielding a first-exit escape bound logarithmic in the initial probability ratio. At every fixed temperature, the same local mechanism persists because harmful actions are polynomially suppressed as they become rare. A key structural insight is that every action better than the corner action is an ally: its contribution to escape is non-negative. Combining corner instability with a monotonic value improvement identity, we prove that DG converges globally to the optimal policy in both bandits and tabular MDPs at an asymptotic O(1/t) rate. We also show, via an exact counterexample, that this tabular mechanism can fail under shared function approximation. In MNIST contextual bandits with a shared-parameter neural network, DG nevertheless recovers from bad initializations faster than standard policy gradient, suggesting that the counterexample marks a boundary of the theory rather than a practical prohibition.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 17 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

7 of 17 references.

References · 7

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  3. 2018Understanding the impact of entropy on policy optimizationcited by 176
  4. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  5. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  6. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  7. 2023Policy Mirror Descent Inherently Explores Action Spacecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policynesscited by 0
  2. 2026Training Language Models to Fail Gracefullycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Delightful Policy Gradient
  2. 2025Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
  3. 2026How Log-Barrier Helps Exploration in Policy Optimization
  4. 2026Does This Gradient Spark Joy?
  5. 2026Delightful Distributed Policy Gradient
  6. 2024Behind the Myth of Exploration in Policy Gradients
  7. 2024Structure Matters: Dynamic Policy Gradient
  8. 2024Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs