Logit Dynamics in Softmax Policy Gradient Methods
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector: |Delta z|2 propto √1-2Pc + C(P) This equation demonstrates that update magnitudes are determined by the chosen action's probability (Pc) and the policy's collision probability (C(P)), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 0 of its 2 references that are in the map. First posted on arXiv 15 Jun 2025.
Builds on
0 of 2 references.
No references here yet.
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
- 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
- 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
- 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimizationcited by 14
- 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization22 Feb 2026
- 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents11 Sep 2025
- 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep 2025
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Softmax and Entropy: Convergence Rates of Policy Gradients with f-SoftArgmax Parameterization & Coupled Regularization
- 2026Sharp Spectral Thresholds for Logit Fixed Points
- 2024Elementary Analysis of Policy Gradient Methods
- 2026Global linear convergence of entropy-regularized softmax policy gradient beyond tabular MDPs
- 2024Towards Provable Log Density Policy Gradient
- 2025Complexity-Regularized Proximal Policy Optimization
- 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processes
- 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic