# Logit Dynamics in Softmax Policy Gradient Methods

arXiv: https://arxiv.org/abs/2506.12912 · HTML: https://mlsanity.com/paper/2506.12912/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 0 of its 2 references that are in the map.
- First posted on arXiv 15 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Built on by (most cited first, 3 of 3)

- [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025)
- [Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents](https://mlsanity.com/paper/2509.09265/) (2025)
- [How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization](https://mlsanity.com/paper/2602.19208/) (2026)

## Abstract (arXiv)

We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector:  |Delta z|2 propto √1-2Pc + C(P)  This equation demonstrates that update magnitudes are determined by the chosen action's probability (Pc) and the policy's collision probability (C(P)), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods.