# Ascent Conditions for Length-Normalized Single-Rollout Policy Updates

arXiv: https://arxiv.org/abs/2610.14670 · HTML: https://mlsanity.com/paper/2610.14670/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [The Mirage of Action-Dependent Baselines in Reinforcement Learning](https://mlsanity.com/paper/1802.10031/) (2018)
- [Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.07508/) (2026)
- [On-Policy RL with Optimal Reward Baseline](https://mlsanity.com/paper/2505.23585/) (2025)
- [Trust Region Masking for Long-Horizon LLM Reinforcement Learning](https://mlsanity.com/paper/2512.23075/) (2025)
- [Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective](https://mlsanity.com/paper/2605.07331/) (2026)
- [Best Practice Critic Optimization](https://mlsanity.com/paper/2608.23566/) (2026)
- [On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards](https://mlsanity.com/paper/2607.23364/) (2026)
- [SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation](https://mlsanity.com/paper/2608.09271/) (2026)
- [On KL-Regularized Policy Optimization](https://mlsanity.com/paper/2610.08963/) (2026)

## Abstract (arXiv)

Dividing each trajectory's score contribution by its realized length can change the direction of a reward-centered policy update. In a finite trajectory model, we derive an exact ascent threshold for a three-outcome softmax policy and exhibit a binary-reward example in which the expected Euclidean update decreases task success. Natural-gradient ascent holds for fully controllable outcome distributions with deterministic binary labels, but can fail under a one-parameter restriction. A third-order U-statistic estimates the first-order return change without bias from independent current-policy trajectories, one per request. In the example, a same-sample plug-in estimator instead has the opposite sign in expectation for batch sizes two through forty-nine. Exact policy-grid enumeration confirms the local sign reversal. In a branching process, length-normalized learning initially loses return, then recovers and exceeds the unnormalized comparator. The analysis concerns the expected update field at the current policy.