mlsanityData to 7 Oct 2026

Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization

arXiv 2609.39402 ↗First posted 30 Sep 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 39 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

36 of 39 references.

Closest ancestors · 24

Same direction, newest first

  1. 2026RTMC: Step-Level Credit Assignment via Rollout Trees13 Apr
  2. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models30 Mar
  3. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs23 Mar
  4. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation23 Mar
  5. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization20 Mar
  6. 2026ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning10 Feb
  7. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning2 Feb
  8. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning12 Oct
  9. 2025GRPO-λ: Credit Assignment improves LLM Reasoning30 Sep
  10. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs22 Sep
  11. 2025Single-stream Policy Optimization16 Sep
  12. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
12 more
  1. 2025Geometric-Mean Policy Optimization28 Jul
  2. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs23 Jun
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library6 Jun
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  6. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  7. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning21 Apr
  8. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  11. 2023Let's Verify Step by Step31 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2025Group Sequence Policy Optimizationcited by 788
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  8. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  9. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  10. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 16
  11. 2025Measuring Reasoning Utility in LLMs via Conditional Entropy Reductioncited by 6
  12. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  3. 2026ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate
  4. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
  5. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR
  6. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  7. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
  8. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning