mlsanityData to 14 Oct 2026

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood

arXiv 2604.12736 ↗First posted 14 Apr 2026cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly in their mathemat ical reasoning performance. However, GRPO and related entropy regularization methods still struggle with token-level sparse-rewards, which is an inherent chal lenge in chain-of-thought (CoT) reasoning. These approaches often rely on undifferen tiated token-level entropy regularization, which easily leads to entropy collapse or model degradation under sparse token rewards. In this work, we propose TEPO, a novel token-level framework that (1) leverages sequence-level likelihood to link group-level rewards with individual tokens via token-level aggregation, and (2) introduces a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. Experiments demonstrate that TEPO not only achieves state-of-the-art performance on mathematical reasoning benchmarks but also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 24 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

12 of 24 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  3. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  4. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 5

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
  2. 2026Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
  3. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  4. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
  5. 2025Geometric-Mean Policy Optimization
  6. 2025Token-Efficient RL for LLM Reasoning
  7. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
  8. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models