mlsanityData to 14 Oct 2026

What are Key Factors for Updates in RL for LLM Reasoning?

arXiv 2606.22570 ↗First posted 21 Jun 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning ability of large language models. However, much of the existing work is guided by heuristic intuition, leading to divergent algorithmic choices, even contradictory ones that nevertheless report empirical gains. To better understand this phenomenon, we conduct a theoretical analysis of RLVR updates. Our study reveals that differences in off-policy degree, determined by the number of gradient steps per rollout, substantially affect the distribution of importance sampling ratios and their clipping behavior, thereby altering which tokens dominate the update. Building on this insight, we characterize gradient expectation as the central quantity governing update dynamics and analyze the roles of token probability, advantage, and importance sampling ratio. Motivated by these findings, we propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries across token groups according to the empirical variance of their importance sampling ratios. Experiments on 3B and 7B models across diverse reasoning benchmarks, spanning mathematical problem solving, tabular QA, and logic puzzles, demonstrate that ACPO outperforms strong baselines such as DAPO and CISPO. These results demonstrate that principled, analysis-driven approaches yield more robust and effective RLVR methods. Code is available in: https://github.com/Control-derek/ACPO

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 39 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

34 of 39 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  2. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization11 Aug
  3. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  4. 2025First Return, Entropy-Eliciting Explore9 Jul
  5. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  6. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  7. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  10. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  12. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
11 more
  1. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  3. 2025Learning to Reason under Off-Policy Guidance21 Apr
  4. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  5. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024OpenAI o1 System Cardcited by 1,244
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  9. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  10. 2025One-shot Entropy Minimizationcited by 32
  11. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  3. 2026A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
  4. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  5. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  7. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origin
  8. 2026Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning