mlsanityData to 14 Oct 2026

SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs

arXiv 2605.18864 ↗First posted 15 May 2026cs.LG cs.AI cs.CLcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent studies observe that reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k, raising the question of whether RLVR genuinely enables large language models to acquire novel reasoning abilities or merely enhances the efficiency of sampling reasoning modes already present in the base model. Prior analyses largely support the latter view, attributing this limitation to structural properties of standard RLVR objectives that result in insufficient exploration pressure. In this work, we argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution, thereby suppressing the emergence of alternative reasoning modes. However, we show that neither removing the KL term nor replacing it with forward-KL provides a satisfactory solution, as both disrupt the efficiency-coverage trade-off by either inducing reward hacking or allocating probability mass to off-target regions. To resolve this tension, we propose SAGE, a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself through a guide function q(x,y), achieving consistent improvements in both pass@1 and pass@k across challenging mathematical reasoning benchmarks. Our code is available at https://github.com/tally0818/SAGE.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 40 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

30 of 40 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning12 Oct
  2. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration4 Oct
  3. 2025Rethinking Entropy Regularization in Large Reasoning Models29 Sep
  4. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration19 Aug
  5. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  6. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  7. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  8. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
  9. 2025BNPO: Beta Normalization Policy Optimization3 Jun
  10. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  11. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning23 May
  12. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
4 more
  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 14

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  9. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  10. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  11. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  12. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
2 more
  1. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  2. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 1
  2. 2026The Model Knows Another Way: Strategy Switching for Effective RLVR Explorationcited by 1
  3. 2026Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  4. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
  5. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
  6. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
  7. 2026Decoupling Exploration from Optimization in RLVR
  8. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers