mlsanityData to 7 Oct 2026

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

arXiv 2603.22117 ↗First posted 23 Mar 2026cs.LG cs.AIcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference Deltalog p between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that Deltalog p more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a textittest-time extrapolation method that amplifies the policy along the learned Deltalog p direction to improve reasoning accuracy without further training; (2) a textittraining-time reweighting method that focuses learning on low-probability (corresponding to higher Deltalog p) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 41 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

36 of 41 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning3 Sep
  2. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  3. 2025Geometric-Mean Policy Optimization28 Jul
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning27 May
  7. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
4 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2025Group Sequence Policy Optimizationcited by 788
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
8 more
  1. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  2. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  3. 2024Learning Dynamics of LLM Finetuningcited by 93
  4. 2024Decoding-time Realignment of Language Modelscited by 67
  5. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  6. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  7. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
  8. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  2. 2026A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillationcited by 5
  3. 2026APPO: Agentic Procedural Policy Optimizationcited by 3
  4. 2026Policy Improvement Reinforcement Learningcited by 2
  5. 2026One-Way Policy Optimization for Self-Evolving LLMscited by 2
  6. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 0
  7. 2026Not only where, But when: Temporal Scheduling for RLVRcited by 0
  8. 2026Mechanism-Guided Selective Unlearning for RLVR-Induced Reasoningcited by 0
  9. 2026Preference Tuning as Spectral Update Reorganizationcited by 0
  10. 2026Beyond Generation and Accuracy: Diagnosing and Enhancing Visual Chain-of-Thought for Geometry Problem Solvingcited by 0
  11. 2026From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillationcited by 0
  12. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  3. 2026What are Key Factors for Updates in RL for LLM Reasoning?
  4. 2025The Path Not Taken: RLVR Provably Learns Off the Principals
  5. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  6. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  7. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  8. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models