mlsanityData to 14 Oct 2026

Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning

arXiv 2509.23808 ↗First posted 28 Sep 2025cs.LG cs.CLcited by 17

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning is often framed as balancing exploration and exploitation in action space, typically operationalized with token-level proxies (e.g., output entropy or confidence). We argue that this apparent trade-off is largely a measurement artifact: token-level statistics reflect next-token uncertainty rather than how reasoning progresses over multi-token semantic structures. We therefore study exploration and exploitation in the hidden-state space of response trajectories. We use Effective Rank (ER) to quantify representational exploration and introduce its temporal derivatives, Effective Rank Velocity (ERV) and Effective Rank Acceleration (ERA), to characterize exploitative refinement dynamics. Empirically and theoretically, ER and ERV exhibit near-zero correlation in semantic space, suggesting the two capacities can be improved simultaneously. Motivated by this, we propose Velocity-Exploiting Rank Learning (VERL), which shapes the RLVR advantage with an auxiliary signal derived from ER/ERV and uses the more stable ERA as a meta-control variable to adaptively balance the incentives. Across multiple base models, RLVR algorithms, and reasoning benchmarks, VERL yields consistent improvements, including large gains on challenging tasks (e.g., 21.4% in Gaokao 2024). The code is available at https://github.com/hf618/VERL.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 96 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

22 of 96 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR11 Aug
  3. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models5 Jun
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2024Mixtral of Expertscited by 1,823
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  10. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
  11. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 54
  12. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 33
5 more
  1. 2025DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Modelscited by 30
  2. 2025MUR: Momentum Uncertainty guided Reasoning for Large Language Modelscited by 17
  3. 2025TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compressioncited by 16
  4. 2025TabularMath: Understanding Math Reasoning over Tables with Large Language Modelscited by 5
  5. 2025Incomplete In-context Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
  2. 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Usecited by 8
  3. 2026Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Modelscited by 8
  4. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  5. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  6. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  7. 2026PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Modelscited by 5
  8. 2026Self-Correcting RAG: Enhancing Faithfulness via MMKP Context Selection and NLI-Guided MCTScited by 5
  9. 2026FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoningcited by 4
  10. 2026Exposure Bias Can Alleviate Itself via Directional and Frequency Rectification in Flow Matchingcited by 4
  11. 2026Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Loopingcited by 3
  12. 2026RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Gamecited by 3
5 more
  1. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?cited by 2
  2. 2026Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translationcited by 1
  3. 2026Agent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Modelscited by 0
  4. 2026Answer Probing-Guided Search for Diverse Solution Exploration of LLMscited by 0
  5. 2026Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  2. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
  3. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
  4. 2026You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
  5. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  6. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  7. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  8. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection