mlsanityData to 7 Oct 2026

DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

arXiv 2602.19895 ↗First posted 23 Feb 2026cs.LG cs.CLcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 45 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

31 of 45 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025ESPO: Entropy Importance Sampling Policy Optimization29 Nov
  2. 2025Diversity-Incentivized Exploration for Versatile Reasoning30 Sep
  3. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models29 Sep
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  6. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  9. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  11. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  12. 2024Progress or Regress? Self-Improvement Reversal in Post-training6 Jul
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2025OpenAI GPT-5 System Cardcited by 910
  8. 2024DeepSeek-V3 Technical Reportcited by 724
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  10. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  11. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  12. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
6 more
  1. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  2. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  3. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  4. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  5. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  6. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forgetcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  2. 2026EVE-Agent: Evidence-Verifiable Self-Evolving Agentscited by 2
  3. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  4. 2026Group Entropy-Controlled Policy Optimizationcited by 2
  5. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  6. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  7. 2026The Past Is Not Past: Memory-Enhanced Dynamic Reward Shapingcited by 0
  8. 2026From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scalecited by 0
  9. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  2. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
  3. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  4. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  5. 2025Rethinking Entropy Regularization in Large Reasoning Models
  6. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  7. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  8. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning