mlsanityData to 7 Oct 2026

Diversity-Incentivized Exploration for Versatile Reasoning

arXiv 2509.26209 ↗First posted 30 Sep 2025cs.AIcited by 28

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a crucial paradigm for incentivizing reasoning capabilities in Large Language Models (LLMs). Due to vast state-action spaces and reward sparsity in reasoning tasks, existing methods often struggle with deficient exploration and poor sample efficiency. In the paper, we propose DIVER (Diversity-Incentivized Exploration for VersatilE Reasoning), an innovative framework that highlights the pivotal role of global sequence-level diversity to incentivize deep exploration for versatile reasoning. We first conduct a primary empirical study to reveal a strong positive correlation between global diversity and reasoning capacity. Building on this insight, we introduce global diversity incentives as an intrinsic reward to promote deep exploration in a semantically structured space. Incorporating the intrinsic reward, we develop a potential-based reward shaping mechanism to preserve optimal policy invariance and design simple heuristics to mitigate possible reward hacking. Experimental results show that DIVER outperforms competitive RLVR baselines with various exploration strategies on both in-domain and out-of-domain tasks, excelling in both Pass@1 and Pass@k evaluations. Our code is available at https://github.com/NJU-RL/DIVER.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 63 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

49 of 63 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  3. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR11 Aug
  4. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  7. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  10. 2025Learning to Reason under Off-Policy Guidance21 Apr
  11. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
4 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 33

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2024A Survey on LLM-as-a-Judgecited by 1,384
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
21 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  8. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  9. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  10. 2017Deep Exploration via Randomized Value Functionscited by 155
  11. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  12. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  13. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  14. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  15. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  16. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  17. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Criticscited by 13
  18. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  19. 2025Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learningcited by 7
  20. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  21. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 27.

28 papers.

Sort:

By area:

  1. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  2. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  3. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  4. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  5. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 6
  6. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 4
  7. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAGcited by 3
  8. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweightingcited by 2
  9. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priorscited by 2
  10. 2026Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoningcited by 2
  11. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  12. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?cited by 2
16 more
  1. 2026TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agentscited by 2
  2. 2026All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Modelscited by 1
  3. 2026Discourse Diversity in Multi-Turn Empathic Dialoguecited by 1
  4. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoningcited by 1
  5. 2026DEI: Diversity in Evolutionary Inference for Quality-Diversity Searchcited by 1
  6. 2026Diverse Thinking Schemata Elicit Better Reasoning in Large Language Modelscited by 1
  7. 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policycited by 1
  8. 2026Know Thy Reasoner: Not All Language Models Explore Alikecited by 0
  9. 2026Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generationcited by 0
  10. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoningcited by 0
  11. 2026CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversitycited by 0
  12. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  13. 2026Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidancecited by 0
  14. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  15. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoningcited by 0
  16. 2026Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  2. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
  3. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  4. 2026SD-E2: Semantic Exploration for Reasoning Under Token Budgets
  5. 2026Reinforced Efficient Reasoning via Semantically Diverse Exploration
  6. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
  7. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  8. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward