# Diversity-Incentivized Exploration for Versatile Reasoning

arXiv: https://arxiv.org/abs/2509.26209 · HTML: https://mlsanity.com/paper/2509.26209/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 49)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)

## Built on by (most cited first, 28 of 28)

- [Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL](https://mlsanity.com/paper/2602.13035/) (2026)
- [Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR](https://mlsanity.com/paper/2605.10781/) (2026)
- [DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning](https://mlsanity.com/paper/2602.19895/) (2026)
- [Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity](https://mlsanity.com/paper/2605.00365/) (2026)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAG](https://mlsanity.com/paper/2603.03292/) (2026)
- [ReLaX: Reasoning with Latent Exploration for Large Reasoning Models](https://mlsanity.com/paper/2512.07558/) (2025)
- [TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents](https://mlsanity.com/paper/2608.22793/) (2026)
- [MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting](https://mlsanity.com/paper/2601.09085/) (2026)
- [How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors](https://mlsanity.com/paper/2605.08817/) (2026)
- [Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning](https://mlsanity.com/paper/2605.27000/) (2026)
- [Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR](https://mlsanity.com/paper/2605.28295/) (2026)
- [Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy](https://mlsanity.com/paper/2608.02087/) (2026)
- [BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?](https://mlsanity.com/paper/2608.02867/) (2026)
- [All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models](https://mlsanity.com/paper/2604.00479/) (2026)
- [Discourse Diversity in Multi-Turn Empathic Dialogue](https://mlsanity.com/paper/2604.11742/) (2026)
- [Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoning](https://mlsanity.com/paper/2605.11461/) (2026)
- [DEI: Diversity in Evolutionary Inference for Quality-Diversity Search](https://mlsanity.com/paper/2605.27130/) (2026)
- [Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation](https://mlsanity.com/paper/2605.28022/) (2026)
- [Diverse Thinking Schemata Elicit Better Reasoning in Large Language Models](https://mlsanity.com/paper/2606.08974/) (2026)
- [Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2608.02149/) (2026)
- [Know Thy Reasoner: Not All Language Models Explore Alike](https://mlsanity.com/paper/2604.10827/) (2026)
- [CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity](https://mlsanity.com/paper/2608.07460/) (2026)
- [Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space](https://mlsanity.com/paper/2608.29188/) (2026)
- [Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidance](https://mlsanity.com/paper/2609.24066/) (2026)
- [Selecting Diverse SFT Traces Improves Post-RL Generalization](https://mlsanity.com/paper/2609.33780/) (2026)
- [Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning](https://mlsanity.com/paper/2609.33781/) (2026)
- [Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization](https://mlsanity.com/paper/2609.39599/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a crucial paradigm for incentivizing reasoning capabilities in Large Language Models (LLMs). Due to vast state-action spaces and reward sparsity in reasoning tasks, existing methods often struggle with deficient exploration and poor sample efficiency. In the paper, we propose DIVER (Diversity-Incentivized Exploration for VersatilE Reasoning), an innovative framework that highlights the pivotal role of global sequence-level diversity to incentivize deep exploration for versatile reasoning. We first conduct a primary empirical study to reveal a strong positive correlation between global diversity and reasoning capacity. Building on this insight, we introduce global diversity incentives as an intrinsic reward to promote deep exploration in a semantically structured space. Incorporating the intrinsic reward, we develop a potential-based reward shaping mechanism to preserve optimal policy invariance and design simple heuristics to mitigate possible reward hacking. Experimental results show that DIVER outperforms competitive RLVR baselines with various exploration strategies on both in-domain and out-of-domain tasks, excelling in both Pass@1 and Pass@k evaluations. Our code is available at https://github.com/NJU-RL/DIVER.