mlsanityData to 14 Oct 2026

Diversity-Aware Policy Optimization for Large Language Model Reasoning

arXiv 2505.23433 ↗First posted 29 May 2025cs.LGcited by 41

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The reasoning capabilities of large language models (LLMs) have advanced rapidly, particularly following the release of DeepSeek R1, which has inspired a surge of research into data quality and reinforcement learning (RL) algorithms. Despite the pivotal role diversity plays in RL, its influence on LLM reasoning remains largely underexplored. To bridge this gap, this work presents a systematic investigation into the impact of diversity in RL-based training for LLM reasoning, and proposes a novel diversity-aware policy optimization method. Across evaluations on 12 LLMs, we observe a strong positive correlation between the solution diversity and Potential at k (a novel metric quantifying an LLM's reasoning potential) in high-performing models. This finding motivates our method to explicitly promote diversity during RL training. Specifically, we design a token-level diversity and reformulate it into a practical objective, then we selectively apply it to positive samples. Integrated into the R1-zero training framework, our method achieves a 3.5 percent average improvement across four mathematical reasoning benchmarks, while generating more diverse and robust solutions.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 59 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

53 of 59 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  2. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  5. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  9. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  11. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  12. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models24 Feb
6 more
  1. 2025LIMR: Less is More for RL Scaling17 Feb
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2024Progress or Regress? Self-Improvement Reversal in Post-training6 Jul
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May

Other references · 35

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  9. 2018Understanding the impact of entropy on policy optimizationcited by 176
  10. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  11. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
  12. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
23 more
  1. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  2. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 90
  3. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  4. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  5. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  6. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 54
  7. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
  8. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 38
  9. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  10. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  11. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  12. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
  13. 2022Continuously Discovering Novel Strategies via Reward-Switching Policy Optimizationcited by 23
  14. 2019Learning Novel Policies For Taskscited by 20
  15. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  16. 2023Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learningcited by 20
  17. 2023Policy Space Diversity for Non-Transitive Gamescited by 15
  18. 2020Non-local Policy Optimization via Diversity-regularized Collaborative Explorationcited by 13
  19. 2024How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problemscited by 12
  20. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  21. 2023DeepScaler: Holistic Autoscaling for Microservices Based on Spatiotemporal GNN with Adaptive Graph Learningcited by 7
  22. 2020Multiple Plans are Better than One: Diverse Stochastic Planningcited by 5
  23. 2024Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 27.

41 papers.

Sort:

By area:

  1. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  2. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  3. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  4. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 20
  5. 2025Revisiting LLM Reasoning via Information Bottleneckcited by 19
  6. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  7. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  8. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  9. 2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinkingcited by 10
  10. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  11. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarkscited by 8
  12. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVRcited by 8
29 more
  1. 2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7
  2. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  3. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 6
  4. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 5
  5. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 5
  6. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  7. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  8. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 3
  9. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  10. 2025Adaptive Originality Filtering: Rejection Based Prompting and RiddleScore for Culturally Grounded Multilingual Riddle Generationcited by 1
  11. 2026Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversitycited by 1
  12. 2026Discourse Diversity in Multi-Turn Empathic Dialoguecited by 1
  13. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoningcited by 1
  14. 2026SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoningcited by 1
  15. 2026Diverse Thinking Schemata Elicit Better Reasoning in Large Language Modelscited by 1
  16. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1
  17. 2025VOYAGER: A Training Free Approach for Generating Diverse Datasets using LLMscited by 0
  18. 2026Residual Skill Optimization for Text-to-SQL Ensemblescited by 0
  19. 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Mattercited by 0
  20. 2026Agent Economics: An Entropy-Controlled Pluralistic Alignment Framework for Preventing Artificial Hivemind in Autonomous Agentscited by 0
  21. 2026Reinforcement Learning without Ground-Truth Solutions can Improve LLMscited by 0
  22. 2026ReCo: Reweighting GRPO Against Distributional Concentrationcited by 0
  23. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  24. 2026Planned Test-Time Scaling with Coordinated Reasoning Pathscited by 0
  25. 2026Improving the Diversity of LLM Outputs without a Trade-offcited by 0
  26. 2026Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMscited by 0
  27. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0
  28. 2026Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoningcited by 0
  29. 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  2. 2025Outcome-based Exploration for LLM Reasoning
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  5. 2025Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
  6. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  7. 2025Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
  8. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement