mlsanityData to 7 Oct 2026

Jointly Reinforcing Diversity and Quality in Language Model Generations

arXiv 2509.02534 ↗First posted 2 Sep 2025cs.CL cs.LGcited by 79

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Post-training of Large Language Models (LMs) often prioritizes accuracy and helpfulness at the expense of diversity. This creates a tension: while post-training improves response quality, it also sharpens output distributions and reduces the range of ideas, limiting the usefulness of LMs in creative and exploratory tasks such as brainstorming, storytelling, or problem solving. We address this challenge with Diversity-Aware Reinforcement Learning (DARLING), a framework that jointly optimizes for response quality and semantic diversity. At its core, DARLING introduces a learned partition function to measure diversity beyond surface-level lexical variations. This diversity signal is then combined with a quality reward during online reinforcement learning, encouraging models to generate outputs that are both high-quality and distinct. Experiments across multiple model families and sizes show that DARLING generalizes to two regimes: non-verifiable tasks (instruction following and creative writing) and verifiable tasks (competition math). On five benchmarks in the first setting, DARLING consistently outperforms quality-only RL baselines, producing outputs that are simultaneously of higher quality and novelty. In the second setting, DARLING achieves higher pass@1 (solution quality) and pass@k (solution variety). Most strikingly, explicitly optimizing for diversity catalyzes exploration in online RL, which manifests itself as higher-quality responses.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 61 of its 83 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

61 of 83 references.

References · 61

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
38 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  7. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  9. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  10. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  11. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  12. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  13. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  14. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  15. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  16. 2023Protein Design with Guided Discrete Diffusioncited by 154
  17. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  18. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  19. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  20. 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 92
  21. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  22. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  23. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  24. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 78
  25. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  26. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 74
  27. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  28. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  29. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  30. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
  31. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  32. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  33. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  34. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  35. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 40
  36. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 40
  37. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 39
  38. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 58.

79 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  3. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  4. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  5. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  6. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  7. 2025Entropy After </Think> for reasoning model early exitingcited by 24
  8. 2026Towards Execution-Grounded Automated AI Researchcited by 19
  9. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  10. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  11. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 13
  12. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMscited by 12
38 more
  1. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  2. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  3. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoningcited by 10
  4. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 10
  5. 2026Where does output diversity collapse in post-training?cited by 10
  6. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  7. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 9
  8. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  9. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVRcited by 8
  10. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  11. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  12. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  13. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  14. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 7
  15. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  16. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 6
  17. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6
  18. 2025Dual-Uncertainty Guided Policy Learning for Multimodal Reasoningcited by 5
  19. 2026Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learningcited by 5
  20. 2026Free Lunch for Pass@k? Low Cost Diverse Sampling for Diffusion Language Modelscited by 5
  21. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 5
  22. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  23. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 4
  24. 2026Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@K Crossover on a Free-Verifier Domaincited by 4
  25. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  26. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  27. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  28. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 3
  29. 2026SPIRAL: Learning to Search and Aggregatecited by 3
  30. 2026The One-Word Census: Answer-Choice Conformity Across 44 Language Modelscited by 3
  31. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  32. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweightingcited by 2
  33. 2026Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoningcited by 2
  34. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  35. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  36. 2026Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learningcited by 2
  37. 2026Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoningcited by 2
  38. 2026Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolutioncited by 1

50 most cited of 79 All 79 (JSON)