mlsanityData to 7 Oct 2026

Post-training Large Language Models for Diverse High-Quality Responses

arXiv 2509.04784 ↗First posted 5 Sep 2025cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has emerged as a popular method for post-training large language models (LLMs). While improving the model's performance on downstream tasks, it often reduces the model's output diversity, leading to narrow, canonical responses. Existing methods to enhance diversity are limited, either by operating at inference time or by focusing on surface-level differences. We propose a novel training method named DQO (Diversity Quality Optimization) based on determinantal point processes (DPPs) to jointly optimize LLMs for quality and semantic diversity. Our approach samples and embeds a group of responses for each prompt, then uses the determinant of a kernel-based similarity matrix to measure diversity as the volume spanned by the embeddings of these responses. DQO is flexible and can be applied on top of existing RL algorithms. Experiments across instruction-following, summarization, story generation, and reasoning tasks demonstrate that our method substantially improves semantic diversity without sacrificing model quality.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

30 of 44 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
18 more
  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2022Matryoshka Representation Learningcited by 268
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  4. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  5. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  6. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  7. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  8. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 78
  9. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  10. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  11. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  12. 2025Diverse Preference Optimizationcited by 48
  13. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 47
  14. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 39
  15. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  16. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  17. 2025Intent Factored Generation: Unleashing the Diversity in Your Language Modelcited by 6
  18. 2025DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 17.

19 papers.

Sort:

By area:

  1. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  2. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  3. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  4. 2026SD-E2: Semantic Exploration for Reasoning Under Token Budgetscited by 3
  5. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  6. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  7. 2026Discourse Diversity in Multi-Turn Empathic Dialoguecited by 1
  8. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoningcited by 1
  9. 2026Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMscited by 1
  10. 2026PLURAL: A Global Dataset for Value Alignmentcited by 1
  11. 2026emb-diversity: A Tool for Embedding-Based Measurement of Data Diversitycited by 1
  12. 2026What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systemscited by 1
7 more
  1. 2025VOYAGER: A Training Free Approach for Generating Diverse Datasets using LLMscited by 0
  2. 2026MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignmentcited by 0
  3. 2026Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learningcited by 0
  4. 2026Strategically Diverse Sampling for Self-Trainingcited by 0
  5. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0
  6. 2026Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Samplingcited by 0
  7. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0