mlsanityData to 14 Oct 2026

Diverse Preference Optimization

arXiv 2501.18101 ↗First posted 30 Jan 2025cs.CLcited by 52

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. In this work we introduce Diverse Preference Optimization (DivPO), an optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and a 74.6% increase in story diversity, while maintaining similar win rates as standard baselines. On general instruction following, DivPO results in a 46.2% increase in diversity, and a 2.4% winrate improvement compared to DPO.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 65 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

55 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2024Mixtral of Expertscited by 1,823
  5. 2018Hierarchical Neural Story Generationcited by 1,193
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023Whose Opinions Do Language Models Reflect?cited by 773
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  13. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  14. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  15. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  16. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  17. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  18. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  19. 2019SPoC: Search-based Pseudocode to Codecited by 176
  20. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  21. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  22. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  23. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  24. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  25. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  26. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  27. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  28. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  29. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  30. 2023Quality-Diversity through AI Feedbackcited by 40
  31. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 38
  32. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 29
  33. 2024Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMscited by 17
  34. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  35. 2024Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinementcited by 13
  36. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  37. 2024Adaptive Decoding via Latent Preference Optimizationcited by 10
  38. 2024Diversity-Rewarded CFG Distillationcited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 25.

52 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  4. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 66
  5. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  6. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  7. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
  8. 2025AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprisecited by 33
  9. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  10. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  11. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  12. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
38 more
  1. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  2. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  3. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  4. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  5. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  6. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  7. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  8. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  9. 2025Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputcited by 6
  10. 2025Domain Regeneration: How well do LLMs match syntactic properties of text domains?cited by 6
  11. 2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 6
  12. 2026PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMscited by 6
  13. 2026Unlocking LLM Creativity in Science through Analogical Reasoningcited by 6
  14. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4
  15. 2025Geometry of Knowledge Allows Extending Diversity Boundaries of Large Language Modelscited by 3
  16. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  17. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  18. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  19. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  20. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matchingcited by 2
  21. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  22. 2026Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoningcited by 2
  23. 2025Generative flow-based warm start of the variational quantum eigensolvercited by 1
  24. 2025Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Modelscited by 1
  25. 2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
  26. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  27. 2026Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learningcited by 1
  28. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigmcited by 0
  29. 2026STEER: Inference-Time Risk Control via Constrained Quality-Diversity Searchcited by 0
  30. 2026Democratic Preference Alignment via Sortition-Weighted RLHFcited by 0
  31. 2026Discovering Novel LLM Experts via Task-Capability Coevolutioncited by 0
  32. 2026POLARIS: Guiding Small Models to Write Long Storiescited by 0
  33. 2026More Is Not More: What Matters for Diversity in LLM Opinions?cited by 0
  34. 2026Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Trainingcited by 0
  35. 2026Enabling Creative Exploration for Vibe Design Agentscited by 0
  36. 2026Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coveragecited by 0
  37. 2026One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generationcited by 0
  38. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Modifying Large Language Model Post-Training for Diverse Creative Writing
  2. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models
  3. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions
  4. 2025Jointly Reinforcing Diversity and Quality in Language Model Generations
  5. 2024GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets
  6. 2024VPO: Leveraging the Number of Votes in Preference Optimization
  7. 2025A Survey of Direct Preference Optimization
  8. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Models