mlsanityData to 7 Oct 2026

Diverse Preference Learning for Capabilities and Alignment

arXiv 2511.08594 ↗First posted 29 Oct 2025cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The ability of LLMs to represent diverse perspectives is critical as they increasingly impact society. However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly reduce the diversity of LLM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect a narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model to systematically overweight majority opinions and sacrifice diversity in its outputs. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty - allowing for fine-grained control over LLM generation diversity. From a capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing a wider range of societal viewpoints and display improved logit calibration. Notably, Soft Preference Learning resembles, but is a Pareto improvement over, standard temperature scaling.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 51 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

38 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Calibrating Language Models with Adaptive Temperature Scaling29 Sep
  2. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  3. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference5 Dec
  4. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  5. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 31

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
19 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  6. 2023Nash Learning from Human Feedbackcited by 220
  7. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  9. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  10. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  11. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  13. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  14. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  15. 2024Delving into LLM-assisted writing in biomedical publications through excess vocabularycited by 75
  16. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  17. 2023Diminished Diversity-of-Thought in a Standard Large Language Modelcited by 43
  18. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  19. 2023Long Horizon Temperature Scalingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 22.

25 papers.

Sort:

By area:

  1. 2026Where does output diversity collapse in post-training?cited by 10
  2. 2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Modelscited by 9
  3. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  4. 2026PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMscited by 5
  5. 2026Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scalingcited by 4
  6. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
  7. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4
  8. 2025Love First, Know Later: Persona-Based Romantic Compatibility Through LLM Text World Enginescited by 3
  9. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  10. 2026PERSPECTRA: A Scalable and Configurable Pluralist Benchmark of Perspectives from Argumentscited by 3
  11. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  12. 2025Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learningcited by 1
13 more
  1. 2026VISPA: Pluralistic Alignment via Automatic Value Selection and Activationcited by 1
  2. 2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
  3. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 0
  4. 2026Know Thy Reasoner: Not All Language Models Explore Alikecited by 0
  5. 2026Entropy Centroids as Intrinsic Rewards for Test-Time Scalingcited by 0
  6. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  7. 2026AI Alignment From Social Choice Perspectivescited by 0
  8. 2026Collective cooperation without individual fidelity in LLM agentscited by 0
  9. 2026When Does the Best Sampling Temperature Rise with the Budget? Sufficient Conditions for Pass@kcited by 0
  10. 2026Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Trainingcited by 0
  11. 2026Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learningcited by 0
  12. 2026Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMscited by 0
  13. 2026TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  4. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset
  5. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributions
  6. 2026BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
  7. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs