mlsanityData to 14 Oct 2026

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

arXiv 2609.14896 ↗First posted 14 Sep 2026cs.CL cs.AI cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 71 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

51 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct

Other references · 48

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  12. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
36 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  2. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  3. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  4. 2015Illuminating search spaces by mapping elitescited by 425
  5. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  6. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  7. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
  8. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  9. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
  10. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  11. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  12. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  13. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 80
  14. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  15. 2025Diverse Preference Optimizationcited by 52
  16. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  17. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  18. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 40
  19. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 38
  20. 2025HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generationcited by 37
  21. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  22. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 27
  23. 2025Multilingual Prompting for Improving LLM Generation Diversitycited by 25
  24. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  25. 2024Learning to Cooperate with Humans using Generative Agentscited by 24
  26. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning taskscited by 22
  27. 2025Literature-Grounded Novelty Assessment of Scientific Ideascited by 19
  28. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  29. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  30. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  31. 2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Modelscited by 10
  32. 2025Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Provingcited by 8
  33. 2025Mode-Conditioning Unlocks Superior Test-Time Scalingcited by 8
  34. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6
  35. 2025Multi-Novelty: Improve the Diversity and Novelty of Contents Generated by Large Language Models via inference-time Multi-Views Brainstormingcited by 4
  36. 2026Persona Switch: Mixing Distinct Perspectives in Decoding Timecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Sharpening Tax in Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Jointly Reinforcing Diversity and Quality in Language Model Generations
  2. 2023Integrating LLMs and Decision Transformers for Language Grounded Generative Quality-Diversity
  3. 2025Structured Diversity Control: A Dual-Level Framework for Group-Aware Multi-Agent Coordination
  4. 2026Behavioral Foundation Models for Quality Diversity
  5. 2026Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
  6. 2024Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning
  7. 2026Learning Generalizable Behaviors for Terminal Agents
  8. 2023Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization