# Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models

arXiv: https://arxiv.org/abs/2505.16245 · HTML: https://mlsanity.com/paper/2505.16245/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Generative artificial intelligence enhances creativity but reduces the diversity of novel content](https://mlsanity.com/paper/2312.00506/) (2023)
- [Homogenization Effects of Large Language Models on Human Creative Ideation](https://mlsanity.com/paper/2402.01536/) (2024)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing](https://mlsanity.com/paper/2404.12253/) (2024)
- [The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text](https://mlsanity.com/paper/2311.09807/) (2023)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)

## Built on by (most cited first, 9 of 9)

- [DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing](https://mlsanity.com/paper/2601.09609/) (2026)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs](https://mlsanity.com/paper/2605.30021/) (2026)
- [Anchorless Diversification for Parallel LLM Ideation](https://mlsanity.com/paper/2605.30150/) (2026)
- [emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity](https://mlsanity.com/paper/2607.19848/) (2026)
- [Playing with Words, Improving with Rewards: Training Language Models for Creative Association](https://mlsanity.com/paper/2605.27832/) (2026)
- [Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection](https://mlsanity.com/paper/2605.28664/) (2026)
- [When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer](https://mlsanity.com/paper/2605.29190/) (2026)
- [Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity](https://mlsanity.com/paper/2608.13430/) (2026)

## Abstract (arXiv)

Diverse language model responses are crucial for creative generation, open-ended tasks, and self-improvement training. We show that common diversity metrics, and even reward models used for preference optimization, systematically bias models toward shorter outputs, limiting expressiveness. To address this, we introduce Diverse, not Short (Diverse-NS), a length-controlled data selection strategy that improves response diversity while maintaining length parity. By generating and filtering preference data that balances diversity, quality, and length, Diverse-NS enables effective training using only 3,000 preference pairs. Applied to LLaMA-3.1-8B and the Olmo-2 family, Diverse-NS substantially enhances lexical and semantic diversity. We show consistent improvement in diversity with minor reduction or gains in response quality on four creative generation tasks: Divergent Associations, Persona Generation, Alternate Uses, and Creative Writing. Surprisingly, experiments with the Olmo-2 model family (7B, and 13B) show that smaller models like Olmo-2-7B can serve as effective "diversity teachers" for larger models. By explicitly addressing length bias, our method efficiently pushes models toward more diverse and expressive outputs.