# The Price of Format: Diversity Collapse in LLMs

arXiv: https://arxiv.org/abs/2505.18949 · HTML: https://mlsanity.com/paper/2505.18949/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability](https://mlsanity.com/paper/2402.18667/) (2024)
- [Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications](https://mlsanity.com/paper/2311.16153/) (2023)
- [Measuring and Improving Semantic Diversity of Dialogue Generation](https://mlsanity.com/paper/2210.05725/) (2022)
- [Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition](https://mlsanity.com/paper/2410.01380/) (2024)
- [UltraGen: Extremely Fine-grained Controllable Generation via Attribute Reconstruction and Global Preference Optimization](https://mlsanity.com/paper/2502.12375/) (2025)

## Built on by (most cited first, 30 of 43)

- [Outcome-based Exploration for LLM Reasoning](https://mlsanity.com/paper/2509.06941/) (2025)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [KL-Regularized Reinforcement Learning is Designed to Mode Collapse](https://mlsanity.com/paper/2510.20817/) (2025)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [Where does output diversity collapse in post-training?](https://mlsanity.com/paper/2604.16027/) (2026)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Unlocking LLM Creativity in Science through Analogical Reasoning](https://mlsanity.com/paper/2605.11258/) (2026)
- [A Single Character can Make or Break Your LLM Evals](https://mlsanity.com/paper/2510.05152/) (2025)
- [Annotations Mitigate Post-Training Mode Collapse](https://mlsanity.com/paper/2605.09995/) (2026)
- [ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline](https://mlsanity.com/paper/2508.06094/) (2025)
- [Test-time Diverse Reasoning by Riemannian Activation Steering](https://mlsanity.com/paper/2511.08305/) (2025)
- [Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steering](https://mlsanity.com/paper/2601.22010/) (2026)
- [Schema-Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding](https://mlsanity.com/paper/2604.14862/) (2026)
- [Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models](https://mlsanity.com/paper/2604.22411/) (2026)
- [The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs](https://mlsanity.com/paper/2605.08737/) (2026)
- [Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning](https://mlsanity.com/paper/2606.03962/) (2026)
- [Losing our Tail, Again: (Un)Natural Selection & Multilingual LLMs](https://mlsanity.com/paper/2507.03933/) (2025)
- [Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models](https://mlsanity.com/paper/2510.15061/) (2025)
- [Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models](https://mlsanity.com/paper/2601.22478/) (2026)
- [What LLMs Think When You Don't Tell Them What to Think About?](https://mlsanity.com/paper/2602.01689/) (2026)
- [Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software](https://mlsanity.com/paper/2602.04894/) (2026)
- [Ex Ante Evaluation of AI-Induced Idea Diversity Collapse](https://mlsanity.com/paper/2605.06540/) (2026)
- [Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)](https://mlsanity.com/paper/2605.27268/) (2026)
- [Anchorless Diversification for Parallel LLM Ideation](https://mlsanity.com/paper/2605.30150/) (2026)
- [Argument Collapse: LLMs Flatten Long-Form Public Debate](https://mlsanity.com/paper/2606.01736/) (2026)
- [A Multi-Timescale Recursive Self-Improvement Engine for Open-Ended Persona Growth](https://mlsanity.com/paper/2607.08252/) (2026)
- [Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling](https://mlsanity.com/paper/2607.18310/) (2026)
- [The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora](https://mlsanity.com/paper/2608.09093/) (2026)
- [Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models](https://mlsanity.com/paper/2608.11426/) (2026)
- [Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT](https://mlsanity.com/paper/2602.11220/) (2026)

## Abstract (arXiv)

Instruction-tuned large language models (LLMs) employ structured templates, such as role markers and special tokens, to enforce format consistency during inference. However, we identify a critical limitation of such formatting: it induces a phenomenon we term diversity collapse, where the model generates semantically similar outputs for open-ended inputs, undermining creativity and variability. We systematically evaluate this effect across tasks like story completion and free-form generation, finding that (1) diversity collapse persists even under high-temperature sampling, and (2) structural tokens in templates significantly constrain the model's output space. To contextualize these findings, we fine-tune the same model using a range of structured prompts and then evaluate them across three axes: downstream task performance, alignment behavior, and output diversity. Our analysis shows that format consistency between fine-tuning and inference is crucial for structure-sensitive tasks (e.g., GSM8K, IFEval), but has marginal influence on knowledge-heavy tasks (e.g., MMLU, WebQuestions). In contrast, output diversity is primarily governed by the presence or absence of structural tokens, with minimal formatting yielding the most diverse outputs. These findings reveal that current prompting conventions, while beneficial for alignment, may inadvertently suppress output diversity, underscoring the need for diversity-aware prompt design and instruction tuning.